Comments on Compiling

[freegamer]

Where to start...

The SVN is very poorly organised.  Why isn't it broken into sub projects / modules?  Why am I getting gtkradiant.exe when I'm checking out the main project?  Split stuff like that out into sub projects so people don't have to download stuff they don't need.

And why the hell must I compile the maps?  WTF is this about?  I had to leave it overnight to do this.  Ridiculous!?!?  Keep the map 'source' in it's own module and commit already compiled maps to the main project.

I mean, to have to wait 10 hours to svn/compile this is ludicrous, it really is.  You could eliminate almost 90% of that with some smarter organisation.

Finally onto something "not a gripe" but a problem instead - I get a segmentation fault...

Code: [Select]

charles@charles-laptop:~/Games/ufoai$ ./ufo

---- filesystem initialization -----
Adding game dir: ./base
using /home/charles/.ufoai/2.2-dev/base for writing
Adding game dir: /home/charles/.ufoai/2.2-dev/base
execing default.cfg
couldn't exec config.cfg
execing keys.cfg

----- network initialization -------
libcurl/7.16.4 OpenSSL/0.9.8e zlib/1.2.3.3 libidn/1.0 initialized.

------ server initialization -------
added 3 maps to the mapcycle
...using language: en_GB.UTF-8

----- console initialization -------
couldn't load history
Console initialized.

------- video initialization -------
SDL version: 1.2.11
I: setting mode 6: 1024x768 (fullscreen: no)
Segmentation fault
Yes, I have libGL.so.1 in /usr/lib and a modern graphics card.

[Mattn]

please try to send a gdb backtrace (see debug wiki article) - otherwise we can't really fix that segfault

[Punkiee]

If you want a compiled game, get the stable version or get the beta version. Those installers are used to maged the binaries, not the source.
If you want the source, get it from svn. Svn manages the source, not the compiled results.

If you did choose the svn version and not the compiled version then you have to compile yourself. I cant see the point of choosing the wrong version and then blaming others on your choice.

[freegamer]

Punkee, that opinion of yours is so obtuse it's stupifying.

The line between source and binary is not that simple.  Even though you seem to think it is, in the UFOAI subversion it is already distorted with .dlls and .exes amongst the files downloaded by a basic trunk/ufoai checkout.

The compiled maps are platform independent are they not?  Why require anybody who is not actually modelling the maps to compile the maps?  It's just making lots of work for everybody else and their computers.  Store the maps in their own subproject (e.g. trunk/maps) and commit the compiled maps to the main game project (i.e. trunk/ufoai) so that all developers have the latest maps without having to compile them (i.e. only: svn up), so that people who want to check out the source to possibly contribute in a non-map way can do so without a 10 hours compilation time.  It makes perfect sense and is an ideal usage of something like SVN.

A similar high-profile project doing it this way is Vega Strike - they have a "masters" subproject where all [lossless] originals get stored (i.e. all .png,.xcf,.blend,etc are in trunk/masters).  They commit the "compiled" (or in this case transformed) lossy images to a data subproject (trunk/data4.x), and non-artists use this data dir to stay current.

Apply a little common sense.  This is the way it should be done.  At the moment everybody is having to do a task that only needs to be done once.

[freegamer]

please try to send a gdb backtrace (see debug wiki article) - otherwise we can't really fix that segfault

I tried but not sure if this will help:

Code: [Select]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1218341184 (LWP 28162)]
0xb7c80f37 in strstr () from /lib/tls/i686/cmov/libc.so.6
(gdb) bt full
#0  0xb7c80f37 in strstr () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.
#1  0xb7d980af in ?? () from /usr/lib/libSDL-1.2.so.0
No symbol table info available.
#2  0x00000000 in ?? ()
No symbol table info available.

[Mattn]

no sorry, that doesn't help - did you compile in release mode?

[Punkiee]

Although i seem to be too stupid to fully appreciate what i am doing, I did a checkout of svn and compiled the game in 2 minutes, without the maps. I copied the maps from the beta and was ready to play in 5 minutes. Because that didnt take 10 hours i must be so blind its stupifying.

svn is still for managing the source, sources as in human readable text. For binaries it is of less importance. Still a few pointers on how to get compiled maps instead of haveing to build your own would be appreciated. I didnt rant about it but that is my lack of common sense.

[freegamer]

Although i seem to be too stupid to fully appreciate what i am doing, I did a checkout of svn and compiled the game in 2 minutes, without the maps. I copied the maps from the beta and was ready to play in 5 minutes. Because that didnt take 10 hours i must be so blind its stupifying.

svn is still for managing the source, sources as in human readable text. For binaries it is of less importance. Still a few pointers on how to get compiled maps instead of haveing to build your own would be appreciated. I didnt rant about it but that is my lack of common sense.

That's a brilliant attempt to be right, but still wrong.

I want the SVN.  Not the beta, but the SVN.  SVN maps is not necessarily the beta maps.  And pre-beta, this was even more true.  Again, you are trying to be clever but missing the entire point that the current SVN layout makes things more difficult than it needs to be.

I reiterate; move the maps source to their own module, and have the actual maps committed to the main ufoai module, and that way it's easier for everybody.

[freegamer]

Perfect example.

I just did an svn up, probably a week since I last did this as I was away.  Look at all the map updates:

Code: [Select]

U    base/maps/rivertown01d.map
U    base/maps/africa04n.map
U    base/maps/corrupt_city04n.map
U    base/maps/rivertown01n.map
U    base/maps/estate01d.map
U    base/maps/estate01n.map
U    base/maps/lake02d.map
U    base/maps/apartments01d.map
U    base/maps/lake02n.map
U    base/maps/const05d.map
U    base/maps/const05n.map
U    base/maps/stadium06d.map
U    base/maps/convoy01d.map
U    base/maps/oriental06d.map
U    base/maps/industrial06d.map
U    base/maps/stadium06n.map
U    base/maps/convoy01n.map
U    base/maps/harvestercrash04d.map
U    base/maps/druglord07d.map
U    base/maps/village07d.map
U    base/maps/oriental06n.map
U    base/maps/industrial06n.map
U    base/maps/harvestercrash04n.map
U    base/maps/druglord07n.map
U    base/maps/village07n.map
U    base/maps/excavation06d.map
U    base/maps/terrain08d.map
U    base/maps/subway07d.map
U    base/maps/excavation06n.map
U    base/maps/terrain08n.map
U    base/maps/bunker01d.map
U    base/maps/italy05d.map
U    base/maps/subway07n.map
U    base/maps/farm08d.map
U    base/maps/bunker01n.map
U    base/maps/italy05n.map
U    base/maps/dam01d.map
U    base/maps/farm08n.map
U    base/maps/mart05d.map
U    base/maps/dam01n.map
U    base/maps/mart05n.map
U    base/maps/harbor07d.map
U    base/maps/harbor07n.map
U    base/maps/transport06d.map
U    base/maps/tower06d.map
U    base/maps/terrain02d.map
U    base/maps/transport06n.map
U    base/maps/tower06n.map
U    base/maps/terrain02n.map
U    base/maps/oriental05d.map
U    base/maps/oriental05n.map
U    base/maps/bridge01d.map
U    base/maps/office06d.map
U    base/maps/bridge01n.map
U    base/maps/office06n.map
U    base/maps/city04d.map
U    base/maps/lake03d.map
U    base/maps/city04n.map
U    base/maps/mine01d.map
U    base/maps/lake03n.map
U    base/maps/powerplant01d.map
U    base/maps/japan06d.map
U    base/maps/mine01n.map
U    base/maps/japan06n.map
U    base/maps/gasstation05d.map
U    base/maps/pdi08d.map
U    base/maps/africa04d.map
U    base/maps/corrupt_city04d.map
U    base/maps/gasstation05n.map
U    base/maps/pdi08n.map
So now I have to recompile all these maps... and so does anybody else using SVN.  Lots of needless time and annoyance all-round.

What I propose would instead take about 30s of time for 1 person instead of the hours of time it takes everybody at the moment.  Somebody updates a map and compiles it.  They commit both the updated source version to trunk/maps and the updated compiled version to trunks/ufoai, now everybody who goes 'svn up' gets the compiled map, doesn't have to mess around recompiling the maps, and in general life is a lot easier especially for new contributors (something that every project wants).

Punkiee, I concede there are ways around the current annoying system, but it should be better than it is.  I'm only trying to _help_ UFO:AI by suggesting a better way of handling things and so far nobody (especially you) has put up a convincing argument as to why it is not already this way other than "SVN handles source" which is already incorrect for UFO:AI which has multiple dlls and gtkradiant.exe amongst the current files in SVN.

Vega Strike do it this way and it makes the lives of contributors and bleeding-edge testers/players much much easier.

[freegamer]

Ok, segfault solved.  An Ubuntu Gutsy update had hosed my nvidia glx driver so GL stuff wouldn't work.  Still, a nicer way of notifying the user that the graphics configuration is not good enough would be appreciated, I'm sure.

[Mattn]

i'm sorry we discussed this internally quite some time now, the majority of the team is against it. Everything that can be redone with the code and tools in svn should not be in binary form. You still have the chance to download the mappack which i update every few days.

[Punkiee]

It is not "the point" but 2 points. First point is the svn layout and so far i have not written a word on that. And i concur that a split between those parts is a good idea because they are different tasks and are not closely coupled. We can see this separation already present in all areas (targets, doc, forum, ...) and even in svn as the directory structure separates for example the maps from other stuff. So if you want to update just part of the code, then "cd src;svn update" should do. But this doesnt work in all scenarios so I am looking forward to a more formalized splitup.

The second point being the long times for maps compiling.

So now I have to recompile all these maps... and so does anybody else using SVN.  Lots of needless time and annoyance all-round.

Apparently the non annoyed svn users just dont recompile their maps. For over a month i have only been looking in the game code and thus didnt need to recompile the language files nor the maps. As shown you can get bootstrapped fast by the mappacks. So all in all, the long time for compiling maps is a problem with a magnitude several orders less then you make it look like.

UFO:AI which has multiple dlls and gtkradiant.exe

Executables in svn? Of course, they ARE the source! Maps are NOT because they are a derived product, we can compile them again from any revision we d like to. source >< derived. source in svn, derived not in svn. The difference between source and derived products IS simple, if you are not confused by the form in which it is delivered. So this svn repository *does* handle source only.

What if you *never* want to compile a map and also *always* want the latest maps? In this case the system fails but i cant see any legit use case for it. The only thing that can solve this is to commit the compiled map for every change to svn. Version control for binaries is, as i put before, of little value. It also breaks the consistency, the integrity of the database subversion! Because for every compiled map is depending on other sources but that cant be made explicit. It does not make sense at all to throw this away for it is the purpose of svn to ensure this. Thats one reason why the first rule of svn is "source only".

All in all:
*subproject already exist but svn is not splitup.
*you dont have to spend a lot of time compiling maps
*the big elephant problem is actually a small mice and the cure for this small improvement has a very high price.
*svn is not what think it is

[freegamer]

I am looking forward to a more formalized splitup.

Glad we agree on that. :-)

The second point being the long times for maps compiling. Apparently the non annoyed svn users just dont recompile their maps. For over a month i have only been looking in the game code and thus didnt need to recompile the language files nor the maps.

This is a good point except... you are not using svn maps - i.e. not using full svn.  Know what that means?  People not testing the exact contents properly, but merely a close estimate of the svn version of the game.  Subtle bugs could easily sneak in with this scenario.

As shown you can get bootstrapped fast by the mappacks. So all in all, the long time for compiling maps is a problem with a magnitude several orders less then you make it look like.

With the svn split you agree with, ok, it is a smaller issue, but it can be made to go away entirely.

Executables in svn? Of course, they ARE the source!

That's a strange statement to make.  A .exe or .dll can not be considered source.  They can be considered libraries or tools.  Talking about derived vs source makes no sense and in all my years of experience I have never heard anybody try to rationalise this like you do.

*svn is not what [freegamer] thinks it is

I know exactly what svn is, but I'm suggesting that binding svn to handling only source is an ideological view, whereas the practical view is to use it for things that aid development and makes it easier for new contributors.  By using svn astutely instead of ideologically, you can distill your svn build process to this:

Code: [Select]

svn co http://ufoai.svn.sourceforge.net/svnroot/ufoai ufoai
cd ufoai
make
And getting completely up to date to this:

Code: [Select]

svn up
make clean && make
Instead you have multiple steps, multiple ways of accomplishing these steps, and an arduous map compilation process.

[Punkiee]

Talking about derived vs source makes no sense and in all my years of experience I have never heard anybody try to rationalise this like you do.

And the other way around: I have never heard anyone suggesting to keep derived results in svn. It stumped me.
Mark that i said up to (now) 3 times that dlls and such are not really the cup of tea for svn, but it is conform the method of a versioning system.

Subtle bugs could easily sneak in with this scenario.

While i am indeed NOT testing the full content of that revision, i am also NOT intending to do that. I only want the code and i am using the full code content for it. The maps are of a certain format that is fixed. That is the only interface needed for the code to work on the maps. Therefore as long as the maps adhere to that format, the code and the maps are completely orthogonal to each other and thus can be tested completely orthogonal. Therefore this scenario is not flawed.
As a result the updating, as you described, can even be done faster without keeping compiled maps in svn, if(!) the split in subprojects happened. The first time checking out is indeed not so trivial but its a one time cost and the process can still be streamlined (a lot).

[freegamer]

Ok, the plot thickens.

You _can_ get the latest compiled maps using rsync.

However, this is not documented anywhere relevant.  I just looked through all the docs related to compiling and not one mention of this, only ever instructions to 'make maps'.  I only actually discovered this rsync stuff through the mailing list discussion on this subject.

You guys need to document that and mention it in the right places.  How are people supposed to know otherwise?