Debian: compiling maps and producing identical files

[apo]

The last technical challenge.

I would like to build all maps from source and I also have to. I already know that those maps must be identical to the official maps to ensure compatibility between client and servers of different architectures.

The maps need to be compiled only once on my own computer but the build process should be reproducible for others.

Simply spoken: What is necessary to create an always identical 0maps.pk3 that will be recognized by all clients and servers? Any ideas welcome.

[geever]

As far as I know mostly (if not only) floating-point weirdness can create different maps, so possibly they need to be compiled on the same CPU arch (but the compiled maps are not arch-dependent then.)

To check they're the same as we share I think the best is comparing to http://ufoai.org/maps/2.5/MAPS . I believe this is the list that is used by map-get (maps-sync make target) to update local maps. And it contains

Code: [Select]

<bsp-path> <bsp-md5> <map-md5> lines of all current maps. This list is recreated on every rebuild cycle but I'm sure we can create & keep similar static list files for the future releases if needed.

-geever

[apo]

Thank you. Is it necessary to keep a specific sorting order for the maps within the 0maps.pk3 file or is this neglectable?

Besides how do you keep the compatibility between different versions of UFO:AI? e.g. OpenArena provides patch pk3 files for new versions, so that you are able to connect to a new server and the new files will be downloaded automatically.

I haven't found a final answer for future upgrades yet. All in all version 2.5 ships 1,2 GB zip compressed data files. Let's assume you changed one map, or one music file in a future release, the whole package would need to be downloaded again. Indeed a major release won't happen frequently but I'm always looking for ideas how to save bandwith and resources.

[geever]

Thank you. Is it necessary to keep a specific sorting order for the maps within the 0maps.pk3 file or is this neglectable?

The order doesn't matter.

Besides how do you keep the compatibility between different versions of UFO:AI? e.g. OpenArena provides patch pk3 files for new versions, so that you are able to connect to a new server and the new files will be downloaded automatically.

Usually we don't keep compatibility between major releases as it really happen rarely (1-2 years) and we change a lot. But last time we had a serious issue (with a map around v2.3) we "released" a patch pk3 too. We don't have an automatic update feature (yet), people had to manually download and add it to the game.

-geever

[apo]

I have successfully compiled all maps for the first time. Of course the md5sums didn't match. 

Two requests:

Could you move http://ufoai.org/maps/2.5/MAPS into your git repository, at least for the final release version?
What do you think about adding the commit string to the file, so that it is clear against which version the maps were compiled?

I'm struggling to see the problem:

For compiling the maps successfully only base/maps and base/textures are required, correct?

I use an intel based x86_64 linux system, your Makefile and commit: 480f1117edca8aea2dbc2073627c8af24ffe15fa from the 18th of August.

It took my core duo from 2009 10 hours to compile all maps. 

[Mattn]

I have successfully compiled all maps for the first time. Of course the md5sums didn't match. 
Two requests:
Could you move http://ufoai.org/maps/2.5/MAPS into your git repository, at least for the final release version?
What do you think about adding the commit string to the file, so that it is clear against which version the maps were compiled?

that's python, i don't touch that but patches are welcome. it's in contrib/map-get.

about adding that file to our repo... I don't see a reason yet to add it to the repo. What could one be? I don't think that this would help you to rebuild the maps to have the same checksum, no?

For compiling the maps successfully only base/maps and base/textures are required, correct?

yes - base/ufos/entities.ufo is needed if you perform checks on the maps

I use an intel based x86_64 linux system, your Makefile and commit: 480f1117edca8aea2dbc2073627c8af24ffe15fa from the 18th of August.

It took my core duo from 2009 10 hours to compile all maps. 

that's long: http://ufoai.org/wiki/Mapping/Compile_time

[apo]

about adding that file to our repo... I don't see a reason yet to add it to the repo. What could one be? I don't think that this would help you to rebuild the maps to have the same checksum, no?

My reasoning is that I would like to ship this file within the source package and integrate it into a check to verify that the outcome of the build process is identical to what you ship officially. I already have to create a source tarball from your git repository but I guess it doesn't really matter if I copy the file from git or perform some kind of wget command to retrieve the file. The only important thing is that this file is preserved somewhere, so that you can always compare the result despite the release of new revisions.

that's long: http://ufoai.org/wiki/Mapping/Compile_time

Indeed. I will try if I can speed things up by using more threads and/or reducing your default nice value.

[geever]

that's python, i don't touch that but patches are welcome. it's in contrib/map-get.

Hehe, I don't touch snakes either, but I can rewrite it in Perl if you would like and extend that...

that's long: http://ufoai.org/wiki/Mapping/Compile_time

I have the feeling for some time that our map compiling times multiplied recently... I wanted to do some tests.. but I don't like when my CPU goes up to 80C.... The stock cooler isn't cool enough for our maps.

-geever

[Mattn]

My reasoning is that I would like to ship this file within the source package and integrate it into a check to verify that the outcome of the build process is identical to what you ship officially. I already have to create a source tarball from your git repository but I guess it doesn't really matter if I copy the file from git or perform some kind of wget command to retrieve the file. The only important thing is that this file is preserved somewhere, so that you can always compare the result despite the release of new revisions.

i will change the make target to generate a md5sum list for all bsps that are put into the final zip file and will upload it to the release folder on sf.net. hope that works, too. i would rather not use the file that you linked, because that file is updated by a process running in the background. i don't exactly know when it's executed and stuff like that. so producing my own md5sums from the local files that i will put into the zip is safer imo.

[O01eg]

As far as I know mostly (if not only) floating-point weirdness can create different maps, so possibly they need to be compiled on the same CPU arch (but the compiled maps are not arch-dependent then.)

What if the server send maps to clients? Then the map on the server and clients will be same.

[geever]

What if the server send maps to clients? Then the map on the server and clients will be same.

It is a good idea for custom maps. But doesn't solve the problem. Our prebuilt packages should be compatible. People download ~1.2G for the game then they should download again each map on each different servers they play on (at worst case)...

-geever

[apo]

I have compiled the maps again, this time with more parallel jobs but the best I can do with a Core Duo with 4 GB RAM is 4,5 hours so far.

Though I still get different md5sums for the *.bsp files.

[Sandro]

Current map compliler is mathematically unstable, so resulting .bsp depends on gcc flags used to compile it, CPU used, number of threads and other factors. So it is hopeless to get identical bsps with it.

Somebody tried to make a Java-based compiler, which theoretically should give identical results on every platform, if strategically peppered with strictfp modifier, but this work was abandonned long ago.
Or someone in future may be willing to rewrite the compiler into the fixed point 

[Mattn]

we once used -ffloat-store - but it was slower: link

[O01eg]

People download ~1.2G for the game then they should download again each map on each different servers they play on (at worst case)...
-geever

It is better than impossibility to play between different server and clients.