compile_maps.bat improvements

[Duke]

Last time when I had to recompile all the maps, I watched the system monitor. I noticed that there was quite a lot of time where the processors were not at 100% cpu usage.

As cpu seems to be the limiting factor, I started a second process with compile_maps, just for fun. And it worked nicely for a while. All the processors were permanently at a 100%. But of course sooner or later both processes ended up compiling the same map coz they are not protected against that in any way.

So the idea is to create some recompile_maps.bat that can spawn a second batchfile that handles n*.map through z*.map, while the recompile_maps.bat is restricted to a*.map to m*.map. We will lose some time for process concurrency overhead, but I have no idea how much that really is. Imho it's worth trying.

Does anyone of the batchfile-artists around want to give it a try ? You might end up in the Guinnes Book of Records for the fastest full compile of UFO maps

[Destructavator]

I remember Muton was working on this at some point, he posted and attached some batch files and other utils meant to compile the maps quicker.  They're old though, and might need to be updated (and dug out of the older forum posts).

[Duke]

Yes, but IIRC Muton's solution was rejected because we don't want additional utils in the project.
I was thinking of a batchfile-only solution.

[Mattn]

he had a batch file only solution, too - autodetecting the cpus you have and so on... - why aren't you using the makefile?

[Destructavator]

- why aren't you using the makefile?

Probably because on Windows, getting a makefile to run can be a pain, and requires having things set up specially, extra software that isn't included with Windows, etc.  Running a batch file, on the other hand, is laughably easy on Windows, just click on the file icon and it runs, no third party stuff needs to be installed, nothing special needs to be prepared ahead of time, runs with a fresh install of Windows right out-of-the-box.

This is very different than Linux, where the opposite can be true.

This is also very different than running Windows virtually in a shell or virtual environment, because some of those virtual environments on top of Linux - at least the ones I've seen - include capability to run makefiles within the shell for convenience.  A native installation of Windows, on the other hand, does not run a makefile very easily (not at all out-of-the-box), and especially a non-technical user may find it beyond their ability or simply give up trying to make it work.

[Mattn]

you would need codeblocks installed anyway - and make is included in that package, too

but to make a long story short - all these things will be removed in the future anyway - only the makefile will be maintained but hopefully integrated into codeblocks or whatever ide we are using then.

[Duke]

@Destructavator:
It's not that hard with windows: open a makefile, select a program to always open it with (ie. make.exe) and further on you'll dclick maps.mk instead of compile_maps.bat.

@Mattn:
I don't mind using make. But the current trunk of maps.mk doesn't seem to work for me. It doesn't find *.map.
I found 2 slashes where it should be backslashes under Windows, but that didn't help.
Which dir am I supposed to run it from ??

My point was to have *two* ufo2map running. Will make do that for me ?

[Mattn]

yes - it will

Code: [Select]

make maps -j 2
you have to run it from within trunk/

you can first try to run the mingw shell (this should really not be needed later, but as there wasn't done any work on this territory yet, this step is maybe still needed)

[Duke]

yes - it will

Code: [Select]

make maps -j 2
you have to run it from within trunk/

That was my first attempt;) Didn't work, even after replacing the slashes.

you can first try to run the mingw shell (this should really not be needed later, but as there wasn't done any work on this territory yet, this step is maybe still needed)

Uhmm...how do I run the mingw shell ? Didn't see any exe that sounded like that, sorry.

[Ralgert]

you can first try to run the mingw shell

Thank you for this big tip. Now i can many batchfiles kick to the moon.

@Duke: \codeblocks\MinGW\msys.bat

[Kildor]

> all these things will be removed in the future anyway
that`s bad.

There is no fast and useable methods to compile one map or one directory under windows, and "compile" button from radiant doesn`t work too. When I edit maps, I need to easy recompiling method.

[Destructavator]

> all these things will be removed in the future anyway
that`s bad.

There is no fast and useable methods to compile one map or one directory under windows, and "compile" button from radiant doesn`t work too. When I edit maps, I need to easy recompiling method.

I agree.  Please don't throw out something that works, not to mention works well and works easy for Windows users.  Keep in mind that the vast majority of people out there that use computers use Windows, I've even seen many highly technical-"geek" type of people who know Linux well still use Windows for most things as their OS of choice.  I don't mean any offense when I say this, but making a change that favors Linux-users and leaves Windows-users behind is a bit of an insulting kick in the teeth that will turn off quite a few people, not to mention at the very least a good number of potential map-makers and other potential contributers for this game.

[Mattn]

i've only said that they will be removed if the alternative is as easy to use

[Destructavator]

OK.  Sorry if I got a bit upset...   

[Duke]

Thx, Ralgert
I fired up msys and got some results:

The first results were desasterous. Then I remembered that I shouldn't use the debug version of ufo2map

The first thing I noticed was that maps.mk doesn't seem to recognize the # of processors correctly, passing  -t 1
So I gave -j 20. Result: 46 minutes.

I noticed that after some 30 mins, cpu went down from 100% and only a few maps were using up the last 15 mins. bunker and a_hangar were the last ones to finish even though they must have been started pretty early in the process.

Next, I forced -t 2 (by editing maps.mk) in order to reduce those last 15 mins. Result: 37 mins

Then I had the idea to start a_hangar asap, so I set -j 500. After some 10 mins, there were only a handful of processes running. After some 15 mins only one of them remained. Guess which. It ended after 22 mins.

Next, I forced -t 4. Result: 19 mins
I guess that sets up a new world record for compiling the UFO maps

Next try: -t 8 (the true # of 'processors' on my system. Result: 32 mins.
Seems to be too much overhead here.
I also got a 'memory allocation failure' here.

As a comparison: the compile_maps.bat takes some 50 mins on my system.

Management summary:
1. Destructavator: if you want to save time, use msys->make
2. We need to figure out why the makefile doesn't detect the # of processors
3. never ever use the debug version for a recompile maps
4. We should figure out why a_hangar and bunker maps take sooo long to compile.
5. This was on a Vista system. Needs to be verified for others.

Sidenote:
- I saw a lot of "missing carpet00n texture" msgs

I hope my findings will also help those who don't have the privilege to own a 4 core/8 processor computer to optimize their compile process.