project-navigation
Personal tools

Author Topic: compile_maps.bat improvements  (Read 20183 times)

Offline Duke

  • Administrator
  • PHALANX veteran
  • *****
  • Posts: 1037
    • View Profile
compile_maps.bat improvements
« on: October 07, 2009, 12:02:46 am »
Last time when I had to recompile all the maps, I watched the system monitor. I noticed that there was quite a lot of time where the processors were not at 100% cpu usage.

As cpu seems to be the limiting factor, I started a second process with compile_maps, just for fun. And it worked nicely for a while. All the processors were permanently at a 100%. But of course sooner or later both processes ended up compiling the same map coz they are not protected against that in any way.

So the idea is to create some recompile_maps.bat that can spawn a second batchfile that handles n*.map through z*.map, while the recompile_maps.bat is restricted to a*.map to m*.map. We will lose some time for process concurrency overhead, but I have no idea how much that really is. Imho it's worth trying.

Does anyone of the batchfile-artists around want to give it a try ? You might end up in the Guinnes Book of Records for the fastest full compile of UFO maps ;)

Offline Destructavator

  • Combination Multiple Specialty Developer
  • Administrator
  • PHALANX Commander
  • *****
  • Posts: 1908
  • Creater of Scorchcrafter, knows the zarakites...
    • View Profile
Re: compile_maps.bat improvements
« Reply #1 on: October 07, 2009, 01:20:31 am »
I remember Muton was working on this at some point, he posted and attached some batch files and other utils meant to compile the maps quicker.  They're old though, and might need to be updated (and dug out of the older forum posts).

Offline Duke

  • Administrator
  • PHALANX veteran
  • *****
  • Posts: 1037
    • View Profile
Re: compile_maps.bat improvements
« Reply #2 on: October 07, 2009, 02:04:22 pm »
Yes, but IIRC Muton's solution was rejected because we don't want additional utils in the project.
I was thinking of a batchfile-only solution.

Offline Mattn

  • Administrator
  • PHALANX Commander
  • *****
  • Posts: 4831
  • https://github.com/mgerhardy/vengi
    • View Profile
    • Vengi Voxel Tools
Re: compile_maps.bat improvements
« Reply #3 on: October 07, 2009, 05:02:04 pm »
he had a batch file only solution, too - autodetecting the cpus you have and so on... - why aren't you using the makefile?

Offline Destructavator

  • Combination Multiple Specialty Developer
  • Administrator
  • PHALANX Commander
  • *****
  • Posts: 1908
  • Creater of Scorchcrafter, knows the zarakites...
    • View Profile
Re: compile_maps.bat improvements
« Reply #4 on: October 07, 2009, 05:52:28 pm »
- why aren't you using the makefile?

Probably because on Windows, getting a makefile to run can be a pain, and requires having things set up specially, extra software that isn't included with Windows, etc.  Running a batch file, on the other hand, is laughably easy on Windows, just click on the file icon and it runs, no third party stuff needs to be installed, nothing special needs to be prepared ahead of time, runs with a fresh install of Windows right out-of-the-box.

This is very different than Linux, where the opposite can be true.

This is also very different than running Windows virtually in a shell or virtual environment, because some of those virtual environments on top of Linux - at least the ones I've seen - include capability to run makefiles within the shell for convenience.  A native installation of Windows, on the other hand, does not run a makefile very easily (not at all out-of-the-box), and especially a non-technical user may find it beyond their ability or simply give up trying to make it work.

Offline Mattn

  • Administrator
  • PHALANX Commander
  • *****
  • Posts: 4831
  • https://github.com/mgerhardy/vengi
    • View Profile
    • Vengi Voxel Tools
Re: compile_maps.bat improvements
« Reply #5 on: October 07, 2009, 06:00:37 pm »
you would need codeblocks installed anyway - and make is included in that package, too ;)

but to make a long story short - all these things will be removed in the future anyway - only the makefile will be maintained but hopefully integrated into codeblocks or whatever ide we are using then.

Offline Duke

  • Administrator
  • PHALANX veteran
  • *****
  • Posts: 1037
    • View Profile
Re: compile_maps.bat improvements
« Reply #6 on: October 07, 2009, 09:31:21 pm »
@Destructavator:
It's not that hard with windows: open a makefile, select a program to always open it with (ie. make.exe) and further on you'll dclick maps.mk instead of compile_maps.bat.

@Mattn:
I don't mind using make. But the current trunk of maps.mk doesn't seem to work for me. It doesn't find *.map.
I found 2 slashes where it should be backslashes under Windows, but that didn't help.
Which dir am I supposed to run it from ??

My point was to have *two* ufo2map running. Will make do that for me ?

Offline Mattn

  • Administrator
  • PHALANX Commander
  • *****
  • Posts: 4831
  • https://github.com/mgerhardy/vengi
    • View Profile
    • Vengi Voxel Tools
Re: compile_maps.bat improvements
« Reply #7 on: October 07, 2009, 10:30:14 pm »
yes - it will

Code: [Select]
make maps -j 2
you have to run it from within trunk/

you can first try to run the mingw shell (this should really not be needed later, but as there wasn't done any work on this territory yet, this step is maybe still needed)

Offline Duke

  • Administrator
  • PHALANX veteran
  • *****
  • Posts: 1037
    • View Profile
Re: compile_maps.bat improvements
« Reply #8 on: October 07, 2009, 10:50:58 pm »
yes - it will

Code: [Select]
make maps -j 2
you have to run it from within trunk/
That was my first attempt;) Didn't work, even after replacing the slashes.

Quote
you can first try to run the mingw shell (this should really not be needed later, but as there wasn't done any work on this territory yet, this step is maybe still needed)
Uhmm...how do I run the mingw shell ? Didn't see any exe that sounded like that, sorry.

Offline Ralgert

  • Rookie
  • ***
  • Posts: 14
    • View Profile
Re: compile_maps.bat improvements
« Reply #9 on: October 08, 2009, 01:06:42 am »
you can first try to run the mingw shell

Thank you for this big tip. Now i can many batchfiles kick to the moon.

@Duke: \codeblocks\MinGW\msys.bat

Offline Kildor

  • Project Artist
  • Captain
  • ***
  • Posts: 757
  • Project mapper and some other stuff`er
    • View Profile
    • http://ufoai.nx0.ru
Re: compile_maps.bat improvements
« Reply #10 on: October 08, 2009, 03:46:01 am »
> all these things will be removed in the future anyway
that`s bad.

There is no fast and useable methods to compile one map or one directory under windows, and "compile" button from radiant doesn`t work too. When I edit maps, I need to easy recompiling method.

Offline Destructavator

  • Combination Multiple Specialty Developer
  • Administrator
  • PHALANX Commander
  • *****
  • Posts: 1908
  • Creater of Scorchcrafter, knows the zarakites...
    • View Profile
Re: compile_maps.bat improvements
« Reply #11 on: October 08, 2009, 05:28:11 am »
> all these things will be removed in the future anyway
that`s bad.

There is no fast and useable methods to compile one map or one directory under windows, and "compile" button from radiant doesn`t work too. When I edit maps, I need to easy recompiling method.

I agree.  Please don't throw out something that works, not to mention works well and works easy for Windows users.  Keep in mind that the vast majority of people out there that use computers use Windows, I've even seen many highly technical-"geek" type of people who know Linux well still use Windows for most things as their OS of choice.  I don't mean any offense when I say this, but making a change that favors Linux-users and leaves Windows-users behind is a bit of an insulting kick in the teeth that will turn off quite a few people, not to mention at the very least a good number of potential map-makers and other potential contributers for this game.


Offline Mattn

  • Administrator
  • PHALANX Commander
  • *****
  • Posts: 4831
  • https://github.com/mgerhardy/vengi
    • View Profile
    • Vengi Voxel Tools
Re: compile_maps.bat improvements
« Reply #12 on: October 08, 2009, 09:03:46 am »
i've only said that they will be removed if the alternative is as easy to use ;)

Offline Destructavator

  • Combination Multiple Specialty Developer
  • Administrator
  • PHALANX Commander
  • *****
  • Posts: 1908
  • Creater of Scorchcrafter, knows the zarakites...
    • View Profile
Re: compile_maps.bat improvements
« Reply #13 on: October 08, 2009, 09:05:13 am »
OK.  Sorry if I got a bit upset...   :P

Offline Duke

  • Administrator
  • PHALANX veteran
  • *****
  • Posts: 1037
    • View Profile
Re: compile_maps.bat improvements
« Reply #14 on: October 09, 2009, 02:43:25 am »
Thx, Ralgert :)
I fired up msys and got some results:

The first results were desasterous. Then I remembered that I shouldn't use the debug version of ufo2map :(

The first thing I noticed was that maps.mk doesn't seem to recognize the # of processors correctly, passing  -t 1 :(
So I gave -j 20. Result: 46 minutes.

I noticed that after some 30 mins, cpu went down from 100% and only a few maps were using up the last 15 mins. bunker and a_hangar were the last ones to finish even though they must have been started pretty early in the process.

Next, I forced -t 2 (by editing maps.mk) in order to reduce those last 15 mins. Result: 37 mins :)

Then I had the idea to start a_hangar asap, so I set -j 500. After some 10 mins, there were only a handful of processes running. After some 15 mins only one of them remained. Guess which. It ended after 22 mins.

Next, I forced -t 4. Result: 19 mins
I guess that sets up a new world record for compiling the UFO maps :)

Next try: -t 8 (the true # of 'processors' on my system. Result: 32 mins.
Seems to be too much overhead here.
I also got a 'memory allocation failure' here.

As a comparison: the compile_maps.bat takes some 50 mins on my system.

Management summary:
1. Destructavator: if you want to save time, use msys->make ;)
2. We need to figure out why the makefile doesn't detect the # of processors
3. never ever use the debug version for a recompile maps ;)
4. We should figure out why a_hangar and bunker maps take sooo long to compile.
5. This was on a Vista system. Needs to be verified for others.

Sidenote:
- I saw a lot of "missing carpet00n texture" msgs

I hope my findings will also help those who don't have the privilege to own a 4 core/8 processor computer to optimize their compile process.