UFO:Alien Invasion
Development => Coding => Topic started by: Duke on October 07, 2009, 12:02:46 am
-
Last time when I had to recompile all the maps, I watched the system monitor. I noticed that there was quite a lot of time where the processors were not at 100% cpu usage.
As cpu seems to be the limiting factor, I started a second process with compile_maps, just for fun. And it worked nicely for a while. All the processors were permanently at a 100%. But of course sooner or later both processes ended up compiling the same map coz they are not protected against that in any way.
So the idea is to create some recompile_maps.bat that can spawn a second batchfile that handles n*.map through z*.map, while the recompile_maps.bat is restricted to a*.map to m*.map. We will lose some time for process concurrency overhead, but I have no idea how much that really is. Imho it's worth trying.
Does anyone of the batchfile-artists around want to give it a try ? You might end up in the Guinnes Book of Records for the fastest full compile of UFO maps ;)
-
I remember Muton was working on this at some point, he posted and attached some batch files and other utils meant to compile the maps quicker. They're old though, and might need to be updated (and dug out of the older forum posts).
-
Yes, but IIRC Muton's solution was rejected because we don't want additional utils in the project.
I was thinking of a batchfile-only solution.
-
he had a batch file only solution, too - autodetecting the cpus you have and so on... - why aren't you using the makefile?
-
- why aren't you using the makefile?
Probably because on Windows, getting a makefile to run can be a pain, and requires having things set up specially, extra software that isn't included with Windows, etc. Running a batch file, on the other hand, is laughably easy on Windows, just click on the file icon and it runs, no third party stuff needs to be installed, nothing special needs to be prepared ahead of time, runs with a fresh install of Windows right out-of-the-box.
This is very different than Linux, where the opposite can be true.
This is also very different than running Windows virtually in a shell or virtual environment, because some of those virtual environments on top of Linux - at least the ones I've seen - include capability to run makefiles within the shell for convenience. A native installation of Windows, on the other hand, does not run a makefile very easily (not at all out-of-the-box), and especially a non-technical user may find it beyond their ability or simply give up trying to make it work.
-
you would need codeblocks installed anyway - and make is included in that package, too ;)
but to make a long story short - all these things will be removed in the future anyway - only the makefile will be maintained but hopefully integrated into codeblocks or whatever ide we are using then.
-
@Destructavator:
It's not that hard with windows: open a makefile, select a program to always open it with (ie. make.exe) and further on you'll dclick maps.mk instead of compile_maps.bat.
@Mattn:
I don't mind using make. But the current trunk of maps.mk doesn't seem to work for me. It doesn't find *.map.
I found 2 slashes where it should be backslashes under Windows, but that didn't help.
Which dir am I supposed to run it from ??
My point was to have *two* ufo2map running. Will make do that for me ?
-
yes - it will
make maps -j 2
you have to run it from within trunk/
you can first try to run the mingw shell (this should really not be needed later, but as there wasn't done any work on this territory yet, this step is maybe still needed)
-
yes - it will
make maps -j 2
you have to run it from within trunk/
That was my first attempt;) Didn't work, even after replacing the slashes.
you can first try to run the mingw shell (this should really not be needed later, but as there wasn't done any work on this territory yet, this step is maybe still needed)
Uhmm...how do I run the mingw shell ? Didn't see any exe that sounded like that, sorry.
-
you can first try to run the mingw shell
Thank you for this big tip. Now i can many batchfiles kick to the moon.
@Duke: \codeblocks\MinGW\msys.bat
-
> all these things will be removed in the future anyway
that`s bad.
There is no fast and useable methods to compile one map or one directory under windows, and "compile" button from radiant doesn`t work too. When I edit maps, I need to easy recompiling method.
-
> all these things will be removed in the future anyway
that`s bad.
There is no fast and useable methods to compile one map or one directory under windows, and "compile" button from radiant doesn`t work too. When I edit maps, I need to easy recompiling method.
I agree. Please don't throw out something that works, not to mention works well and works easy for Windows users. Keep in mind that the vast majority of people out there that use computers use Windows, I've even seen many highly technical-"geek" type of people who know Linux well still use Windows for most things as their OS of choice. I don't mean any offense when I say this, but making a change that favors Linux-users and leaves Windows-users behind is a bit of an insulting kick in the teeth that will turn off quite a few people, not to mention at the very least a good number of potential map-makers and other potential contributers for this game.
-
i've only said that they will be removed if the alternative is as easy to use ;)
-
OK. Sorry if I got a bit upset... :P
-
Thx, Ralgert :)
I fired up msys and got some results:
The first results were desasterous. Then I remembered that I shouldn't use the debug version of ufo2map :(
The first thing I noticed was that maps.mk doesn't seem to recognize the # of processors correctly, passing -t 1 :(
So I gave -j 20. Result: 46 minutes.
I noticed that after some 30 mins, cpu went down from 100% and only a few maps were using up the last 15 mins. bunker and a_hangar were the last ones to finish even though they must have been started pretty early in the process.
Next, I forced -t 2 (by editing maps.mk) in order to reduce those last 15 mins. Result: 37 mins :)
Then I had the idea to start a_hangar asap, so I set -j 500. After some 10 mins, there were only a handful of processes running. After some 15 mins only one of them remained. Guess which. It ended after 22 mins.
Next, I forced -t 4. Result: 19 mins
I guess that sets up a new world record for compiling the UFO maps :)
Next try: -t 8 (the true # of 'processors' on my system. Result: 32 mins.
Seems to be too much overhead here.
I also got a 'memory allocation failure' here.
As a comparison: the compile_maps.bat takes some 50 mins on my system.
Management summary:
1. Destructavator: if you want to save time, use msys->make ;)
2. We need to figure out why the makefile doesn't detect the # of processors
3. never ever use the debug version for a recompile maps ;)
4. We should figure out why a_hangar and bunker maps take sooo long to compile.
5. This was on a Vista system. Needs to be verified for others.
Sidenote:
- I saw a lot of "missing carpet00n texture" msgs
I hope my findings will also help those who don't have the privilege to own a 4 core/8 processor computer to optimize their compile process.
-
> why a_hangar and bunker maps take sooo long to compile.
I know nothing about bunker, but a_hangar has *many* glowing surfaces. If you up verbose of ufo2map, you can see, the main time ufo2map spends on Lightning calculation.
-
I suggest reading this
http://ufoai.ninex.info/forum/index.php?topic=3604.0
and
http://ufoai.ninex.info/forum/index.php?topic=3583.0
-
Thx, Muton, very interesting.
Your findings differ a bit from mine. As I wrote above, for me -t4 was better than -t2, which was better than -t1.
Any idea why ?
I didn't use -O3 or -march=core2.
btw as there is no -march=i7, do you expect core2 to be better than nothing ?
-
> Thx, Muton, very interesting.
> Your findings differ a bit from mine. As I wrote above, for me -t4 was better than -t2, which was better than -t1.
> Any idea why ?
You are missleaded
If you call ufo2map with -t 1 and generate another ufo2map also with -t 1 .....
until you run out of cores (So you run n ufo2map process simultaneously)
You are faster than using -t n, If you set the affinitymask correctly (which my script does)
btw: There is no magic thats normal
> I didn't use -O3 or -march=core2.
> btw as there is no -march=i7, do you expect core2 to be better than nothing ?
You own a Core7 :P
Yes there is no core7 in gcc but the important thing is SSE
If you use SSE you can remove option -ffloat-store
Just use core2duo optimation
btw. There is a option for SSE4.1
-
Hm, the next time I need a full recompilation I'll try -t 1: yesterday I did w/o parameters (2 cores correctly found) and it took about 5 hours, leaving the PC nearly unusable most of the time. :( Especially when alienb/hangar was compiled, I couldn't even move the mouse! I never noticed such a severe impact before, were there changes in the process itself?
-
set ufo2map priority to low
or just use my script, if you want
-
priority on windoze? :D
I'll check out your script, thanks.
-
Muton,
I just compiled a_hangar.map without optimization, -o3 and -march=core2. The results are 73s, 56s and 50s, respectively. Thx for your hints :)
But I really doubt that running just as many processes as you have processors can be the optimum. Because ufo2map has a single-threaded part that doesn't use much cpu. With or without affinity mask: if you don't have additional waiting processes to fill that gap, the cpu will be unused in that period.
So you may want to increase the # of processes in your script and see what happens ;)
-
I'm shure it will take longer
The problem is the cache must be filled with data from process|thread a <-> b
which is very time consuming.
And this for only a fiew cycles ...
Even if you dont set the affinity mask for a ufo2map process the compile time will increase
'couse windows (xp--) reaisign threads from core n to core x and vice versa all the time
which is very time consuming.
They've made some improvements on win7 in that point ...
On a core7 this idle times are'nt a real problem
'couse a core7 is dynamically overclocked if other cores are at "idle"
Its not realy that easy and yes there are some improvements on win7
No i'm not a friend of win7
-
A little update on compile times:
Using an ufo2map compiled with release, -o3 and -core2.
Using msys, maps.mk forced to -t 4, -j 50.
(my old approach with -j 500 ran into memory alloc probs, so I had to reduce -j)
Result: 26 minutes on my machine (i7).
Quite a lot of new big maps have been added recently, so the result seems reasonable.
Another interesting result: I deleted alienb/a_hangar, bunker and england.bsp and ran make again. Those three maps alone took 6:48 to complete.
-
A little update on compile times:
Using an ufo2map compiled with release, -o3 and -core2.
Using msys, maps.mk forced to -t 4, -j 50.
(my old approach with -j 500 ran into memory alloc probs, so I had to reduce -j)
Result: 26 minutes on my machine (i7).
Quite a lot of new big maps have been added recently, so the result seems reasonable.
Another interesting result: I deleted alienb/a_hangar, bunker and england.bsp and ran make again. Those three maps alone took 6:48 to complete.
We have a list of Map compiling times (http://ufoai.ninex.info/wiki/index.php/Mapping/Compile_time) on the wiki. We should update it regularly.
ps. -j 50 isn't a usual setting.... :)
-geever
-
Tried again with -j 5, but it didn't really make a difference. The idea behind -j 500 was this: because of -t4, the last job running can only make use of half of the 8 'processors' of the i7. So I wanted the 'big maps' to be started asap.
I also updated the table you linked to.
Currently running compile_maps.bat for a comparison ;)
-
The core7 own HT again right?
can you test it with my map compile script too
http://ufoai.ninex.info/forum/index.php?action=dlattach;topic=3583.0;attach=1302
/clean /1t4core
-
Where do I have to unzip it to ?
-
Maybe I should just look into the zip before asking ;)
Found it.
-
Built diff and affinity, then got this:
C:\UFO>\ufo\contrib\scripts\compile_maps_Muton /clean /1t4core
C:\Users\me\AppData\Local\Temp\ufo2map_core*.* konnte nicht gefunden werden
Its seems this is not the root folder of Ufoai
-
I've updated the script
http://ufoai.ninex.info/forum/index.php?action=dlattach;topic=3583.0;attach=2549
-
Same result, but it's probably nor your fault.
I figured out that for some unknow reason (Vista?) the %tmp% var is set to "C:\Users...", but the dir is actually named C:\Benutzer....
I changed your script to set tmp explicitely, but only to get:
C:\UFO>cd /d C:\UFO\contrib\scripts\
Das System kann den angegebenen Pfad nicht finden.
Its seems this is not the root folder of Ufoai
Looks like there is more than one problem to solve before this runs on my machine.
Well, be both are convinced that out method is better than the other's method and we both have a good reasoning. I at least gave your method a try. Guess now it's your turn to try mine ;)
-
the %tmp% var is ok
this code troubles you
if /I NOT "%this%" == "ufoai\" (
@echo Its seems this is not the root folder of Ufoai
pause
exit /b 1
)
It seems you stored your source under c:\ufo
and not c:\ufoai
I've replaced it with some reliable code
if NOT EXIST "%UfoAIroot%INSTALL" (
@echo Its seems this is not the root folder of Ufoai
@echo Because im unable to find the file %UfoAIroot%INSTALL
pause
exit /b 1
)
http://ufoai.ninex.info/forum/index.php?action=dlattach;topic=3583.0;attach=2550
What is this option -j for
im unable to find information in the help output
btw i have some troubles on my machine using -o3 or -o2 on ufo2map
C::B reports some warnings on both optimations
and ufo2map crash (lightning) on my machine if i run it with -t 2 (-t 1 generate no errors during "maping")
only -o1 is working during "maping" and there are no warnings by C::B
I've just copmpleted a testrun
ufo2map compiler options
<Compiler>
<Add option="-march=k8" />
<Add option="-O1" />
<Add option="-Wall" />
<Add option="-msse3" />
<Add option="-mfpmath=sse" />
<Add option="-mieee-fp" />
<Add option="-D__GNUWIN32__" />
<Add option="-DWINVER=0x501" />
<Add option="-DNODEBUG" />
</Compiler>
took using -t 2
02:09:00
and using my script
02:00:34
yesterday i've compiled with this options
using my script
01:37:43
using -t 2 crash ufo2map during "maping"
<Compiler>
<Add option="-march=k8" />
<Add option="-O3" />
<Add option="-fexpensive-optimizations" />
<Add option="-Wall" />
<Add option="-msse3" />
<Add option="-mfpmath=sse" />
<Add option="-mieee-fp" />
<Add option="-D__GNUWIN32__" />
<Add option="-DWINVER=0x501" />
<Add option="-DNODEBUG" />
</Compiler>
-
It seems you stored your source under c:\ufo
and not c:\ufoai
Correct.
What is this option -j for
im unable to find information in the help output
-j is the number of make-jobs to start.
btw i have some troubles on my machine using -o3 or -o2 on ufo2map
C::B reports some warnings on both optimations
Yes. Same here. But works.
Ok, I tried your update. It managed to delete the bsps, but then got into trouble:
475 maps to go
current C:\UFO\base\maps\bunker.map bunker.bsp
ufo2map.exe -v 4 -extra -t 1 maps\bunker.map
"\Acer\Empowering" kann syntaktisch an dieser Stelle nicht verarbeitet werden.
Why does try to scan the programs dir ?
And WTH does it not react if I set 'echo on' in line 2 ??
-
what are those warnings? i don't get any
-
ufo2map.exe -v 4 -extra -t 1 maps\bunker.map
"\Acer\Empowering" kann syntaktisch an dieser Stelle nicht verarbeitet werden.
Why does try to scan the programs dir ?
And WTH does it not react if I set 'echo on' in line 2 ??
Why does try to scan the programs dir ?
I dont
the script is somwhere at
if "%_1t4core%" == "true" (
you can add a pause after
call :ufo2map_check %%a
@echo ufo2map_check %%a
pause
or
call :ufo2mapcmd "%tmp%\ufo2map_core%%a.cmd" %%a
@echo ufo2mapcmd "%tmp%\ufo2map_core%%a.cmd" %%a
pause
And WTH does it not react if I set 'echo on' in line 2 ??
turn it off this way
:: @echo off
or
rem @echo off
=================================
<Compiler>
<Add option="-march=k8" />
<Add option="-O3" />
<Add option="-fexpensive-optimizations" />
<Add option="-Wall" />
<Add option="-msse3" />
<Add option="-mfpmath=sse" />
<Add option="-mieee-fp" />
<Add option="-D__GNUWIN32__" />
<Add option="-DWINVER=0x501" />
<Add option="-DNODEBUG" />
</Compiler>
-------------- Clean: windows in ufo2map ---------------
Cleaned "ufo2map - windows"
-------------- Build: windows in ufo2map ---------------
Compiling: ..\..\src\tools\ufo2map\common\bspfile.c
Compiling: ..\..\src\common\ioapi.c
Compiling: ..\..\src\common\mem.c
Compiling: ..\..\src\common\routing.c
Compiling: ..\..\src\common\tracing.c
Compiling: ..\..\src\common\unzip.c
V:\MinGW\ufoai\src\common\unzip.c: In function 'unzlocal_getShort':
V:\MinGW\ufoai\src\common\unzip.c:205: warning: 'i' may be used uninitialized in this function
V:\MinGW\ufoai\src\common\unzip.c: In function 'unzlocal_getLong':
V:\MinGW\ufoai\src\common\unzip.c:233: warning: 'i' may be used uninitialized in this function
Compiling: ..\..\src\ports\windows\win_shared.c
Compiling: ..\..\src\shared\byte.c
Compiling: ..\..\src\shared\entitiesdef.c
Compiling: ..\..\src\shared\mathlib.c
Compiling: ..\..\src\shared\parse.c
Compiling: ..\..\src\shared\shared.c
Compiling: ..\..\src\tools\ufo2map\brushbsp.c
Compiling: ..\..\src\tools\ufo2map\bsp.c
Compiling: ..\..\src\tools\ufo2map\check\check.c
Compiling: ..\..\src\tools\ufo2map\check\checkentities.c
Compiling: ..\..\src\tools\ufo2map\check\checklib.c
Compiling: ..\..\src\tools\ufo2map\common\aselib.c
Compiling: ..\..\src\common\files.c
Compiling: ..\..\src\tools\ufo2map\common\imagelib.c
V:\MinGW\ufoai\src\tools\ufo2map\common\imagelib.c: In function 'LoadTGA':
V:\MinGW\ufoai\src\tools\ufo2map\common\imagelib.c:74: warning: dereferencing type-punned pointer will break strict-aliasing rules
V:\MinGW\ufoai\src\tools\ufo2map\common\imagelib.c:78: warning: dereferencing type-punned pointer will break strict-aliasing rules
Compiling: ..\..\src\tools\ufo2map\common\polylib.c
Compiling: ..\..\src\tools\ufo2map\common\scriplib.c
Compiling: ..\..\src\tools\ufo2map\common\trace.c
Compiling: ..\..\src\tools\ufo2map\csg.c
Compiling: ..\..\src\tools\ufo2map\faces.c
Compiling: ..\..\src\tools\ufo2map\levels.c
Compiling: ..\..\src\tools\ufo2map\lighting.c
Compiling: ..\..\src\tools\ufo2map\lightmap.c
Compiling: ..\..\src\tools\ufo2map\map.c
Compiling: ..\..\src\tools\ufo2map\patches.c
Compiling: ..\..\src\tools\ufo2map\portals.c
Compiling: ..\..\src\tools\ufo2map\routing.c
Compiling: ..\..\src\tools\ufo2map\textures.c
Compiling: ..\..\src\tools\ufo2map\threads.c
Compiling: ..\..\src\tools\ufo2map\tree.c
Compiling: ..\..\src\tools\ufo2map\ufo2map.c
Compiling: ..\..\src\tools\ufo2map\writebsp.c
Linking console executable: ..\..\ufo2map.exe
Output size is 288,00 KB
Process terminated with status 0 (0 minutes, 44 seconds)
0 errors, 4 warnings
-------------------------
on -o2 NO -fexpensive-optimizations
....
V:\MinGW\ufoai\src\tools\ufo2map\common\imagelib.c: In function 'LoadTGA':
V:\MinGW\ufoai\src\tools\ufo2map\common\imagelib.c:74: warning: dereferencing type-punned pointer will break strict-aliasing rules
V:\MinGW\ufoai\src\tools\ufo2map\common\imagelib.c:78: warning: dereferencing type-punned pointer will break strict-aliasing rules
....
-------------------------
on -o1 -fexpensive-optimizations
no warning
-
Warnings confirmed.
-
ok, :: worked.
It seems to have a problem with ";C:\Program Files (x86)\Acer\Empowering Techno" when trying to set the path.
-
oh microsoft ;(
http://ufoai.ninex.info/forum/index.php?action=dlattach;topic=3583.0;attach=2551
added
a short algo to replace path dirs with 8.3 naming
set path=%path%;%this%;
set this=
:pathloop
for /F "tokens=1* delims=;" %%a in ("%path%") do (
set path=%%b
if exist "%%a" (
set this=%%~dpsnxa;%this%
)
)
if NOT "%path%" == "" goto pathloop
set path=%this%
set this=
-
Congrats, that did it :)
Recompile maps is currently running....
-
finished
started at 23:05:07,34
finished at 23:54:02,46
There is something wrong with the starting time. My stopwatch said it took 30m13s.
It started as a tight race. CPU was at 100% much more often than I thought. That's because you start with the big 'static' maps in \maps. Then a little slowdown for \maps\africa, but not much. 100% with the big alienb tiles again.
When we come to the RMA maps with the tiny tiles (eg. farm), cpu drops to below 80% sometimes.
At 26:30, you're running out of maps to compile.
Around 27:30, alienb\a_hangar is the last map still running (on one puny core, no, half of a core) for the remaining time.
I think the script could do better if the maps were sorted by size, running the big ones first, but I still believe you can't afford the idle times.
EDITED: typos
-
Hi Muton,
oh microsoft ;(
added
a short algo to replace path dirs with 8.3 naming
This had still issues with path elements containing <"> (whyever, one of the MS-XML-Patches did so).
I removed the quote signs from the path, afterwards I got one step further, but...
Its seems this is not the root folder of Ufoai
Because im unable to find the file C:\PROGRA~1\Spiele\UFOAI-~1\INSTALL
A "dir /x" reveals here is no "ufoai-~1":
01.01.2009 16:16 <DIR> UFOAI-~1.1 UFOAI-2.2.1
06.12.2009 14:48 <DIR> UFOAI-~1.3_D ufoai-2.3_dev
Is this a problem specific to my installation (German version of XP) or is there more room for improvements to the path detection routine? ;)
€dit: Before the "pathloop" the variable %this% is still correctly set to UFOAI-~1.3_D...?!
€²: After setting UfoAIroot manually in the script, it continues to the next error - in my case, code::blocks is not found via path, so "affinity" could not be compiled. I also "hardcoded" the path into the script, now it finally starts... Anyway a lot of messages scroll through:
find: /I: no medium
find: /V: No such file or directory
find: /I: no medium
find: /V: No such file or directory
find: prefabs: No such file or directory
48 map/s to compileOnly 48? Seems not enough to me...?
Afterwards error messages pop up, stating e.g. "file .svn/.map doesn't exist" or "file .sv/prop-base/UFOAIBot.pl doesn't exist"... Seemingly always 2 per map to compile.
Would be nice if a complete compilation would only take about 5 mins (including clicking away the error windows), but I can't believe it. :-P
-
I've updated the script the last time
http://ufoai.ninex.info/forum/index.php?action=dlattach;topic=3583.0;attach=2558
>This had still issues with path elements containing <"> (whyever, one of the MS-XML-Patches did so).
In that case i give up :P
> find: /I: no medium
Because of my last modification a query is broken
find had to be replaced with findstr
something i realy do not understand
Somethimes i think cmd.exe has its own (female)AI
(fixed in uploaded one)
> UFOAI-~1.3_D
> Is this a problem specific to my installation (German version of XP) or is there more room for improvements to the path detection routine?
The 8.3 name has an extension
set UfoAIroot=%%~dpsna\
replaced by
set UfoAIroot=%%~dpsnxa\
should fix it (fixed in uploaded one)
And the bad thing is
on my system not all maps have been build
3-4 where missed
( NOT fixed in uploaded one)
I dont want to put more effort into it
because i have another fine working script
that im currently working on.
http://ufoai.ninex.info/forum/index.php?topic=3843.0
-
Thanks for you efforts, Muton. Obviously it's better not to follow this one any longer. ;)
Is the EXE-file linked in 1st post still valid, or is there already a new one?
If I understand correctly, nothing else is needed (or comes from SVN)?
-
Is the EXE-file linked in 1st post still valid, or is there already a new one?
If I understand correctly, nothing else is needed (or comes from SVN)?
The exe has changed a little bit
Alone its not more than a preview
During the testrun the focus is on the buildscript
if its working as expected i'll do the next step.
Definitely the script will lack the "UfoAI from Scratch" function
I currently dont know how to handle codeblocks.sh
-
okido, I'll wait until you need testers for the new one. :)