project-navigation
Personal tools

Author Topic: compile_maps.bat improvements  (Read 20181 times)

Offline Kildor

  • Project Artist
  • Captain
  • ***
  • Posts: 757
  • Project mapper and some other stuff`er
    • View Profile
    • http://ufoai.nx0.ru
Re: compile_maps.bat improvements
« Reply #15 on: October 09, 2009, 03:25:36 am »
> why a_hangar and bunker maps take sooo long to compile.
I know nothing about bunker, but a_hangar has *many* glowing surfaces. If you up verbose of ufo2map, you can see, the main time ufo2map spends on Lightning calculation.

Offline Muton

  • Sergeant
  • *****
  • Posts: 496
    • View Profile

Offline Duke

  • Administrator
  • PHALANX veteran
  • *****
  • Posts: 1037
    • View Profile
Re: compile_maps.bat improvements
« Reply #17 on: October 09, 2009, 09:06:36 pm »
Thx, Muton, very interesting.
Your findings differ a bit from mine. As I wrote above, for me -t4 was better than -t2, which was better than -t1.
Any idea why ?

I didn't use -O3 or -march=core2.
btw as there is no -march=i7, do you expect core2 to be better than nothing ?

Offline Muton

  • Sergeant
  • *****
  • Posts: 496
    • View Profile
Re: compile_maps.bat improvements
« Reply #18 on: October 11, 2009, 07:36:41 am »
> Thx, Muton, very interesting.
> Your findings differ a bit from mine. As I wrote above, for me -t4 was better than -t2, which was better than -t1.
> Any idea why ?

You are missleaded
If you call ufo2map with -t 1 and generate another ufo2map also with -t 1 .....
until you run out of cores (So you run n ufo2map process simultaneously)
You are faster than using -t n, If you set the affinitymask correctly (which my script does)
btw: There is no magic thats normal

> I didn't use -O3 or -march=core2.
> btw as there is no -march=i7, do you expect core2 to be better than nothing ?

You own a Core7  :P
Yes there is no core7 in gcc but the important thing is SSE
If you use SSE you can remove option -ffloat-store
Just use core2duo optimation
btw. There is a option for SSE4.1
« Last Edit: October 11, 2009, 07:39:05 am by Muton »

Offline Borsti67

  • Squad Leader
  • ****
  • Posts: 164
    • View Profile
Re: compile_maps.bat improvements
« Reply #19 on: October 11, 2009, 12:40:10 pm »
Hm, the next time I need a full recompilation I'll try -t 1: yesterday I did w/o parameters (2 cores correctly found) and it took about 5 hours, leaving the PC nearly unusable most of the time. :( Especially when alienb/hangar was compiled, I couldn't even move the mouse! I never noticed such a severe impact before, were there changes in the process itself?

Offline Muton

  • Sergeant
  • *****
  • Posts: 496
    • View Profile
Re: compile_maps.bat improvements
« Reply #20 on: October 11, 2009, 09:19:04 pm »
set ufo2map priority to low
or just use my script, if you want

Offline Borsti67

  • Squad Leader
  • ****
  • Posts: 164
    • View Profile
Re: compile_maps.bat improvements
« Reply #21 on: October 12, 2009, 01:00:53 pm »
priority on windoze? :D
I'll check out your script, thanks.

Offline Duke

  • Administrator
  • PHALANX veteran
  • *****
  • Posts: 1037
    • View Profile
Re: compile_maps.bat improvements
« Reply #22 on: October 13, 2009, 02:01:19 am »
Muton,
I just compiled a_hangar.map without optimization, -o3 and -march=core2. The results are 73s, 56s and 50s, respectively. Thx for your hints :)

But I really doubt that running just as many processes as you have processors can be the optimum. Because ufo2map has a single-threaded part that doesn't use much cpu. With or without affinity mask: if you don't have additional waiting processes to fill that gap, the cpu will be unused in that period.

So you may want to increase the # of processes in your script and see what happens ;)

Offline Muton

  • Sergeant
  • *****
  • Posts: 496
    • View Profile
Re: compile_maps.bat improvements
« Reply #23 on: October 13, 2009, 07:29:02 pm »
I'm shure it will take longer
The problem is the cache must be filled with data from process|thread a <-> b
which is very time consuming.
And this for only a fiew cycles ...

Even if you dont set the affinity mask for a ufo2map process the compile time will increase
'couse windows (xp--) reaisign threads from core n to core x and vice versa all the time
which is very time consuming.
They've made some improvements on win7 in that point ...

On a core7 this idle times are'nt a real problem
'couse a core7 is dynamically overclocked if other cores are at "idle"
Its not realy that easy and yes there are some improvements on win7

No i'm not a friend of win7

Offline Duke

  • Administrator
  • PHALANX veteran
  • *****
  • Posts: 1037
    • View Profile
Re: compile_maps.bat improvements
« Reply #24 on: November 30, 2009, 04:54:08 pm »
A little update on compile times:

Using an ufo2map compiled with release, -o3 and -core2.
Using msys, maps.mk forced to -t 4, -j 50.
(my old approach with -j 500 ran into memory alloc probs, so I had to reduce -j)
Result: 26 minutes on my machine (i7).

Quite a lot of new big maps have been added recently, so the result seems reasonable.

Another interesting result: I deleted alienb/a_hangar, bunker and england.bsp and ran make again. Those three maps alone took 6:48 to complete.

Offline geever

  • Project Coder
  • PHALANX Commander
  • ***
  • Posts: 2561
    • View Profile
Re: compile_maps.bat improvements
« Reply #25 on: November 30, 2009, 06:47:31 pm »
A little update on compile times:

Using an ufo2map compiled with release, -o3 and -core2.
Using msys, maps.mk forced to -t 4, -j 50.
(my old approach with -j 500 ran into memory alloc probs, so I had to reduce -j)
Result: 26 minutes on my machine (i7).

Quite a lot of new big maps have been added recently, so the result seems reasonable.

Another interesting result: I deleted alienb/a_hangar, bunker and england.bsp and ran make again. Those three maps alone took 6:48 to complete.

We have a list of Map compiling times on the wiki. We should update it regularly.

ps. -j 50 isn't a usual setting.... :)

-geever

Offline Duke

  • Administrator
  • PHALANX veteran
  • *****
  • Posts: 1037
    • View Profile
Re: compile_maps.bat improvements
« Reply #26 on: November 30, 2009, 09:34:56 pm »
Tried again with -j 5, but it didn't really make a difference. The idea behind -j 500 was this: because of -t4, the last job running can only make use of half of the 8 'processors' of the i7. So I wanted the 'big maps' to be started asap.

I also updated the table you linked to.

Currently running compile_maps.bat for a comparison ;)

Offline Muton

  • Sergeant
  • *****
  • Posts: 496
    • View Profile
Re: compile_maps.bat improvements
« Reply #27 on: November 30, 2009, 10:41:26 pm »
The core7 own HT again right?
can you test it with my map compile script too
http://ufoai.ninex.info/forum/index.php?action=dlattach;topic=3583.0;attach=1302

/clean /1t4core

Offline Duke

  • Administrator
  • PHALANX veteran
  • *****
  • Posts: 1037
    • View Profile
Re: compile_maps.bat improvements
« Reply #28 on: November 30, 2009, 10:47:53 pm »
Where do I have to unzip it to ?

Offline Duke

  • Administrator
  • PHALANX veteran
  • *****
  • Posts: 1037
    • View Profile
Re: compile_maps.bat improvements
« Reply #29 on: November 30, 2009, 10:59:17 pm »
Maybe I should just look into the zip before asking ;)
Found it.