Reducing size of music in svn by reconverting to VBR without loosing quality !!!

[MCR]

I did some experiments regarding the soundtrack of the game:
@ present all files are encoded using ogg, but many of them used different bitrates when encoding.
Most of the files are encoded using CBR (constant bit rate).
During some tests I managed to reduce the filesize needed massively, without reducing the quality.

Here one example:
Reencoding the music from CBR to VBR, when using ABR-quality -1, that means reencoding files with 128 CBR to 112 VBR for example, with a upper limit of 160 VBR reduced the filesize for the music from 219 MB to 189 MB (!).
But we could do even better reduction, when we define the upper limit with 128 VBR.

Reducing the filesize of the tracks would also help in reducing loading-speed a little bit & would help us to optimize the way content of the game is stored.
Backups of the original encodings should be made in case the original is needed again.

If developers want I can optimize the reencoding & prepare a patch.

Do not misunderstand me: I do not want to trade quality against size here, but keep the quality as high as possible, but the filesize reasonable & optimized...
Why should we use 300Kilobit per second for silence ?

Additional Info (2010-04-16): When I am talking about upper limits here, I am meaning the upper limits for the average bitrate, not for the highest bitrate, so upper limit of 128 VBR means:

32 KBit Minimum, 128 KBit Average, 224 KBit Maximum

[FYI, to avoid misunderstandings]

[MCR]

I posted this on the patchtracker also, because I think it would be a really good way to cut the size needed by UFO:AI. 
Afterwards we could take a look @ the other samples used in-game... 

[MCR]

Update:

Gren is helping with the soundsamples & has recoded most of them already & also added new & 'fresh' sounds to the database.
Unfortunately he got the information a little late, that some of those (over 30) are of unknown source & should be replaced completely, but I hope he still finds the motivation to continue on the task of reorganization as imho I feel that he is the right man for our samples & soundfx !

I had a very positive discussion with Destructavator yesterday & I think we finally came to good conclusions how to best do this stuff:

1.) Backups of the original tracks will be made first, so we can always get back to the originals...

2.) Tracks already encoded @ a low or reasonable VBR bitrate will not be recoded...

3.) Destrucatavator will try & test this settings for the rest of the tracks:
                                                OGG VBR: min. Bitrate 32KBit, average Bitrate 128KBit, max. Bitrate 224KBit

This specification should also be the one we should want to get from the music contributors, or we decide to have a (big) place for our original tracks somewhere & do the ogg encoding ourselves, then I would suggest to ask the contributors for files encoded with some lossless codec like flac...

An extensive test on different methods of encoding & their qualities can be found here:
http://www.hydrogenaudio.org/forums/index.php?showtopic=36465

Interesting to note is that the free ogg-codec in the latest versions is the best of all modern lossy formats 

Here some quotes:
• Vorbis: Vorbis is now –thanks to Aoyumi– an excellent audio format for 180 kbps encodings (and classical music). It has apparently no flaws (lowest note was 3.5) but only minor problems audible from time to time: additional noise, fatness, or softened details. One third of the tested samples are transparent to my ears with aoTuV encodings. My own limits are close to be reached. Vorbis also gets 14 times (on 18) the first place. Those amazing performance can be compared to MPC which is transparent only once and get the first place four times (on 18). Statistically aoTuV is still tied to MPC on those samples. But the ANOVA and the FRIEDMAN pval indicates that aoTuV is the winner with a high probability of 92% (ANOVA) or 93% (friedman). Bootstraped results are nevertheless less favorable (pval=0,264) and seems to indicate that additional samples won’t with certainty increase the gap (correct me if I’m wrong).
Anyway, Vorbis is now excellent.

• Vorbis: Vorbis is now impressive. Last year constant noise boost or coarseness spoiled the performance of this format. I was surprised to hear those problems on –q6,00 setting supposed to be free of them due to lossless coupling. Aoyumi has apparently identifies precisely the cause of this problem and he worked to solve it. Not entirely though: some remaining trails are still audible with few samples but the intensity is now really frail (at least on the tested samples). Other artifacts are also corrected: micro-attacks on Orion_II (sample 09) are now much less dusty (aoTuV performed even better than any other tested format at this bitrate!). Performance are remarkable and the slightly gain in bitrate consumption is the icing on the cake. Just a question: Does aoTuV mean Tuned for Victory? Or Tuned by Visitors? From what planet is Aoyumi coming from?

[Destructavator]

Actually, in talking further about this with Mattn, I found that point #1 isn't really necessary because the trunk is stored in SVN, which already backs up every version of every file ever committed - meaning the current versions of each file are already backed up, and can be reverted at any time.

I'd also like to point out for #3 that I'll try several rates for each track, I have a feeling some tracks may be able to be squeezed even smaller, but yes, those are the rates I'll be starting with.

I glanced at that link (Thanks, MCR), that test was extensive but apparently only done with classical music - I've worked with enough sound (programmed music/audio on computers since the 1980s) to know that mellow, classical music compresses smaller than other styles, and we have a variety of stuff in the soundtracks, although the link does look a bit interesting.  (More complex sounds, including what we use in some UFO: AI tracks, would need more room to preserve clarity and quality.  It varies.)

As it is right now, the tracks are all inconsistent, some going over a bit rate of nearly 500, which isn't very efficient in saving space...

[MCR]

 
Point 1. Already Done. Without doing anything. I think I am beginning to love the svn.
No need for backups ? As normal user you got to get used to thinkin' that way first...
Great. 

I gave you that link, not because of the classical music tested there  , but because it describes clearly which ogg encoder offers the best quality & why... 
I know that we have no classical music in-game 

Here you can download a easy-to-use encoder with built-in aoTuVb5.7 ogg Encoder (win32 version):

http://rarewares.org/ogg-oggdropxpd.php#oggdrop-aotuv

This is from Wikipedia:
Quote:
aoTuV is a modified version of the reference encoder created by a Japanese developer named Aoyumi (aoTuV stands for Aoyumi Tuned Vorbis), which improves its quality, especially at lower bitrates. Initially built upon Vorbis 1.0, the improvements of aoTuV Beta2 were merged back into Vorbis 1.1. Further improvements were released, the current Beta 5.7 version (released on March 3, 2009) is an improvement of libvorbis 1.2.1 RC2.
Quote end.

[Destructavator]

... Here you can download a easy-to-use encoder with built-in aoTuVb5.7 ogg Encoder (win32 version):

http://rarewares.org/ogg-oggdropxpd.php#oggdrop-aotuv

This is from Wikipedia:
Quote:
aoTuV is a modified version of the reference encoder created by a Japanese developer named Aoyumi (aoTuV stands for Aoyumi Tuned Vorbis), which improves its quality, especially at lower bitrates. Initially built upon Vorbis 1.0, the improvements of aoTuV Beta2 were merged back into Vorbis 1.1. Further improvements were released, the current Beta 5.7 version (released on March 3, 2009) is an improvement of libvorbis 1.2.1 RC2.
Quote end.

Thanks, but I've already got (a number of) my own encoders - As I said once before, I've worked with computers, music, and audio together since the 1980s, have followed DAWs since they first became available to the common home computer user, worked with a good many codecs (and variants of them), and now hand-code my own VST plug-ins in C++, so I've got plenty of tools to choose from already.  I also work with digital video and combining audio with it, although the video stuff I admit I don't have quite as much expertise in.

I've also seen plenty of other reviews and blind tests of different audio stuff, much like the link you gave earlier, but thanks anyways.

What I will need help with, though, is after I convert some stuff I could use a second opinion on whether or not the new, smaller file still sounds decent enough quality-wise.

Also, if someone else could work on the SFX while I work on the music tracks that would be nice.  (I know the SFX take less space already, but every little bit can help.)

[MCR]

What I will need help with, though, is after I convert some stuff I could use a second opinion on whether or not the new, smaller file still sounds decent enough quality-wise.

I am quite sure we will not make out a difference when using the suggested settings, at least I will not, but nevertheless I am not for squeezing the sound down until we hear the difference 
My idea here was just to take the most modern & best variation of ogg to make sure our encoding will have the highest quality for the money (our bitrate)...
I did not know until I researched it, that there are many different & enhanced variations of the ogg codec available & I did not know this enhanced & tuned version of ogg, so I thought I will post this info here, that was in no way an offense...

Also, if someone else could work on the SFX while I work on the music tracks that would be nice.  (I know the SFX take less space already, but every little bit can help.)

Gren is doing a lot of work regarding the SFX already. He made new sounds as well as converting the old ones. We just have to doublecheck all the stuff as human errors happen when humans work...
I will try to review the 'new' SFX ASAP...
There are also issues, because more than 30 of the SFX sounds are from unknown origin & have to be replaced completely with new sounds, so there is lots of stuff happening on this field already...

For more information regarding the SFX look here:

http://ufoai.ninex.info/forum/index.php?topic=4670.0

[Destructavator]

Alright, I started updating some music files in the trunk, I was actually able to bring the upper limit down to 160, although the target rate is still 128, and I think the tracks still sound good enough.

It's taking a while because I'm not blindly batch-converting them, I'm actually testing each one at different encoding levels one by one.

More will come later...

[MCR]

Good to read that & IMHO a talented musician like you is the best man we can get for this task...
Take your time, delivering quality always consumes time... 
I am glad to read that you also look into normalizing some of the tracks, as it is a very good idea not to have some tracks killing the speakers, while others will turn off the subwoofer because the level is so low...
Just joking here, this never was the case with the soundtrack in UFO:AI, but nevertheless optimization here can help somewhat also...

[Janglur]

This would also reduce lag on older/slower systems, since OGG is a very demanding format for something as weak as music.  Combined with the OpenGL, it uses an absolutely disproportionate amount of system resources.

Changing the music to VBR would reducing the audio glitching many people (such as myself) experience.