Official Debian release for UFO:AI

[apo]

Hi everyone,

I am a member of the Debian Games Team and I am working on an official release of UFO:AI for Debian and its derivatives. So far i'm trying to get the bigger picture by figuring out in how many packages the game has to be split and if there are any license issues to consider.

You have done an extraordinary job with documenting the licenses of each and every file of the game. From Debian's point of view there exist only two issues which i think are easily solvable.

All files that are licensed under CC-BY-SA-2.0 and CC-BY-SA-2.5 are considered non-free by Debian but clause 4b) of CC-BY-SA-2.5 grants the right to upgrade the license to a later version while CC-BY-SA-3.0 is considered to be a DFSG-free license. CC-BY-2.5 is even less restrictive (no copyleft) and upgrading to CC-BY-3.0 should be no problem at all.

I have tried to compile the stable version 2.4 but the game failed to build from source with the latest packages from Debian unstable.

Code: [Select]

CamWnd.cpp:411: undefined reference to `CameraSettings::getMode() const'
However the latest development version compiles fine. I have built upon the already available Debian packaging work but upgraded almost all files to more recent technologies like source format 3.0, dh sequencer and so on.

My questions (so far):

I intend to split the game data in multiple source packages, depending on the frequency of updates, because the data is very large. Which data files change regularly and what files are more static?

UFO:AI seems to depend on many different fonts. What fonts are strictly needed to make the game run? The current unofficial Debian packaging pulls in almost 150 MB of font data.

Why do you build-depend on binutils-dev? The game appears running fine without binutils.

Do you provide a changelog somewhere?

I think i will find more questions in the future but that's it for now.

Thanks for reading this far and for developing UFO.

Markus

[Mattn]

Hi and welcome to the board.

Great to hear this - our main goal for 2.5 was to get it into debian - 2.4 still contains a lot of invalid licensed stuff. I will check the licensing of the files as you said.

Btw. please also send the updates for to the debian files upstream.

We are delieving the game in pk3 files (zip files) and already group them in: music, maps, videos, scripts and so on (run make pk3 to see the pk3 files in base/). You would either have to skip this for the debian packages or are somehow tied to the pk3 structure/grouping.

As for new versions and updates - the maps and the ufo scripts should at least be packaged alone. If we have minor releases, this is almost always about map bugs, script bug/rebalancing or code bugs. So datawise, base/ufos/ base/maps/**.bsp should get two packages on their own.

for releasing patches we normally just offer a new pk3 file - the game always loads the pk3 files in alphabetical order and only loads the first file from it. That means that a pk3 file 0ufos.pk3 which contains e.g. ui.ufo and ui2.ufo can be overridden by a 1ufos.pk3 which brings a patched version of ui2.ufo - so ui.ufo is still used from 0ufos.pk3, ui2.ufo is used from 1ufos.pk3. I'm not sure how to deal with this in the debian way. An option would be of course to just create a new 0ufos.pk3 and update the ufo-scripts.deb or whatever you will call it.

I think we will do a cleanup of translations before we release 2.5 - because some are outdated and not yet updated for 2.5. and the fonts are only needed for e.g. the chinese translation.

we depend on binutils-dev because we do backtraces on crashes and upload them to our server - if you don't use binutils-dev (libbfd) for building the game - the backtrace feature is disabled automatically.

we will create a changelog for the final 2.5 version. we only provide changelogs for every official release - and we provide monthly changelogs in the news for the current dev builds.

getting this into debian would be a huge step for us - so big thanks for offering us this opportunity. and if you have any other question - just ask and i will do my best to answer it.

[Mattn]

oh, btw. do you also plan to include uforadiant into debian? if yes, that would be awesome. but it would be even cooler to also get picomodel packages then - https://github.com/ufoai/picomodel

uforadiant is using this lib to load the models. it's also included in our codebase, but it should imo go into debian as well if you package uforadiant.

[apo]

Hi Mattn,

Great to hear this - our main goal for 2.5 was to get it into debian - 2.4 still contains a lot of invalid licensed stuff. I will check the licensing of the files as you said.

That would be great. If http://ufoai.org/licenses/ is up-to-date then there are no blockers anymore for an inclusion in Debian main. However i discovered license.txt in base/textures and it claims that there are still textures of unknown origin. If this file is outdated and all other information correct, i see no reason at the moment why UFO:AI can't be included in Debian.

Btw. please also send the updates for to the debian files upstream.

I will always send all patches upstream as long as they are of general interest for others, i promise. Though please consider to remove the debian directory in your source packages as soon as an official version exist. Keeping it in your git repository is a sane default, though. I will create an initial git repository for the Debian package soon and everyone can follow the progress there.

Some best practices for upstreams are described here:

https://wiki.debian.org/UpstreamGuide

I think we will do a cleanup of translations before we release 2.5 - because some are outdated and not yet updated for 2.5. and the fonts are only needed for e.g. the chinese translation.

Ok, i only need to know what fonts are essential to make the game work. I think it is sane to suggest all other ones (e.g. the chinese translation). I assume the dejavu- and free fonts are mandatory?

we depend on binutils-dev because we do backtraces on crashes and upload them to our server - if you don't use binutils-dev (libbfd) for building the game - the backtrace feature is disabled automatically.

Allright, then i will omit binutils-dev and aim for an optimized release version only.

we will create a changelog for the final 2.5 version. we only provide changelogs for every official release - and we provide monthly changelogs in the news for the current dev builds.

I saw your really nice monthly updates on the frontpage. I am only asking because it would be nice to include some kind of text file in the official release that documents all changes since the last version. Another option would be to convert the git commits to a changelog file. That's nice to have but not a big issue.

oh, btw. do you also plan to include uforadiant into debian? if yes, that would be awesome. but it would be even cooler to also get picomodel packages then - https://github.com/ufoai/picomodel

Yes, I intend to package uforadiant as well. I don't know much about picomodel yet but if you think it is a valuable tool to enhance the game or to make developing of UFO:AI easier, I'm absolutely open for packaging it.

Here are my current thoughts how to split the game in multiple packages for ease of use.

Source package: ufoai

It builds those binary packages:

ufoai (game client)
ufoai-server (dedicated server)
ufoai-uforadiant (mapping tool)
ufoai-tools (ufo2map, ufomodel, ufoslicer)

ufoai-common (architecture-independent files which are shared between server and client, e.g. language files, documentation)
libufoai-game (the game.so shared library)

I am not sure about the last two binary packages. I only saw that a shared library called game.so is built and as far as I understand, it is required by both, client and server. Hence I think it would make sense to package the library in a single binary package and let client and server depend on it.

Game data:

I haven't found any information about required data packages for client and server yet. I presume the client will need all files under /base in the git repository to run correctly. My goal is to create a ufoai-server binary package that is similar to the existing openarena-server package in Debian. That means you can easily run the server with the help of an init script. Does the server require all data packages as well?

I intend to keep the .pk3 structure but i would like to build as much as possible from source. I had the following structure in mind:

Source package: ufoai-maps
It builds the binary package ufoai-maps.

All maps need to be compiled from source and they are quite large so i think they deserve a separate source package.

Source package: ufoai-music

It builds the binary package ufoai-music

Very simple package that only installs the music files to the right place. I would make ufoai-music a recommendation but not a strictly required dependency of ufoai.

Source package: ufoai-misc

It builds the binary package ufoai-misc.

Basically that would be everything else that is architecture-independent like textures, videos, scripts, pics, sounds and so on.

The reasoning behind the splitting is that nobody has to upload a 900 MB data package to the mirrors every time something changes.

Although the data is splitted in multiple source packages, it think it makes sense to keep the same pk3 files you are offering today. It was great, if we could use xz compression for the source and binary packages. Probably the zip format for pk3 files is mandatory, but i was thinking about compressing everything with xz (which saves a lot of bandwith) and to create zip compressed binary data packages in Debian's post-installation step. That means the user downloads xz compressed files and automatically runs a script that creates zip compressed pk3 files in the end.

Unfortunately the downside is that it might take several minutes to recompress the data on the user's pc. So obviously it would be better if we could ship xz compressed pk3 files right from the start. I assume that's easier said than done.

[Mattn]

That would be great. If http://ufoai.org/licenses/ is up-to-date then there are no blockers anymore for an inclusion in Debian main. However i discovered license.txt in base/textures and it claims that there are still textures of unknown origin. If this file is outdated and all other information correct, i see no reason at the moment why UFO:AI can't be included in Debian.

It's updated automatically by a cron job afair (long time since I touched this - but it's updated automatically in every case)

Ok, i only need to know what fonts are essential to make the game work. I think it is sane to suggest all other ones (e.g. the chinese translation). I assume the dejavu- and free fonts are mandatory?

DejaVuSans.ttf is the only font that's needed for

I saw your really nice monthly updates on the frontpage. I am only asking because it would be nice to include some kind of text file in the official release that documents all changes since the last version. Another option would be to convert the git commits to a changelog file. That's nice to have but not a big issue.

We will have such a list once 2.5 is officially released - when do you plan to publish the debian version? And you plan to do this on the 2.5 codebase, no? (just to be sure - 2.4 wouldn't work due to license restrictions - the license page is only valid for the master branch)

Yes, I intend to package uforadiant as well. I don't know much about picomodel yet but if you think it is a valuable tool to enhance the game or to make developing of UFO:AI easier, I'm absolutely open for packaging it.

that would be nice and i totally agree that it makes modding easier - and maybe we can attract some more mappers this way

Here are my current thoughts how to split the game in multiple packages for ease of use.
[...]

Makes sense - but you can also decide to hardlink the game.so (./configure --enable-hardlinkedgame) then libufoai-game would not be needed. And in case of a patch it's unlikely that the ufo or ufoded binaries are not updated, too. But this is up to you. And yes - game.so is loaded whereever the server runs - if it's a listen server, then the ufo loads the game.so - if it's a dedicated server, then ufoded loads the game.so

Game data:

I haven't found any information about required data packages for client and server yet. I presume the client will need all files under /base in the git repository to run correctly. My goal is to create a ufoai-server binary package that is similar to the existing openarena-server package in Debian. That means you can easily run the server with the help of an init script. Does the server require all data packages as well?

Well - to be executed the server does not need the music, pics, sounds and so on - but... the server can also be configured in a way to offer these files as downloads for clients that don't have one particular file. I don't think this will ever happen - but well... it's a feature. And new user created content would be available on the server anyway. But to be honest i've never tried to run the server on a minimal dataset.

I intend to keep the .pk3 structure but i would like to build as much as possible from source. I had the following structure in mind:

Source package: ufoai-maps
It builds the binary package ufoai-maps.

All maps need to be compiled from source and they are quite large so i think they deserve a separate source package.

Source package: ufoai-music

Please don't bulid the maps from source - please use those that we deliever when we release 2.5 officially. The problem here is floating point stuff. You would make cross plattform multiplayer stuff impossible because the checksum is different on different architectures. If you plan to release before we official release 2.5, please use the maps-sync script (make maps-sync)

Although the data is splitted in multiple source packages, it think it makes sense to keep the same pk3 files you are offering today. It was great, if we could use xz compression for the source and binary packages. Probably the zip format for pk3 files is mandatory, but i was thinking about compressing everything with xz (which saves a lot of bandwith) and to create zip compressed binary data packages in Debian's post-installation step. That means the user downloads xz compressed files and automatically runs a script that creates zip compressed pk3 files in the end.

Unfortunately the downside is that it might take several minutes to recompress the data on the user's pc. So obviously it would be better if we could ship xz compressed pk3 files right from the start. I assume that's easier said than done.

I think we once used 7zip to create the pk3 files and had no problems with it - so go for it. I don't think one has to convert them.


Martin

[apo]

Hi,

I have created an initial, unfinished, git repo for the UFO:AI packaging. http://anonscm.debian.org/gitweb/?p=pkg-games/ufoai.git

At the moment it covers only the binaries and the data packages are the next goal.

DejaVuSans.ttf is the only font that's needed for

Thanks. languages.ttf is the other required font file otherwise the game won't start up.

when do you plan to publish the debian version? And you plan to do this on the 2.5 codebase, no?

At the moment I'm working with the latest code from git, 2.5-dev. There isn't any fixed date for a Debian release. I suggest we take our time to create proper Debian packages and let others comment on our efforts. I assume everything could be working at the end of summer. Then it will take another few weeks to get the packages into Debian because they need to be checked by our ftp-masters.

Makes sense - but you can also decide to hardlink the game.so (./configure --enable-hardlinkedgame) then libufoai-game would not be needed.

That was a good idea. I have done that.

Well - to be executed the server does not need the music, pics, sounds and so on - but... the server can also be configured in a way to offer these files as downloads for clients that don't have one particular file. I don't think this will ever happen - but well... it's a feature. And new user created content would be available on the server anyway. But to be honest i've never tried to run the server on a minimal dataset.

That means the ufoai-server package does not need to depend on data packages to make it useful as long as the client depends on all data packages?

UFO:AI is very similar to OpenArena, thanks to the underlying Quake technology. However the openarena-server depends on data packages to enforce compatibility in case the player uses an older version (a new pak file will be automatically downloaded). I think we should discuss the desirable behaviour later, if i also know more about a sane default (system-wide) server configuration.

Please don't bulid the maps from source - please use those that we deliever when we release 2.5 officially. The problem here is floating point stuff. You would make cross plattform multiplayer stuff impossible because the checksum is different on different architectures. If you plan to release before we official release 2.5, please use the maps-sync script (make maps-sync)

That is easier said than done. I understand that it is more work for me to build the maps from source and that we have to be careful to ensure that the pk3 files are identical to the official ones. The ability to build everything from source with tools in Debian main is a requirement for software to be included in Debian. We simply need to make sure that the build process is reproducable on the official release architecture. The maps themselves are architecture-independent, that means they are only built once on my local (amd64) pc. I just have to make sure that the package matches your official release.

The OpenArena maintainer has done a similar job in the past.

anonscm.debian.org/gitweb/?p=pkg-games/openarena-data.git


The binaries could avoid some useless dependencies if they weren't linked against certain libraries. I am attaching my build log which illustrates the problem. (It is one of the last build steps at the end, dpkg-shlibdeps) I think this could be optimized somehow.

I am now going to learn more about creating the data packages.


Edit:

If i use p7zip aka 7z to compress the pk3 files, ufo is no longer able to detect them. The default compression algorithm is lzma. Zip compression works well with p7zip-full and zip. Is there an option to make ufoai detect those lzma compressed pk3 packages somehow? Otherwise there wouldn't be any advantage in switching to p7zip.




 

[Mattn]

Thanks. languages.ttf is the other required font file otherwise the game won't start up.

yes - but that one for sure isn't in debian repos

That means the ufoai-server package does not need to depend on data packages to make it useful as long as the client depends on all data packages?

It still needs the maps and scripts of course - but other stuff.... I would have to test that, too. There might be a few files that are still needed besides that (motd, some configs)

If i use p7zip aka 7z to compress the pk3 files, ufo is no longer able to detect them. The default compression algorithm is lzma. Zip compression works well with p7zip-full and zip. Is there an option to make ufoai detect those lzma compressed pk3 packages somehow? Otherwise there wouldn't be any advantage in switching to p7zip.

Strange, last time i checked it worked fine. I will have to recheck someday. There is no special flag to set. If it doesn't work out of the box... well - it doesn't work Sorry for that.

[Sandro]

Currently, UFO:AI supports only ZIP/DEFLATE as the compression algorithm. Adding support for LZMA could be a good idea, since it will give better compression for maps and models, but do we have a manpower for that?

[apo]

Hi

I have updated the ufoai packaging at http://anonscm.debian.org/gitweb/?p=pkg-games/ufoai.git
and also created two new git repositories for the ufoai-data and ufoai-music source packages.

http://anonscm.debian.org/gitweb/?p=pkg-games/ufoai-data.git
http://anonscm.debian.org/gitweb/?p=pkg-games/ufoai-music.git

So far only ufoai-maps is missing and I'm using the precompiled maps for testing purposes at the moment.

I have filed a bug report for a reproducible segfault that seems to be triggered by the sound system (probably sdl_mixer). So far I couldn't find a solution

http://ufoai.org/bugs/ufoalieninvasion/issues/5072

yes - but that one for sure isn't in debian repos

The question is, is languages.ttf something which is worth packaging separately? Normally fonts deserve a separate package because they might be useful for other software too. If that's not the case here, I suggest shipping the font with ufoai itself.

Currently, UFO:AI supports only ZIP/DEFLATE as the compression algorithm. Adding support for LZMA could be a good idea, since it will give better compression for maps and models, but do we have a manpower for that?

Yes, that would be great.

I have packaged all data in pk3 files for 2.5-dev and in total it takes up 1,2 GB disk space. 0music.pk3 and 0sound.pk3 are already highly compressed ogg files. Even lzma wouldn't improve the situation here. You can gain the most by compressing maps, textures and pictures with a better compression algorithm which will approximately save up to 150 MB.

I will open new threads for specific packaging issues because I think it will be more effective to follow them separately.

Outstanding points so far:

- Compiling the maps and producing a version that is identical to the official UFO:AI release.
- dedicated server: creating a system-wide server installation.
- picomodel: standalone package or integrated into ufoai?
- license audit.

[Mattn]

I have filed a bug report for a reproducible segfault that seems to be triggered by the sound system (probably sdl_mixer). So far I couldn't find a solution

http://ufoai.org/bugs/ufoalieninvasion/issues/5072

I will have a look as soon as possible.

The question is, is languages.ttf something which is worth packaging separately? Normally fonts deserve a separate package because they might be useful for other software too. If that's not the case here, I suggest shipping the font with ufoai itself.

This font is not useful for anything else than ufoai. So yes, ship it with ufoai, no separate package is needed.

- picomodel: standalone package or integrated into ufoai?

Would be great to get an own package for it. But it's not something that is a must-have.

Offtopic
Which reminds me of a totally different thing: How can I request an update for a specific package where a new version is available but it's not yet in debian? I'm talking about libcunit - there is a new version available that fixes a lot of warnings due to added const keywords. It would be great to have this updated. But I don't know much about this process. Can you guide me a little?

[Mattn]

added some comments to the SDL_mixer crash related bugreport - your feedback would be nice.

[apo]

Thanks for your feedback! I will try to get a backtrace with valgrind and report back to the bug tracker later.

Another question about the models. Shall I compile them? At the moment I'm using a model pk3 without precompiled models and the game seems to run fine.

Offtopic
Which reminds me of a totally different thing: How can I request an update for a specific package where a new version is available but it's not yet in debian? I'm talking about libcunit - there is a new version available that fixes a lot of warnings due to added const keywords. It would be great to have this updated. But I don't know much about this process. Can you guide me a little?

You can file a bug report with priority wishlist against libcunit1. The best tool for that is reportbug. You can also simply write an e-mail to submit@bugs.debian.org. Just use the following header.

Package: libcunit1
Version: 2.1-0.dfsg-12
Severity: wishlist

Dear maintainer,

your text here....

[Mattn]

Thanks for your feedback! I will try to get a backtrace with valgrind and report back to the bug tracker later.

Another question about the models. Shall I compile them? At the moment I'm using a model pk3 without precompiled models and the game seems to run fine.

compiling the mdx files speed up model (and thus game) loading times. Though it's not needed, it's really recommend to pre-compile them. Otherwise the mdx files are generated on the fly whenever a new model is loaded.

You can file a bug report with priority wishlist against libcunit1. The best tool for that is reportbug. You can also simply write an e-mail to submit@bugs.debian.org. Just use the following header.

Thanks a lot - will do.

[apo]

Thanks for updating ufo to SDL2! I'd love to use SDL2 in Debian but unfortunately nobody has packaged SDL2_ttf yet. Perhaps that would also fix the SDL_mixer bug I reported.

[Mattn]

SDL_mixer and SDL_ttf as well as SDL_image didn't get that many updates - but still a lot of fixes made it into the hg. So it's at least not impossible

Beware, our SDL2 port might still be a little bit wanky. But it would be worth a try - and if just for the fullscreen and multi monitor fixes that made it into SDL2. We still have some input problems though (which i'm after)