Accéder au contenu principal

Games, Tests and GitLab CI

We are getting midterm of the GNOME 3.30 development cycle and many things already happened in the Games world. I will spare the user facing news for later as today I want to tell you about development features we desperatly needed as maintainers: tests and continuous integration.

TL;DR: GLib, Meson, Flatpak and GitLab CI make writing and running tests super easy! 😁 This will allow Games to be more stable and to have more features.

The More the Buggier

Not only does Games and retro-gtk are slowly becoming bigger and more complex, but to handle many platforms Games has to come flatpaked with Libretro cores. Games and retro-gtk are currently only tested manually and as far as I know, this is also true for the vast majority of the Libretro cores we distribute. That's quite a large number of untested lines of code, it is already impossible to test all of them manually and the test matrix is not going smaller. We are not immune to introducing new bugs or to accidentally reintroducing bugs we already fxed. We also caught some bugs in Libretro cores which caused a loss of save data, which is absolutely unacceptable (luckily, it was only in the unstable Flatpak and we mitigated the issue as soon as possible).

The Libretro API offers an interface for cores to define variables with a non-empty list of possible values that the frontend can set to the desired value; these variables can be seen as core-specific runtime APIs that Games could use to implement platform-specific features. Unfortunately, for Games to rely on such unstable APIs we would need to react quickly to their changes, changes which can't be detected at compilation time. Because of the lack of a good way to detect such changes and for the sake of maintainability, Games doesn't use the variables at all to implement platform-specific features.

These problems have simple answers: unit tests, behaviorial tests and continuous integration.

Mandatory meme.

Unit Tests

Meson and the GLib testing framework make implementing unit tests and code coverage super easy. retro-gtk has been ported to Meson a few versions ago so adding unit tests for its central RetroCore class was a breeze! Games has a Meson port by Alexander Mikhaylenko and Denis Ollier (thanks a lot!) that is waiting changes in the master GNOME runtime to be mergeable; once merged, unit tests will be written.

RetroCore requires a path to a Libretro core to be constructed and to run, so to test it correctly I wrote the retro-dummy Libretro cores which implements the strict minimum required by the Libretro API.

Tests are particularly convenient to try reproducing bugs and to ensure we don't reintroduce them by accident; for example until recently Games was crashing when loading a UTF-16 encoded cue sheet, so I want to write tests checking that various file formats are parsed correctly.

Our unit testing guidelines recommends to install tests system-wide, by doing so we have been able to flatpak the retro-gtk tests in org.gnome.Retro.UnitTests and we can now easily run them in a sandboxed environment.


Unit tests are nice to check that small bits of code behave as expected, but retro-gtk being very dynamic by nature, testing the behavior at the application level is needed. To complement the unit tests, I wrote retro-reftest to run reference tests on Libretro cores via retro-gtk, allowing to compare the actual reactions of retro-gtk when running special test Libretro cores to the expected reactions.

First, you write a reference test file which looks like this:

[Retro Reftest]


[Frame 0]

Then you run retro-reftest with this file and the --generate option to generate the test outputs (in that case there is only test.png), then you pass the reftest descriptor file to retro-reftest again and if all goes as expected, you should have this output:

/test/Boot: OK
/test/Options: OK
/test/0/State Refresh: OK
/test/0/Run: OK
/test/0/Video: OK

If you want to know more about how retro-reftest works and how to write reference test files, check the Retro Reference Test Case Specification page. The tests are still quite simple and they are expected to grow the ability to send various inputs, to set the options (called variables by Libretro), to test the rumble, to load the state of the core, to set the number of frames to run ahead of time…

Libretro offers a set of test cores which have been added to org.gnome.Retro.UnitTests with corresponding reference tests to test retro-gtk and to ensure it behaves correctly.

Continuous Integration

Tests are useless if they are not run, thankfully Games and retro-gtk moved to GitLab so we can benefit from its continuous integration tool to run org.gnome.Retro.UnitTests on a regular basis, but we can got way further.

By frequently rebuilding the flatpak of Games we can check that the bundled Libretro cores compile successfully, and thanks to retro-reftest we can also ensure they work as expected with retro-gtk! This is very important because as far as I know Libretro cores are not tested much by upstream and they are certainly not tested with retro-gtk which is what matters the most for us (if you know cores which are tested besides simply building, please let me know! This allows us to catch regressions or improvements in the Libretro cores early and to quickly react accordingly.

These Libretro core integration tests are shipped in org.gnome.Games.UnitTests and are regularly run via GitLab CI. Later, this flatpak will also receive and run the unit tests of Games and I would love to have code coverage support like GLib does. Thanks to all who allowed the CI to work in GNOME's GitLab as well as in Games and retro-gtk!

Digression About Test Data

Finding freely redistributable test data (understand: test programs) for many platforms is super hard! Most persons producing them just let the tests freely available but without specifying a license, making them by default proprietary software and hence unsafe to redistribute. I managed to find some test suites with free software licenses, but it was mostly out of luck. Please, tests and demos producers, consider distributing your work under a free license rather than none!

Alternatively, if you know how to write free software for any exotic platform please consider writing some simple test programs: even a Hello, world! is extremely valuable to us!

With Tests and CI Come Features

Now that we have tests running very often and with retro-gtk being able to check the variables offered by a core, we will be rapidly notified of any breakage in these unstable APIs and hence making using them more sustainable. With these core-specific APIs we could implement platform-specific features or set sensible defaults, for example:

  • you could select the palette of your Game Boy Color game,
  • you could choose on which kind of Game Boy your game runs on (some Game Boy Color games unlock bonuses when running on a Game Boy Advance),
  • we could force a Nintendo DS emulator to not mimick a mouse but to instead use the actual mouse pointer,
  • we could force the anaglyph mode of a Virtual Boy emulator to something Games can build upon,

Not being in constant fire monitoring mode and fearing breakages less will allow us to make Games grow! 😁


Enregistrer un commentaire

Posts les plus consultés de ce blog

Librem 5 ❤️ GNOME 3.32

I am glad to announce that the tooling I am working on since the beginning of the year is ready to be used!Thanks to new features introduced into libhandy 0.0.3 and 0.0.4 and thanks to a few fixes to Adwaita in GTK+ 3.24.1, you can make GTK+ 3 apps adaptive to work both on the desktop and on the upcoming GNOME-based Librem 5 phone.We are early in the GNOME 3.32 release schedule and the Librem 5 will be released a bit after it, so if you want your apps to work on the Librem 5, now is the best time: use libhandy 0.0.4 and up, use GTK+ 3.24.1 and up and target GNOME 3.32! A few apps like Fractal, Podcasts, Calls and Chatty are already using libhandy's adaptive capabilities, and other apps are working on their adaptive transition like Contacts, Games, Geary and Settings (all are works in progress). libhandy is available in Debian Unstable and Arch's AUR repository, and I wish it would be in Fedora already to let GNOME Settings' CI pass.For the moment, libhandy is a GTK+ 3 widg…

My Name is Handy, Lib Handy

Libhandy 0.0.7 just got released! I didn't blog about this mobile and adaptive oriented GTK widget library since the release of its 0.0.4 version three months ago, so let's catch up on what has been added since.List RowsA common pattern in GNOME applications is lists, which are typically implemented via GtkListBox. More specific patterns arose, where rows have a title at the start, an optional subtitle below it, actions at the end and an icon or some other widget like a radio button as a prefix. These rows can also be expanded to reveal nested rows or anything else that fits the need.So far every application using these patterns implemented the rows by hand for each and every row. It made using these a bit cumbersome and it led to inconsistencies in sizing, even inside a single application. To make these patterns easier to use, we implemented HdyActionRow, HdyComboRow and HdyExpanderRow. HdyActionRowThe action row is a simple and flexible row, it lets you give it a title, a sub…

GNOME at Capitole du Libre 2018

Last Saturday and Sunday I went to the Capitole du Libre 2018 to animate the GNOME booth and help on the Purism one. It was my first time at the Capitole du Libre, and I really enjoyed the event even though I couldn't attend any talk but the closing keynote, which was really interesting as it was a round table with Tristan Nitot and Stéphane Bortzmeyer. The event is about free software and free culture in general, beside GNOME and Purism you could find associations as varied as Nos oignons, Framasoft, OSGeo, Dogmazic, Monnaie Libre Occitanie, Khaganat, ./, 0 A.D., various distros you already all know and more. I was happy to note that the attending crowed was extremely diverse, ranging from persons barely knowing what free software is to Arnaud Bonatti or persons in their early 10s and already more involved into free culture than I am. I was also happy to recognize persons I already met at the JDLL, either animating booths or visiting the GNOME booth.The event welcomed (IIRC…