Accéder au contenu principal

Games, Tests and GitLab CI

We are getting midterm of the GNOME 3.30 development cycle and many things already happened in the Games world. I will spare the user facing news for later as today I want to tell you about development features we desperatly needed as maintainers: tests and continuous integration.

TL;DR: GLib, Meson, Flatpak and GitLab CI make writing and running tests super easy! 😁 This will allow Games to be more stable and to have more features.

The More the Buggier

Not only does Games and retro-gtk are slowly becoming bigger and more complex, but to handle many platforms Games has to come flatpaked with Libretro cores. Games and retro-gtk are currently only tested manually and as far as I know, this is also true for the vast majority of the Libretro cores we distribute. That's quite a large number of untested lines of code, it is already impossible to test all of them manually and the test matrix is not going smaller. We are not immune to introducing new bugs or to accidentally reintroducing bugs we already fxed. We also caught some bugs in Libretro cores which caused a loss of save data, which is absolutely unacceptable (luckily, it was only in the unstable Flatpak and we mitigated the issue as soon as possible).

The Libretro API offers an interface for cores to define variables with a non-empty list of possible values that the frontend can set to the desired value; these variables can be seen as core-specific runtime APIs that Games could use to implement platform-specific features. Unfortunately, for Games to rely on such unstable APIs we would need to react quickly to their changes, changes which can't be detected at compilation time. Because of the lack of a good way to detect such changes and for the sake of maintainability, Games doesn't use the variables at all to implement platform-specific features.

These problems have simple answers: unit tests, behaviorial tests and continuous integration.

Mandatory meme.

Unit Tests

Meson and the GLib testing framework make implementing unit tests and code coverage super easy. retro-gtk has been ported to Meson a few versions ago so adding unit tests for its central RetroCore class was a breeze! Games has a Meson port by Alexander Mikhaylenko and Denis Ollier (thanks a lot!) that is waiting changes in the master GNOME runtime to be mergeable; once merged, unit tests will be written.

RetroCore requires a path to a Libretro core to be constructed and to run, so to test it correctly I wrote the retro-dummy Libretro cores which implements the strict minimum required by the Libretro API.

Tests are particularly convenient to try reproducing bugs and to ensure we don't reintroduce them by accident; for example until recently Games was crashing when loading a UTF-16 encoded cue sheet, so I want to write tests checking that various file formats are parsed correctly.

Our unit testing guidelines recommends to install tests system-wide, by doing so we have been able to flatpak the retro-gtk tests in org.gnome.Retro.UnitTests and we can now easily run them in a sandboxed environment.

Reftests

Unit tests are nice to check that small bits of code behave as expected, but retro-gtk being very dynamic by nature, testing the behavior at the application level is needed. To complement the unit tests, I wrote retro-reftest to run reference tests on Libretro cores via retro-gtk, allowing to compare the actual reactions of retro-gtk when running special test Libretro cores to the expected reactions.

First, you write a reference test file which looks like this:

[Retro Reftest]
Path=/test
Core=/app/lib/libretro/test_libretro.so

[Options]
test_aspect=4:3;16:9;
test_samplerate=30000;20000;
test_opt0=false;true;
test_opt1=0;
test_opt2=0;1;foo;3;

[Frame 0]
State=Refresh
Video=test.png

Then you run retro-reftest with this file and the --generate option to generate the test outputs (in that case there is only test.png), then you pass the reftest descriptor file to retro-reftest again and if all goes as expected, you should have this output:

/test/Boot: OK
/test/Options: OK
/test/0/State Refresh: OK
/test/0/Run: OK
/test/0/Video: OK

If you want to know more about how retro-reftest works and how to write reference test files, check the Retro Reference Test Case Specification page. The tests are still quite simple and they are expected to grow the ability to send various inputs, to set the options (called variables by Libretro), to test the rumble, to load the state of the core, to set the number of frames to run ahead of time…

Libretro offers a set of test cores which have been added to org.gnome.Retro.UnitTests with corresponding reference tests to test retro-gtk and to ensure it behaves correctly.

Continuous Integration

Tests are useless if they are not run, thankfully Games and retro-gtk moved to GitLab so we can benefit from its continuous integration tool to run org.gnome.Retro.UnitTests on a regular basis, but we can got way further.

By frequently rebuilding the flatpak of Games we can check that the bundled Libretro cores compile successfully, and thanks to retro-reftest we can also ensure they work as expected with retro-gtk! This is very important because as far as I know Libretro cores are not tested much by upstream and they are certainly not tested with retro-gtk which is what matters the most for us (if you know cores which are tested besides simply building, please let me know! This allows us to catch regressions or improvements in the Libretro cores early and to quickly react accordingly.

These Libretro core integration tests are shipped in org.gnome.Games.UnitTests and are regularly run via GitLab CI. Later, this flatpak will also receive and run the unit tests of Games and I would love to have code coverage support like GLib does. Thanks to all who allowed the CI to work in GNOME's GitLab as well as in Games and retro-gtk!

Digression About Test Data

Finding freely redistributable test data (understand: test programs) for many platforms is super hard! Most persons producing them just let the tests freely available but without specifying a license, making them by default proprietary software and hence unsafe to redistribute. I managed to find some test suites with free software licenses, but it was mostly out of luck. Please, tests and demos producers, consider distributing your work under a free license rather than none!

Alternatively, if you know how to write free software for any exotic platform please consider writing some simple test programs: even a Hello, world! is extremely valuable to us!

With Tests and CI Come Features

Now that we have tests running very often and with retro-gtk being able to check the variables offered by a core, we will be rapidly notified of any breakage in these unstable APIs and hence making using them more sustainable. With these core-specific APIs we could implement platform-specific features or set sensible defaults, for example:

  • you could select the palette of your Game Boy Color game,
  • you could choose on which kind of Game Boy your game runs on (some Game Boy Color games unlock bonuses when running on a Game Boy Advance),
  • we could force a Nintendo DS emulator to not mimick a mouse but to instead use the actual mouse pointer,
  • we could force the anaglyph mode of a Virtual Boy emulator to something Games can build upon,

Not being in constant fire monitoring mode and fearing breakages less will allow us to make Games grow! 😁

Commentaires

Posts les plus consultés de ce blog

GTK+ Apps on Phones

As some of you may already know, I recently joined Purism to help developing GTK+ apps for the upcoming Librem 5 phone.Purism and GNOME share a lot of ideas and values, so the GNOME HIG and GNOME apps are what we will focus on primarily: we will do all we can to not fork nor to reinvent the wheel but to help allowing existing GTK+ applications to work on phones.How Fit are Existing GTK+ Apps?Phones are very different from laptops and even tablets: their screen is very small and their main input method is a single thumb on a touchscreen. Luckily, many GNOME applications are touch-friendly and are fit for small screens. Many applications present you a tree of information you can browse and I see two main layouts used by for GNOME applications to let you navigate it.A first kind of layout is found in applications like Documents, I'll call it stack UI: it uses all the available space to display the collection of information sources (in that case, documents), clicking an element from t…

One Widget to Adapt Them All and to The Librem 5 Port Them

In my previous article I shared my plans to help porting existing GTK+ applications to Purism's upcoming Librem 5 phone without having to fork them. This article will present the GTK+ widget I developed for Purism to make this happen.For more information on what Purism is working on for the Librem 5, please check Nicole Faerber's latest article.C'est pas sorcierThe underlying idea is to allow applications to dynamically switch between the two main GNOME application layouts: a row of panels — each panel being the view of an element from the previous one — and a stack of panels. The goal isn't to changes applications using the stack paradigm but the ones using the row one, allowing them to reach smaller sizes and to be usable on constrained sizes while keeping their initial paradigm and design when the screen space is sufficient. The development cost to port the applications to this adaptive design should be as low as possible.To achieve that, I wrote a GTK+ widget which…

The Path to GNOME Games 3.26

Games received a non-negligible amount of changes that you will find in 3.26. These changes can be big as much small, and more are to come!Building the Games CollectionGames presents your games collection and if everything goes as expected, it does so without the need of any input from you. From an implementation point of view it sounds simple to do, just ask Tracker “Hey, gimme all the games” and it’s done. If only it was that simple! 😃 The system has no idea which files represent games and which doesn’t, but it can associate a MIME type to each file thanks to shared-mime-info. shared-mime-info already had a few video game related MIME types and we added a lot more such as application/x-genesis-rom.That done, we can query Tracker for files having specific MIME types that we know to often represent video game files. Unfortunately, each of these files doesn’t necessarily represent a game and a game isn’t necessarily represented by a single file: some files may be invalid and hence rep…