Accéder au contenu principal

Games, Tests and GitLab CI

We are getting midterm of the GNOME 3.30 development cycle and many things already happened in the Games world. I will spare the user facing news for later as today I want to tell you about development features we desperatly needed as maintainers: tests and continuous integration.

TL;DR: GLib, Meson, Flatpak and GitLab CI make writing and running tests super easy! 😁 This will allow Games to be more stable and to have more features.

The More the Buggier

Not only does Games and retro-gtk are slowly becoming bigger and more complex, but to handle many platforms Games has to come flatpaked with Libretro cores. Games and retro-gtk are currently only tested manually and as far as I know, this is also true for the vast majority of the Libretro cores we distribute. That's quite a large number of untested lines of code, it is already impossible to test all of them manually and the test matrix is not going smaller. We are not immune to introducing new bugs or to accidentally reintroducing bugs we already fxed. We also caught some bugs in Libretro cores which caused a loss of save data, which is absolutely unacceptable (luckily, it was only in the unstable Flatpak and we mitigated the issue as soon as possible).

The Libretro API offers an interface for cores to define variables with a non-empty list of possible values that the frontend can set to the desired value; these variables can be seen as core-specific runtime APIs that Games could use to implement platform-specific features. Unfortunately, for Games to rely on such unstable APIs we would need to react quickly to their changes, changes which can't be detected at compilation time. Because of the lack of a good way to detect such changes and for the sake of maintainability, Games doesn't use the variables at all to implement platform-specific features.

These problems have simple answers: unit tests, behaviorial tests and continuous integration.

Mandatory meme.

Unit Tests

Meson and the GLib testing framework make implementing unit tests and code coverage super easy. retro-gtk has been ported to Meson a few versions ago so adding unit tests for its central RetroCore class was a breeze! Games has a Meson port by Alexander Mikhaylenko and Denis Ollier (thanks a lot!) that is waiting changes in the master GNOME runtime to be mergeable; once merged, unit tests will be written.

RetroCore requires a path to a Libretro core to be constructed and to run, so to test it correctly I wrote the retro-dummy Libretro cores which implements the strict minimum required by the Libretro API.

Tests are particularly convenient to try reproducing bugs and to ensure we don't reintroduce them by accident; for example until recently Games was crashing when loading a UTF-16 encoded cue sheet, so I want to write tests checking that various file formats are parsed correctly.

Our unit testing guidelines recommends to install tests system-wide, by doing so we have been able to flatpak the retro-gtk tests in org.gnome.Retro.UnitTests and we can now easily run them in a sandboxed environment.


Unit tests are nice to check that small bits of code behave as expected, but retro-gtk being very dynamic by nature, testing the behavior at the application level is needed. To complement the unit tests, I wrote retro-reftest to run reference tests on Libretro cores via retro-gtk, allowing to compare the actual reactions of retro-gtk when running special test Libretro cores to the expected reactions.

First, you write a reference test file which looks like this:

[Retro Reftest]


[Frame 0]

Then you run retro-reftest with this file and the --generate option to generate the test outputs (in that case there is only test.png), then you pass the reftest descriptor file to retro-reftest again and if all goes as expected, you should have this output:

/test/Boot: OK
/test/Options: OK
/test/0/State Refresh: OK
/test/0/Run: OK
/test/0/Video: OK

If you want to know more about how retro-reftest works and how to write reference test files, check the Retro Reference Test Case Specification page. The tests are still quite simple and they are expected to grow the ability to send various inputs, to set the options (called variables by Libretro), to test the rumble, to load the state of the core, to set the number of frames to run ahead of time…

Libretro offers a set of test cores which have been added to org.gnome.Retro.UnitTests with corresponding reference tests to test retro-gtk and to ensure it behaves correctly.

Continuous Integration

Tests are useless if they are not run, thankfully Games and retro-gtk moved to GitLab so we can benefit from its continuous integration tool to run org.gnome.Retro.UnitTests on a regular basis, but we can got way further.

By frequently rebuilding the flatpak of Games we can check that the bundled Libretro cores compile successfully, and thanks to retro-reftest we can also ensure they work as expected with retro-gtk! This is very important because as far as I know Libretro cores are not tested much by upstream and they are certainly not tested with retro-gtk which is what matters the most for us (if you know cores which are tested besides simply building, please let me know! This allows us to catch regressions or improvements in the Libretro cores early and to quickly react accordingly.

These Libretro core integration tests are shipped in org.gnome.Games.UnitTests and are regularly run via GitLab CI. Later, this flatpak will also receive and run the unit tests of Games and I would love to have code coverage support like GLib does. Thanks to all who allowed the CI to work in GNOME's GitLab as well as in Games and retro-gtk!

Digression About Test Data

Finding freely redistributable test data (understand: test programs) for many platforms is super hard! Most persons producing them just let the tests freely available but without specifying a license, making them by default proprietary software and hence unsafe to redistribute. I managed to find some test suites with free software licenses, but it was mostly out of luck. Please, tests and demos producers, consider distributing your work under a free license rather than none!

Alternatively, if you know how to write free software for any exotic platform please consider writing some simple test programs: even a Hello, world! is extremely valuable to us!

With Tests and CI Come Features

Now that we have tests running very often and with retro-gtk being able to check the variables offered by a core, we will be rapidly notified of any breakage in these unstable APIs and hence making using them more sustainable. With these core-specific APIs we could implement platform-specific features or set sensible defaults, for example:

  • you could select the palette of your Game Boy Color game,
  • you could choose on which kind of Game Boy your game runs on (some Game Boy Color games unlock bonuses when running on a Game Boy Advance),
  • we could force a Nintendo DS emulator to not mimick a mouse but to instead use the actual mouse pointer,
  • we could force the anaglyph mode of a Virtual Boy emulator to something Games can build upon,

Not being in constant fire monitoring mode and fearing breakages less will allow us to make Games grow! 😁


  1. Thank you for the nice article here. Really nice and keep update to explore more gaming tips and ideas.

    Game Testing Compaies

    Android Game Tester

    Game Automated Testing

    Gameplay Testing

  2. It’s interesting to read content. nice post.
    and also we are providing E-Learning Portal Videos for students and working Professionals
    Hurry Up! Bag All Courses in Rs - 10000 /- + taxes
    41 Career building courses.
    Designed by 33 industrial experts
    600+ hours of video Content
    DevOps and Cloud E-Learning Portal

  3. Erectile dysfunction is a condition where a man is not able to get an erection. Even if they are able to get an erection, it does not last for too long. Suffering from erectile dysfunction can affect the person both physically and mentally. Therefore a person needs to take medical help to make the condition better. Also suffering from erectile dysfunction can affect the relation of the person with their partners. The medication that has brought about a wave of change in this field is the use of Viagra for erectile dysfunction. It works by targeting the basic cause of the issue thus helping millions of men all across the world. If you are a man who has been facing an issue with getting and maintaining an erection for a long time now, then you should
    .Buy Viagra online


Enregistrer un commentaire

Posts les plus consultés de ce blog

libhandy 0.0.10

libhandy 0.0.10 just got released, and it comes with a few new adaptive widgets for your GTK app. You can get this new version here . The View Switcher GNOME applications typically use a GtkStackSwitcher to switch between their views. This design works fine on a desktop, but not so well on really narrow devices like mobile phones, so Tobias Bernard designed a more modern and adaptive replacement — now available in libhandy as the HdyViewSwitcher . In many ways, the HdyViewSwitcher functions very similarly to a GtkStackSwitcher : you assign it a GtkStack containing your application's pages, and it will display a row of side-by-side, homogeneously-sized buttons, each representing a page. It differs in that it can display both the title and the icon of your pages, and that the layout of the buttons automatically adapts to a narrower version, depending on the available width. We have also added a view switcher bar, designed to be used at he bottom of the window: HdyView

Moving the Blog

I am moving this blog to greener lands: . The existing articles will remain here on Blogger, and new articles will land on the Plume instance.

GTK+ Apps on Phones

As some of you may already know, I recently joined Purism to help developing GTK+ apps for the upcoming Librem 5 phone . Purism and GNOME share a lot of ideas and values, so the GNOME HIG and GNOME apps are what we will focus on primarily: we will do all we can to not fork nor to reinvent the wheel but to help allowing existing GTK+ applications to work on phones. How Fit are Existing GTK+ Apps? Phones are very different from laptops and even tablets: their screen is very small and their main input method is a single thumb on a touchscreen. Luckily, many GNOME applications are touch-friendly and are fit for small screens. Many applications present you a tree of information you can browse and I see two main layouts used by for GNOME applications to let you navigate it. A first kind of layout is found in applications like Documents, I'll call it stack UI : it uses all the available space to display the collection of information sources (in that case, documents), clicking a