Tests need to be run all the time to be valuable. When they fail just after you introduce an error, you will know right away that what caused it. If your tests run too infrequently, you will have to troubleshoot longer and they will break more often.
This is why adding test runs to the CI is one of the first things I do when I set up a new project. When I started Flutter two years ago, Github had not yet released its Github Actions. Instead, I went for Gitlab who had their free tier of Gitlab CI.
Until now, I haven’t been happy with my set up though. It has been really hard to read the output of the test run because the format is verbose.
But a few weeks back I decided to revisit this and I’m happy to share today my updated CI flow with you.
Why not having a test report is tricky
A picture says more than a thousand words. Here is how one of my test failures used to look like. It is hard to find which test is failing:
At the bottom of this test log, it will tell you how many tests failed. But to find which tests failed, you have to read through the log to try to pick them out.
Instead, I would usually rerun the full test suite locally and find the failures. But there is a better way!
Gitlab supports displaying test reports
When I first started using Gitlab CI I could not find any support for test reports but it looks like it existed already back in Aug 2018.
If you have used Jenkins or any other older CI system you are probably familiar with the JUnit XML test report. It’s become kind of a standard for reporting test results. It has wide adoption across platforms and tools and is simple but does the job.
As such that’s the format Gitlab supports. Finally, there is also a good package for Flutter to translate the Flutter test result machine output to JUnit XML so we can use it.
This is how it looks:
With this, we get a nice clean breakdown of the tests ran and the results.
Compared to sifting through the log at the top, this is heaven. It makes troubleshooting a lot faster.
How to add Flutter junit report to your Gitlab CI
Let’s get practical, this is my actual Gitlab CI file (and the repo for the container I am using)
The magic lines here are the
artifacts part which will take the junit xml file generated and upload it to gitlab. Of course, you will also need to get access to the generator pub package which I do with this line:
pub global activate junitreport
I call the
tojunit package piping in the Flutter test report with machine output and then pipe this to an output file.
flutter test --coverage --machine | tojunit > testreport.xml
You may notice that I’m splitting my pipeline into three steps, linting, testing and coverage. This is a bit overkill and does slow the pipeline down slightly. Initially, I did this to easily find issues in linting and I wanted to add a failure on low test coverage.
For the linting, I have found that it saves much more time to do a lint with a pre-commit hook. As for the coverage check out my post on how to report test coverage in Gitlab with Flutter.
Gitlab CI and the this new package for generating an xml report makes this dead simple. Just go do it, I’m sure it’ll take you no more than 10-15 minutes.
Let me know how it goes or if you have any improvements to recommend, are you using some different report? I’m all ears!