Daily tests of torbrowser-launcher, and on every git commit too

Tor Browser for reasons beyond this blog post is not part of Debian. To easily and securely use it, one can run

sudo apt-get install torbrowser-launcher
and then run torbrowser-launcher which will download Tor Browser and well, launch it. And this sometimes breaks, when things change, which is rather frequently the case…

So yesterday I finally woke up to see what I wished to see every morning since November 15th last year, when I started testing torbrowser-launcher systematically on jenkins.debian.net:

This is the graphical status overview of torbrowser tests on Debian which are tests installing torbrowser-launcher on and from sid, stretch, jessie-backports, jessie and wheezy-backports, which first download torbrowser via https and via tor, then launches it to finally connect to a debian mirror via an onion-address and then to www.debian.org. In addition to these there are also tests installing the package from stretch on jessie as well as the package from sid on stretch and jessie. Finally there are also tests for building the package from our git branches for various suites and finally we also build a package based on the upstream git master branch merged with our sid packaging.

There are 17 different tests currently and they are configured in just two files, a 220 line yaml file defining the jenkins jobs and 528 lines of bash script containing the actual test code. The tests are using schroot, xvfb, kvkbd and gocr and are executed either daily or weekly (those testing the package from ftp.d.o) or on every commit and at least once every month (those testing the package build from git).

At least briefly I have been looking at the this page every day since setting up these tests two months ago. Back then, torbrowser-launcher was broken in many suites and then it broke some more and instead of fixing it, I first made these tests so that at least in future there will be automatic notifications when things break. This has worked out rather nicely and torbrowser-launcher got fixed over time too. Doing so in jessie proper took longest as we missed 8.2. Once jessie was fixed the fix for wheezy-backports was also finally accepted very quickly. Which was this Monday, so yesterday on Tuesday I could enjoy for the first time an all "green page"! :-)

And then on Wednesday, which is now also yesterday, the nice green page became less green due two new issues: #805173 really needs ca-certificates as depends and not recommends and then also the rendering of the new 5.5 version of torbrowser changed slightly… Both issues have been addressed by now.

I'm curious how this page will look tomorrow morning. And when I'll consider it the number of false negatives to be low enough so as I'll happily enable notifications to be send to the pkg privacy maintainers mailing list. I think it's almost the case and will keep an eye on the results the coming weeks and months.

Last not least: if you want to run similar tests for your Debian project, I'd be glad to help! See you at FOSDEM?