- The art of writing new product test cases
- Limitations and issues
The automated test suite is essentially a glue between:
- Sikuli and dogtail, used for simulating user interaction,
- libvirt, used for running a specific build of Tails in a virtual machine, and
- cucumber, for defining features and scenarios testing them using the above two components.
Its goal is to automate the development and release testing processes with a Continuous Integration server.
Core developers can also run it on lizard.
With this tool, it is possible to:
- Create, modify and destroy virtual machines that can run Tails from a variety of media (primarily DVD and USB drives).
- Create different kinds of virtual storage (IDE, USB, DVD...), and either cold or hot plug/unplug them into/from the virtual machine.
- Modify the virtual machine's hardware, e.g. its amount of RAM, its processor features (PAE), etc.
- Simulate user interaction with the system under testing and check its state.
- Run arbitrary shell commands inside the virtual machine.
- Take screenshots from the display of the virtual machine, at particular events, like when a scenario fails. Complete test sessions can also be captured into a video.
- Capture and analyze the network traffic of the system under testing.
Fundamentally speaking there are two types of tests:
- Tests that make sure that the Tails sources behave correctly at
build time (example:
features/build.feature); these ones are aptly tagged
- Tests that make sure that the product built from Tails sources, i.e.
a Tails ISO image, behaves correctly at run time; these ones are
The requirement of these are quite different; for instance, a
@product test must have access to a Tails ISO image to test, whereas a
@source test doesn't. Therefore their environments are setup
separately using our custom
BeforeFeature hook, with the respective
run_test_suite script is a wrapper on top of
sets the correct environment up:
- It uses
Xvfbso that the
DISPLAYenvironment variable points to an unused X display (tip: use
Xvfb) with screen size and color depth matching what the Sikuli images are optimized for (that is, 1024x768 / 24-bit).
- It sets the
SIKULI_HOMEenvironment variable to the directory where sikuli script is installed for Java (
- It runs
$DISPLAYto prevent the mouse pointer from masking GUI elements looked for by Sikuli.
- It passes
cucumbercalls, to get access to our custom
This started out as a hack, and while it has evolved it largely remains so. A proper, reliable, established replacement would be welcome, but seems unlikely given the requirements:
Requirements on the host (the remote shell client):
- can execute a command with blocking until command completion on the server, and get back return code, stdout and stderr separately.
- can spawn a command without blocking (except an ACK that the server has run the command, perhaps).
- usable by an unprivileged user
Requirements on the guest (the remote shell server):
- has to work without a network connection.
- should interfere minimally with the surrounding system (e.g. no firewall exceptions; actually we don't want any network traffic at all from it, but this kind of follows from the previous requirement any way)
- must start before Tails Greeter. Since that's the first point of user interaction in a Tails system (if we ignore the boot menu), it seems like a good place to be able to assume that the remote shell is running.
- config/chroot local-includes/usr/local/lib/tails-autotest-remote-shell
- config/chroot local-includes/lib/systemd/system/tails-autotest-remote-shell.service
@source features, scenarios or steps should be pretty
straightforward for anyone with experience with
cucumber, but the
same can't be said about
@product tests, so below we give some
pointers for the latter.
In addition to the Sikuli, libvirt and Cucumber documentation linked to in the introduction:
- ruby-libvirt API
- sikuli API
sniffapplication, which is installed in Tails, is quite useful to navigate the GUI element hierarchy. However,
accerciser(not installed) is even better, due to its ability to highlight elements. Another useful tool is
ipython(not installed) with its TAB-completion.
First, one should have a good look at especially
features/step_definitions/common_steps.rb and the other features to
get a general idea of what already is possible. There is a good chance
there's already implementations for many steps necessary for reaching
the desired state right before when the stuff special to the intended
In order to learn some basic step dependencies, and concepts and features of the automated test suite used in features and scenarios, let's walk through a few typical steps, in order:
Given a computer
This is how each scenario (or background) should start. This step
destroys any residual VM from a previous scenario, and sets up a
completely fresh one, with all defaults. The defaults are defined in
features/domains/default.xml, but some highlights are:
- One virtual
x86_64CPU with one core
- A reasonable amount of RAM
- ACPI, APIC and PAE enabled
- UTC harware clock
- A DVD drive loaded with the Tails from the ISO
- No other storage plugged
- USB 2.0 controller
- Ethernet interface, plugged into a network bridged with the host
However, most of the time we do not set up a computer from scratch
using this step, but restore from a snapshot (also called checkpoint)
using the one of the
Given Tails has booted ... steps generated in
features/step_definitions/snapshots.rb. An example of such a step,
and indeed one of the most common ones, is:
Given Tails has booted from DVD and logged in and the network is connected
These steps will actually run multiple steps, saving one or more snapshots along the way. See the next section for details about this.
Returning back to what we'd do after the
Given a computer step,
there's a number of steps that reconfigures the computer...
And I create a 10 GiB disk named "some_disk"
The identifier (
some_disk) is later used if we want to plug it or
otherwise act on it. Note that all media created this way are backed
by qcow2 images, which grow only as they consume
capacity. All such media are destroyed after the feature ends.
This step does not necessarily have to be run this early, but it does if we want to plug it as a non-removable drive...
And I plug ide drive "some_disk"
Note that we can decide which type of drive
some_disk is here. If we
pick a non-removable media, like in this case, it has to be done
before we start the virtual machine. Removable media, such as USB
drives, can be plugged into a live system at any later point.
And the network is plugged
This plugs the network interface to the host-bridged network (which is the default). There's also the "unplugged" version, which generally is preferred for features that don't rely on the network (mostly so we won't hammer the Tor network unnecessarily). These steps can also be performed on a already running system under test.
And I start the computer
This is where we actually boot the virtual machine.
And the computer boots Tails
- verifies that we see the boot menu
- adds any boot options added via
I set Tails to boot with options ...
- makes sure that Tails Greeter starts
makes sure that the remote shell is up and running
And I enable more Tails Greeter options
This is required for steps enabling Tails Greeter options, like the next one. Note that the "I set sudo password ..." step has to be run before the other Tails Greeter option steps as it relies on keyboard navigation.
And I set sudo password "asdf"
Beyond the obvious, after this step all steps requiring the administrative password have access to it.
And I log in to a new session And the Tails desktop is ready And I have a network connection And Tor is ready
All these should be pretty obvious. It could be mentioned that the last two steps, like many others, depend on the remote shell to be working.
And all notifications have disappeared
The notifications can block GUI elements that we're looking for later
with Sikuli, so it's important that they are gone in essentially all
tests of GUI applications. If we have a network connection, so the
time syncing starts and shows its notifications, then this step should be
run after the previous step. Otherwise it always depends on the
has started step.
And available upgrades have been checked
Since Tor is working, the check for upgrades will be run. We have to wait for it to complete because that generally will break later if we use snapshots.
To speed up the test suite and get consistent results when setting up
state, we make heavy use of virtual machine snapshots. We encourage
contributors to read the snapshot definitions in
features/step_definitions/snapshots.rb carefully. We generate steps
from these descriptions, and they are created lazily on first use (and
then reused in subsequent instances, across features). Some things to
make note of:
Snapshots may have parents, which means that they start by running the parent's step, generating its snapshot, recursively to a "root" snapshot without parent.
Snapshots can be made "temporary". If the snapshot description's
:temporaryfield is set to
true, then the snapshot will be cleared after the feature it was created in finishes. This is a way to reduce the disk space needed for running the test suite, and is encouraged to use for features where a very perticular state is set up, that isn't reused in any other feature.
Debugging snapshot creation is made a lot easier by enabling the
debugformatter, which will print the steps as they are run.
Each such new scenario should sniff network traffic, and check that
all Internet traffic has only flowed through Tor. See the
feature for a simple example of how to do it.
In essense, the tools we have for interacting with the Tails instance are:
- the VM helper class,
- Sikuli and dogtail, and
- the remote shell.
It should be fairly obvious when to use the VM helper class (stuff relating to the virtual hardware), but there is some overlap between Sikuli, dogtail and the remote shell.
Sikuli and dogtail:
- They should be used in all instances where we want to simulate user interaction with the Tails system. For instance, when we start an application we usually want to click our way through the GNOME applications menu to stay true to what an actual user would do.
- They come in handy when we want to verify some state pertaining to the GUI.
- In general we prefer dogtail, since it is more precise given its direct access to GUI elements and their state, and Sikuli's images carry a quite heavy maintenance burden. However, Dogtail is not possible to use before GNOME has started, so we rely on Sikuli for anything before or after that. Also, for some applications it's simply hard to "navigate" to the right element in the hierarchy of elements of some GUI, due to ambiguity and/or lack of labels.
The remote shell:
- Useful for testing non-GUI state.
- Useful for reaching some state required before we test stuff depending on user interaction (which should use Sikuli!).
- It is acceptable (but not encouraged) to use the remote shell for simulating user terminal interaction when we need to analyze the commands output. This is because of Sikuli's OCR capabilities are poor, and cannot be depended on.
These things are good to know when developing new features, scenarios or steps.
The remote shells have a few surprises in store:
pgrep -fdetects itself, which can have potentially serious consequences if not dealt with.
- Minor groups are not set, so e.g. the
groupscommand may not do what you expect, but
These are the currently known oddities of the remote shell, but there may be more, so beware, and make sure to verify that any commands sent through it does what you want them to do.
Although very rare, the remote shell can get into a state where it stops responding, resulting in the test suite waiting for a response forever.
When creating a disk (at least when backed by a
raw image) via the
storage helper, and then plugging it to a Tails instance as SATA
drive, GNOME will report that the drive is failing when inside Tails,
and indeed several SMART tests fail. For now, plug hard disks as IDE
only (or USB, of course).
When creating snapshots, anything stored in a variable in any of those steps will only be available in subsequent steps in that scenario, not in other scenarios restoring from that snapshot. The exception is when global variables are used, which is an acceptable workaround, but it requires minute control to get right, and is hard to follow. Please try to avoid this until #5847 is solved.
When a scenario is broken for an extended period of time (e.g. when rebasing Tails on a new Debian version, which usually breaks tons of stuff) the current method of temporarily disabling affected tests is simply to remove them in Tails' Git, making sure that we won't forget about them.
Note that such removals should be isolated per commit so that they are easy to revert when whatever was broken is fixed. Hence, such a commit could either:
remove a single scenario if something unique to that scenario is broken;
remove multiple scenarios possibly spanning multiple features, if they're broken for the same reason, e.g. a step they all use is broken;
remove a complete feature, if the complete feature is broken for the same reason.
Each such commit must have a ticket created, referencing the commit to revert, have an assignee and an appropriate milestone set so they are not forgotten. Bonus point: push a branch that reverts the commit, and then set it as the ticket's Feature Branch.