Basic usage

Note that all traffic that normally would be sent through the Tor network is sent directly from the system running the automated test suite to the destinations! In other words, nothing is torified, and some tests contact hosts associated with Tails, Tor, Riseup etc. It will, for instance, be obvious to your ISP that you are running the Tails automated test suite.

Use the run_test_suite script found in the Tails source root to run all automated Cucumber test features. See the setup documentation in case you don't have a testing environment yet. Note that the full Tails source tree must be readable by the user running the test suite.

It's important to note that some features only depend on the Tails sources, and some on the actual product of the sources, i.e. a Tails ISO image. These features are tagged @source and @product, respectively. The arguments passed to run_test_suite may only affect one of these types of features and not the other.

A typical example run of a few @product features could be:

./run_test_suite --view --capture test-0.17.webm \
    --iso path/to/tails.iso \
    features/apt.feature features/erase_memory.feature

which will test only the apt and erase_memory features (if no feature paths are given, all features in features/cucumber will be tested) of the given ISO image tails.iso while showing the test session in a VNC viewer (--view) and also capturing it into a video called test-0.17.web (--capture). Similarly, to test a @source feature, we'd simply run something like:

./run_test_suite features/build.feature

Actually, run_test_suite is just a wrapper around cucumber, so any cucumber option can be passed too, although after an -- so they are not confused with the wrapper's options. For instance:

./run_test_suite ... -- --format debug features/apt.feature

will enable the debug formatter, which in Tails' Cucumber setup will enable debugging information to be printed (which is very useful when debugging or developing for the test suite) unlike in vanilla Cucumber, where it's used for debugging the formatting subsystem.

For full instructions, see its --help.

Note: since the test suite itself uses virt-viewer to interact with the Tails guest you cannot use it as that will steal the session from the test suite, leaving it with a blank screen.

Configuration

The test suite can be configured in the following ways:

  1. run_test_suite parameters, which takes precedence over

  2. the local configuration file features/config/local.yml, which takes precedence over

  3. the local configuration files in features/config/*.d (with internal precedence according to lexical order), which takes precedence over

  4. the default configuration file features/config/defaults.yml.

However, some values are treated as secrets and have no defaults. These secrets are generally information about online sevices to be used in certain features, like host, port and credentials -- stuff we don't want to make public. These must be set explicitly in order for those features to run.

A Git repository, shared among a bunch of core Tails contributors, includes some of the needed secrets (more specifically, those that can be use by concurrent test suite runs):

git clone tails@git.tails.boum.org:test-suite-shared-secrets \
   features/config/shared-secrets.d

Non-secret configuration

Here's a list of all non-secret key-value pairs that can be supported by the local configuration file:

  • CAPTURE: Captures failed scenarios into videos stored in the temporary directory (see TMPDIR below) using x264 encoding. Defaults to false.

  • CAPTURE_ALL: Keep videos for all scenarios, including those that succeed (implies CAPTURE). Defaults to false.

  • MAX_NEW_TOR_CIRCUIT_RETRIES: Integer. Upon failure, some test steps may be run again after requesting that connections are made using new Tor circuits. This configuration variable limits how many times forcing a circuit will be attempted. Defaults to 10.

  • INTERACTIVE_DEBUGGING: Boolean value. If set to true, the test suite run is suspended on failure until ENTER is pressed, and an interactive Ruby shell (pry) is available. This is useful for investigating the state of Cucumber's world and the VM guest to see exactly why a test failed. Defaults to false.

  • SIKULI_RETRY_FINDFAILED: Boolean value. If set to true, print a warning whenever Sikuli fails to find an image and allow one retry after pressing ENTER. This is useful for updating outdated images, or when extracting new images. Defaults to false.

  • TMPDIR: String value. Directory where various temporary files are written during a test, e.g. VM snapshots and memory dumps, failure screenshots, pcap files and disk images. Defaults to "/tmp/TailsToaster".

  • NOTIFY_USER_COMMAND: String value. This arbitrary shell command will be executed whenever pause() is called, e.g. on test failure when INTERACTIVE_DEBUGGING (--interactive-debugging) is enabled. This is pretty useful when multitasking with long test suite runs, so you immediately are notified when a test fails (or when you reached a temporary pause() breakpoint).

"Secret" configuration

This section describes the formats for all secret configurations that must be configured in the local configuration file for certain features or scenarios to work. If any of these are omitted, parts of the test suite will fail.

Pidgin

These secrets are required for some scenarios in pidgin.feature. Here's an example which explains the format:

Pidgin:
  Accounts:
    XMPP:
      Tails_account:
        username: "test"
        domain: "jabber.org"
        password: "opensesame"
      Friend_account:
        username: "friend"
        domain: "jabber.org"
        password: "trustno1"
        otr_key: |
          (privkeys
           (account
          (name friend)
          (protocol xmpp)
          (private-key
           (dsa
          [...]

Note that the fields used in the above example show the mandatory fields.

The XMPP account described by Tails_account (to be used in Tails' Pidgin) and Friend_account (run via a bot from the tester host) must be subscribed to each other but to no other XMPP account. Also, for the Friend_account, it's important that the otr_key's name field is the same as username, and that the protocol field is xmpp.

If a "Connect Server" is needed for any of the accounts, it can be set in the optional connect_server field.

In case the Tails_account's conference server doesn't allow creating arbitrary chat rooms, a specific one that is known to work can be set in the optional chat_room field. It should still be a room with a strange name that is highly likely to always be empty; otherwise the test will fail.

XMPP services known to work well both for Tails_account and Friend_account are:

  • riseup.net (use connect_server: xmpp.riseup.net)
  • jabber.org (doesn't allow creating arbitrary chatrooms, so setting chat_room may be needed)
  • jabber.ccc.de

SSH

These settings are required for ssh.feature. The format is:

$TYPE:
  hostname: 1.2.3.4
  private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIJKAIBAAKCAgEAwJJK8LFxTWVnKUeOBdw+w69fDMmJugJmCx1TF/QS7kPfVPRl
    lz3hNOpdgZ0BkvC2Fd+mHAUKDWU5SHfCtYl2XyUkJ0p00844rphX+Bl0kVM7ISXt
    [...]
    -----END RSA PRIVATE KEY-----
  public_key: "ssh-rsa AAAAB3NzaC1yc2EA..."
  port: 22
  username: "someuser"

where $TYPE is SSH or SFTP. Secrets must be specified for both SSH and SFTP. If port is not specified, 22will be used.

The SSH test expects the remote system to have a default bash shell prompt.

Thunderbird

These settings are required for thunderbird.feature. The format is:

Icedove:
  address:  user@example.com
  password: trustno1

The email account's inbox must contain at least one email at all times.