Some test results that might be useful to keep are saved.

Read this document from the branch used to prepare the release.

Changes

Keeping an eye on the changes between released versions is one of the many safeguards against releasing crap.

Source

Compare the to-be-released source code with previous version's one e.g.:

Boot the candidate ISO and find the commit it was build from with the tails-version command.

Then, from the source tree, see the diff:

git diff --find-renames <old ISO commit>..<ISO commit>

e.g. git diff --find-renames 334e1c485a3a79be9fff899d4dc9d2db89cdc9e1..cfbde80925fdd0af008f10bc90c8a91a578c58e3

Result

Compare the list of bundled packages and versions with the one shipped last time. .packages are usually attached to the email announcing the ISO is ready.

/usr/bin/diff -u \
    wiki/src/torrents/files/tails-i386-1.3.1.packages \
    tails-i386-1.3.2.packages \
    | wdiff --diff-input  --terminal

Check the output for:

  • new packages that may cause harm or make the images unnecessarily big
  • packages that could be erroneously removed
  • new versions of software we might not have audited yet (including: does the combination of our configuration with software X version Y+1 achieve the same wished results as with software X version Y?)

Image size

Check the image size has not changed much since the last release.

In a directory with many Tails ISO images:

find -iname "tails*.iso" -exec ls -lh '{}' \; | sort -rhk 5

Automated test suite

Our long term goal is to eliminate the manual test suite (except the parts which require real hardware) and have the automated test suite run all our tests. It's design, and how to write new tests, are documented on a dedicated page.

Running the automated test suite

See setup and usage.

Automated test suite migration progress

The manual test suite below either contains tests that cannot be automated, has no automated test implemented yet, or has a test implemented, but it either hasn't been reviewed, had a confirmed pass by someone other than the test author, or has issues. The latter is tracked by tickets prefixed with todo/test_suite:.

Tor Browser

Security and fingerprinting

  • Run the tests the TBB folks use.
  • Compare the fingerprint of Tails and the latest TBB using at least https://panopticlick.eff.org/
    • The exposed User-Agent should match the latest TBB's one.
    • Update the fingerprint section of the known issues page if needed.
  • WebRTC should be disabled:
    • In about:config check that media.peerconnection.enabled is set to false.
    • http://mozilla.github.io/webrtc-landing/, especially the getUserMedia test. It's expected that the audio test works if you agree to share a microphone with the remote website; anything else should fail.
    • http://net.ipcalf.com/ should display ifconfig | grep inet | grep -v inet6 | cut -d" " -f2 | tail -n1
  • One should be able to switch identities from the web browser.
  • Running getTorBrowserUserAgent should produce the User-Agent set by the installed version of Torbutton, and used in the Tor Browser.

Functionality

  • Browsing (by IP) a HTTP or HTTPS server on the LAN should be possible.
  • Browsing (by IP) a FTP server on the LAN should be possible.

Tor

Claws

  • Check mail over IMAP using:
    • a "clearnet" IMAP server.
    • a hidden service IMAP server (e.g. Riseup, zsolxunfmbfuq7wf.onion with SSL).
  • Send an email using:
    • a "clearnet" SMTP server.
    • a hidden service SMTP server (see above).
  • Check that the profile works and is torified:
    1. Send an email using Claws and a non-anonymizing SMTP relay (a SMTP relay that writes the IP address of the client it is relaying email for in the Received header).
    2. Then check that email's headers once received, especially the Received: one.
  • Also check that the EHLO/HELO SMTP message is not leaking anything at the application level:
    1. Start Claws using the panel icon.
    2. Disable SSL/TLS for SMTP in Claws (so take precautions for not leaking your password in plaintext by either changing it temporarily or using a disposable account).
    3. Run sudo tcpdump -n -i lo -w dump to capture the packets before Tor encrypts it, then close tcpdump
    4. Check the dump for the HELO/EHLO message and verify that it only contains localhost: tcpdump -A -r dump
    5. Check the Received: and Message-Id fields in the received message: it must not leak the hostname, nor the local IP.
  • Make sure Claws Mail use its dedicated SocksPort when connecting to IMAP / POP3 / SMTP servers by monitoring the output of this command:

    sudo watch -n 0.1 'netstat -taupen | grep claws'
    

WhisperBack

  • I should be able to send a bug report with WhisperBack.
  • When we receive this bug report on the tails-bugs mailing-list, Schleuder tells us that it was sent encrypted.

Erase memory on shutdown

  • memlockd must be running
  • udev-watchdog must monitor the right device when booted off USB (automate: ticket #5560)
  • udev-watchdog must monitor the right device when booted off DVD (automate: ticket #5560)
  • After booting from DVD, remove Tails boot medium and check that the memory erasure process is started (Loading new kernel, at least). (automate: ticket #5472)
  • After booting from USB, remove Tails boot medium and check that the memory erasure process is started (Loading new kernel, at least).

Root access control

  • Check you can login as root with su neither with the amnesia password nor with the live one.
  • Check that the $TAILS_USER_PASSWORD variable, if still existing in the system environment after the boot has finished, does not contain the clear text password.

Virtualization support

  • Test that Tails starts and the browser launches in VirtualBox.

I2P

Make sure that I2P is up-to-date, at least if the changelogs mention that security critical bugs were fixed.

Start I2P by appending i2p to the kernel command line.

  • Check that I2P starts when a network interface is up:
    • Within 30 seconds you should get the "I2P router console is ready" pop-up
    • Start the I2P Browser via "Applications -> Internet -> I2P Browser":
      • You get the "Starting I2P Browser..." pop-up.
      • The router console (http://127.0.0.1:7657) opens successfully upon success.
      • On exiting I2P Browser, check that its chroot gets properly torn down on exit (there should be nothing mounted inside /var/lib/i2p-browser).
    • After a few minutes you should get the "I2P is ready" pop-up
    • Go to http://127.0.0.1:7657/i2ptunnelmgr in the I2P Browser:
      • You should get "Network: Hidden" in the "General" section.
      • The numbers in the "Peers" section of the sidebar should be non-zero.
      • Check that you can reach some eepsites within Iceweasel, like http://i2p-projekt.i2p and http://forum.i2p.
    • Check that you can connect to the I2P IRC server through Pidgin and the preconfigured IRC account on 127.0.0.1.
  • Check I2P failure modes:
    • Router console failure:
      • Boot without network so I2P doesn't start automatically.
      • Block the router console port: nc -l -p 7657 -t 127.0.0.1
      • Plug the network
      • You should get the "I2P failed to start" pop-up, and I2P should not be running (check with service i2p status)
    • Bootstrap failure:
      • Detach the network immediately after getting the "I2P router console is ready" pop-up
      • Wait for up to six minutes
      • You should get the "I2P is not ready" pop-up
      • The I2P router console should still be accessible on http://127.0.0.1:7657

SSH

  • Connecting (by IP) over SSH to a server on the LAN should work.

APT

 grep -r deb.tails.boum.org /etc/apt/sources.list*
  • Make sure the Tails repository suite in matching the release tag (for example the release version number) is in APT sources.
  • Make sure the Tails repository unversioned suites (e.g. testing, stable and devel) are not in APT sources.

Incremental upgrades

  • List the versions from which an upgrade paths to this one is described. In the stable or testing branch:

    git grep -l "  version: '\?0.23'\?" wiki/src/upgrade/
    
  • For each description file, open it and verify if it allows incremental upgrade or only full upgrade.

  • For each previous version from which an upgrade paths is described, install it and try to upgrade:

    • For every incremental upgrade path: make sure the resulting updated system "works fine" (boots and pretends to be the correct version).
    • For upgrade paths that only propose a full upgrade: make sure the user is told to do a manual upgrade.

    If:

    then:

      echo 'TAILS_CHANNEL="alpha"' | sudo tee --append /etc/os-release && \
      tails-upgrade-frontend-wrapper
    

    Else, use a local test setup:

    • A web server on the LAN.
    • A copy of wiki/src/upgrade from the stable or testing branch, for example in /var/www/tails/upgrade/v1/Tails/0.14~rc2/i386/stable/updates.yml
    • A copy of the iuk directory of our HTTP mirrors, for example in /var/www/tails/stable/iuk/Tails_i386_0.14-rc2_to_0.14.iuk.

      To synchronize your local copy:

      torsocks rsync -rt --progress --delete rsync.torproject.org::amnesia-archive/tails/stable/iuk/ /var/www/tails/stable/iuk/
      
    • Patch /etc/hosts in Tails to point to your web server:

      echo "192.168.1.4    dl.amnesia.boum.org" | sudo tee --append /etc/hosts
      
    • Patch sudo configuration to allow passing arbitrary arguments to tails-upgrade-frontend:

      sudo sed -i \
          -e 's,/usr/bin/tails-upgrade-frontend ""$,/usr/bin/tails-upgrade-frontend,' \
          /etc/sudoers.d/zzz_upgrade
      
    • Call the upgrader must be called, from inside the system to upgrade, with every needed option to use the local web server rather than the online one, for example:

      DISABLE_PROXY=1 SSL_NO_VERIFY=1 \
      tails-upgrade-frontend-wrapper --override-baseurl \
      http://192.168.1.4/tails
      

Windows Camouflage

Enable I2P in the boot loader menu, and enable Windows camouflage via the Tails Greeter checkbox, and then:

  • Tails OpenPGP Applet's context menu should look readable
  • The Tor Browser, Unsafe Browser and I2P Browser should all use the Internet Explorer theme.
  • Vidalia should not start.

Unsafe Browser

Google must be the default, pre-selected search plugin.

Real (non-VM) hardware

[can't-automate]

  • Boot on bare-metal on USB.
  • Boot on bare-metal on DVD.
  • Measure boot time (from syslinux menu the GNOME dektop ready - quickly press enter in the greeter), then on some reference bare metal hardware, and compare with previous version. The new one should not be significantly slower to start.

Documentation

  • The "Tails documentation" desktop launcher should open the getting started page (automate: ticket #8788):
    • in one language to which the website is translated
    • in one language to which the website is not translated (=> English)
  • Browse around in the documentation shipped in the image. Internal links should be fine.

Internationalization

Boot and check basic functionality is working for every supported language. You really have to reboot between each language.

  • The chosen keyboard layout must be applied.
  • The virtual keyboard must work and be auto-configured to use the same keyboard layout as the X session.
  • In the Tor Browser:
    • Disconnect.me must be the default, pre-selected search plugin.
    • the Disconnect.me, Startpage and Wikipedia search plugins must be localized for the supported locales:

      . /usr/local/lib/tails-shell-library/tor-browser.sh
      supported_tor_browser_locales
      

Spellchecking

  • Check that every supported language is listed in the list of languages for spell checking.
  • For a few languages, check the spell checking:
    • Type something in the textarea.
    • Right-click and select a language.
    • Verify that the spelling suggestion are from that language.
  • Once ticket #5962 is fixed, the browser spelling dictionary must be localized (for languages that are supported by our branding extension).

Misc

  • Check that Tails Greeter's "more options" screen displays properly on a display with 600 px height, preferably in a language that's more verbose than English (e.g. French).
  • Check that all seems well during init (mostly that all services start without errors), and that /var/log/syslog seems OK.