Some test results that might be useful to keep are saved.

Changes

Keeping an eye on the changes between released versions is one of the many safeguards against releasing crap.

Source

Compare the to-be-released source code with previous version's one e.g.:

Boot the candidate ISO and find the commit it was build from with the tails-version command.

Then, from the source tree, see the diff:

git diff --find-renames <old tag>..<ISO commit>

e.g. git diff 0.17..06fa1ab80d55c9f29274b7459bd198edb1a8d53d

Result

Compare the list of bundled packages and versions with the one shipped last time. .packages are usually attached to the email announcing the ISO is ready.

/usr/bin/diff -u \
    wiki/src/torrents/files/tails-i386-0.16.packages \
    tails-i386-0.17.packages \
    | wdiff --diff-input  --terminal

Check the output for:

• new packages that may cause harm or make the images unnecessarily big
• packages that could be erroneously removed
• new versions of software we might not have audited yet (including: does the combination of our configuration with software X version Y+1 achieve the same wished results as with software X version Y?)

Image size

Check the image size has not changed much since the last release.

In a directory with many Tails ISO images:

find -iname "tails*.iso" -exec ls -lh '{}' \; | sort -rhk 8

Automated test suite

Our long term goal is to eliminate the manual test suite (except the parts which require real hardware) and have the automated test suite run all our tests. It's design, and how to write new tests, are documented on a dedicated page.

Running the automated test suite

See setup and usage.

Automated test suite migration progress

The manual test suite below either contains tests that cannot be automated, has no automated test implemented yet, or has a test implemented, but it either hasn't been reviewed, had a confirmed pass by someone other than the test author, or has issues. The latter is tracked by tickets prefixed with todo/test_suite:.

Iceweasel

Security and fingerprinting

• Run the tests the TBB folks use.
• Compare the fingerprint of Tails and the latest TBB using at least https://panopticlick.eff.org/
  • The exposed User-Agent should match the latest TBB's one.
  • Update the fingerprint section of the known issues page if needed.
• WebRTC should be disabled:
  • In about:config check that media.peerconnection.enabled is set to false.
  • http://mozilla.github.io/webrtc-landing/, especially the getUserMedia test. It's expected that the audio test works if you agree to share a microphone with the remote website; anything else should fail.
  • http://net.ipcalf.com/ should display ifconfig | grep inet | grep -v inet6 | cut -d" " -f2 | tail -n1
• One should be able to switch identities from the web browser.
• Running getTorBrowserUserAgent should produce the User-Agent set by the installed version of Torbutton, and used in Iceweasel.

Functionality

• Browsing (by IP) a HTTP or HTTPS server on the LAN should be possible.
• Browsing (by IP) a FTP server on the LAN should be possible.

Pidgin

• pidgin: is an IRC session really torified?
• if you are running an IRC server: check there
• else: see if the connection to the IRC server appears in Vidalia connections list
• Check OTR is working.
• Check that IRC is working with the default OFTC profile.
• Check that XMPP is working with a new test profile.
• Check if pidgin doesn't leak too many informations on replying to different CTCP requests:
  • Start Tails, launch pidgin, and join #tails.
  • Also join #tails from the webchat of OFTC on https://webchat.oftc.net/ using another nickname.
  • Try to send /ctcp <Tails_account_nick> COMMAND from the webchat to pidgin:
    • You should get no answer apart for the commands listed in ticket #5823.
    • List of /ctcp commands, see this page:
      • PING
      • VERSION
      • FINGER
      • USERINFO
      • CLIENTINFO
      • TIME

Tor

• The version of Tor should be the latest stable one, which is the highest version number before alpha releases on http://deb.torproject.org/torproject.org/pool/main/t/tor/.
• firewall: is the Tor-enforcement effective?
• check output of iptables -L -n -v
• check output of iptables -t nat -L -n -v
• try connecting to the Internet after unsetting $http_proxy and $HTTP_PROXY using a piece of software that does not obey the GNOME proxy settings, and is not explicitly torified in Tails: unset http_proxy ; unset HTTP_PROXY ; wget --no-proxy http://monip.org (should result in a "Connection refused".
• firewall: is IPv6 traffic blocked?
• check output of ip6tables -L -n
• at a place with working IPv6: try connecting to a known-working IPv6-enabled server on its IPv6 address over TCP and icmp6.
• is /etc/resolv.conf OK both before/after DHCP has been set up? it should always read nameserver 127.0.0.1
• bridge mode should work:
  1. Set up an administrator password.
  2. Enable network configuration in Tails Greeter.
  3. Configure a bridge or a few in Tor Launcher:
     • bridge 198.252.153.59:9001
     • obfs2 198.252.153.59:16492
     • obfs3 198.252.153.59:16493
  4. Use the Internet.
  5. Check with sudo watch "netstat -taupen | grep ESTABLISHED" that the only outgoing direct network connections go to the configured bridges.
• Verify that all destinations reached from an intensive Tails session are tor routers or authorities:
  1. Boot Tails without the network in.
  2. Set up an administration password.
  3. Start dumping your whole session's network activity with sudo tcpdump -n -i any -w dump (or better, do the dump on another machine, or on the host OS if Tails is running in a VM).
  4. Plug the network.
  5. Wait for Tor to be functional.
  6. Save /var/lib/tor/cached-microdesc-consensus out of the VM (it's needed to analyze the network dump later on).
  7. Do a lot of network stuff (why not run do this while doing all the other tests but I2P and the unsafe browser, which would show many false positives?)

  8. Then check all destinations, e.g. by using tshark and the script below:

     # set DUMP to the output of tcpdump above
     DUMP=dump
     # set CONSENSUS to Tor's consensus from the Tails session
     CONSENSUS=cached-microdesc-consensus
     NODES=$(mktemp)
     awk '/^r / { print $6 }' ${CONSENSUS} > ${NODES}
     # Note that these default directory authorities may change! To be
     # sure, check in Tor's source, src/or/config.c:~900
     DIR_AUTHS="
     128.31.0.39
     86.59.21.38
     194.109.206.212
     82.94.251.203
     76.73.17.194
     212.112.245.170
     193.23.244.244
     208.83.223.34
     171.25.193.9
     154.35.32.5
     "
     tshark -r ${DUMP} -T fields -e ip.dst | sort | uniq | \
     while read x; do
         ip_expr=$(echo ${x} | sed -e "s@\.@\\\.@g")
         if echo ${DIR_AUTHS} | grep -qe "${ip_expr}"; then
             continue
         fi
         if ! grep -qe "^${ip_expr}$" ${NODES}; then
             echo "${x} is bad"
         fi
     done
     rm ${NODES}
     

     Note that this script will produce some false positives, like your gateway, broadcasts, etc.

Stream isolation

See our stream isolation design page for details such as port numbers, that are not duplicated here to avoid desynchronization.

Assumptions for the following tests: first, Tor stream isolation features properly do their work; second, our torrc sets the right SocksPort options to implement what we want.

Note: the following commands would advantageously be replaced with the appropriate tcpdump or tshark filters.

• Make sure Claws Mail use its dedicated SocksPort when connecting to IMAP / POP3 / SMTP servers. As root:

  watch -n 0.1 'netstat -taupen | grep claws'
  

• Make sure these use the SocksPort dedicated for Tails-specific applications:

  • htpdate — as root, run:

    service htpdate stop \
       && rm -f /var/run/htpdate/{done,success} \
       && service htpdate start
    

    ... with the following command running as root in another terminal:

    watch -n 0.1 'netstat -taupen | grep curl'
    

  • security check — run tails-security-check with the following command running as root in another terminal:

    watch -n 0.1 'netstat -taupen | grep perl'
    

  • incremental upgrades — run tails-upgrade-frontend-wrapper with the following command running as root in another terminal:

    watch -n 0.1 'netstat -taupen | grep perl'
    

• Make sure iceweasel uses its dedicated SocksPort: quit Iceweasel then start it with the following command running as root in another terminal:

  watch -n 0.1 'netstat -taupen | grep iceweasel'
  

• Make sure other applications use the default system-wide SocksPort:

  • Polipo — run:

    wget https://tails.boum.org/
    

    ... with the following command running as root in another terminal:

    watch -n 0.1 'netstat -taupen | grep polipo'
    

  • Gobby 0.4 — start Gobby 0.4 from the Applications menu and connect to a server, with the following command running as root in another terminal:

    watch -n 0.1 'netstat -taupen | grep gobby'
    

  • Gobby 0.5 — start Gobby 0.5 from the Applications menu and connect to a server (for example gobby.debian.org), with the following command running as root in another terminal:

    watch -n 0.1 'netstat -taupen | grep gobby'
    

  • SSH — run (no need to authenticate the server or to login):

    ssh lizard.tails.boum.org
    

    ... with the following command running as root in another terminal:

    watch -n 0.1 'netstat -taupen | grep -E "connect-proxy|ssh"'
    

  • whois — run:

    whois example.com
    

    ... with the following command running as root in another terminal:

    watch -n 0.1 'netstat -taupen | grep whois'
    

• Make sure a random application run using torify and torsocks uses the default system-wide SocksPort. Run:

  torify /usr/bin/gobby-0.5
  

  ... and connect to a server (for example gobby.debian.org), with the following command running as root in another terminal:

    watch -n 0.1 'netstat -taupen | grep gobby'
  

  Then do the same test for:

  torsocks /usr/bin/gobby-0.5
  

Use of untrusted partitions

• is any local hard-disk swap partition used as swap? boot on a (possibly virtual) machine that has a cleartext swap partition not managed by LVM. This swap partition must not be used by Tails. Run cat /proc/swaps.
• is a persistence volume on a local hard-disk partition used? (Hint: setup a libvirt USB disk with GPT and a partition labeled TailsData, set the removable flag on it, check that tails-greeter proposes to enable persistence. Then remove the removable flag, and check that tails-greeter does not propose to enable persistence anymore.)

Claws

• Check mail over IMAP using:
  • a "clearnet" IMAP server.
  • a hidden service IMAP server (e.g. TorMail, jhiwjjlqpyawmpjx.onion, or Riseup, xyvp43vrggckj427.onion with SSL).
• Send an email using:
  • a "clearnet" SMTP server.
  • a hidden service SMTP server (see above).
• Check that the profile works and is torified:
  1. Send an email using Claws and a non-anonymizing SMTP relay (a SMTP relay that writes the IP address of the client it is relaying email for in the Received header).
  2. Then check that email's headers once received, especially the Received: one.
• Also check that the EHLO/HELO SMTP message is not leaking anything at the application level:
  1. Start Claws using the panel icon.
  2. Disable SSL/TLS for SMTP in Claws (so take precautions for not leaking your password in plaintext by either changing it temporarily or using a disposable account).
  3. Run sudo tcpdump -n -i lo -w dump to capture the packets before Tor encrypts it, then close tcpdump
  4. Check the dump for the HELO/EHLO message and verify that it only contains localhost: tcpdump -A -r dump
  5. Check the Received: and Message-Id fields in the received message: it must not leak the hostname, nor the local IP.

WhisperBack

• can a bug report e-mail be sent?
• is it correctly encrypted?

Time

1. Boot Tails without a network cable connected. (e.g. virsh domif-setlink tails-dev 52:54:00:05:17:62 down.)
2. Set an administration password.

3. set the time to an obviously wrong one:

       date --set="Mon, 01 Mar 2000 15:45:34 - 0800"
   

4. Connect the network cable. (e.g. virsh domif-setlink tails-dev 52:54:00:05:17:62 up)

=> the date should be corrected and Tor/Vidalia should start correctly.

Erase memory on shutdown

• check that memlockd is running
• check that udev-watchdog monitors the right device (ticket #5560)
  • on USB
  • on DVD
• After booting from DVD, remove Tails boot medium and check that the memory erasure process is started (Loading new kernel, at least). ticket #5472
• After booting from USB, remove Tails boot medium and check that the memory erasure process is started (Loading new kernel, at least).

Root access control

• Check you can login as root with su neither with the amnesia password nor with the live one.
• Check that the $TAILS_USER_PASSWORD variable, if still existing in the system environment after the boot has finished, does not contain the clear text password.

Virtualization support

• Test that Tails starts and the browser launches in VirtualBox.

I2P

• Make sure that I2P is up-to-date, at least if the changelogs mention that security critical bugs were fixed.
• Check that "Applications -> Internet -> I2P" works:
  • You get the "Starting I2P..." pop-up.
  • The router console opens in Iceweasel upon success.
  • You get the "I2P failed to start" pop-up on failure (e.g. no network so tordate failed).
• Check that I2P connects to the network:
  • Go to http://127.0.0.1:7657/i2ptunnelmgr
  • You should get "Network: Hidden" in the "General" section.
  • The numbers in the "Peers" section of the sidebar should be non-zero.
• Check that you can reach some eepsites within Iceweasel, like http://i2p-projekt.i2p and http://forum.i2p.
• Check that you can connect to the I2P IRC server through Pidgin and the preconfigured IRC account on 127.0.0.1.

Git

• clone a repository over git://
• clone a repository over https://
• clone a repository over SSH

SSH

• Connecting over SSH to a server on the Internet should work (and appear in Vidalia's connections list).
• Connecting (by IP) over SSH to a server on the LAN should work.
• Connecting to a sftp server on the Internet using GNOME's "Connect to a server" should work.

APT

 grep -r deb.tails.boum.org /etc/apt/sources.list*

• Make sure the Tails repository suite in matching the release tag (for example the release version number) is in APT sources.
• Make sure the Tails repository unversioned suites (e.g. testing, stable and devel) are not in APT sources.

Incremental upgrades

• List the versions from which an upgrade paths to this one is described. In the stable or testing branch:

  git grep -l "  version: '0.23'" wiki/src/upgrade/
  

• For each description file, open it and verify if it allows incremental upgrade or only full upgrade.

• For each previous version from which an upgrade paths is described, install it and try to upgrade:

  • For every incremental upgrade path: make sure the resulting updated system "works fine" (boots and pretends to be the correct version).
  • For upgrade paths that only propose a full upgrade: make sure the user is told to do a manual upgrade.

  If the IUKs and update-description files have been published on the alpha channel already (see https://archive.torproject.org/amnesia.boum.org/tails/alpha/):

  echo 'TAILS_CHANNEL="alpha"' | sudo tee --append /etc/os-release && \
  tails-upgrade-frontend-wrapper
  

  Else, use a local test setup:

  • A web server on the LAN.
  • A copy of wiki/src/update from the stable or testing branch, for example in /var/www/tails/update/v1/Tails/0.14~rc2/i386/stable/updates.yml

  • A copy of the iuk directory of our HTTP mirrors, for example in /var/www/tails/stable/iuk/Tails_i386_0.14-rc2_to_0.14.iuk.

    To synchronize your local copy:

    usewithtor rsync -rt --progress rsync.torproject.org::amnesia-archive/tails/stable/iuk/ /var/www/tails/stable/iuk/
    

  • Patch /etc/hosts in Tails to point to your web server:

    echo "192.168.1.4    dl.amnesia.boum.org" | sudo tee --append /etc/hosts
    

  • Call the upgrader must be called, from inside the system to upgrade, with every needed option to use the local web server rather than the online one, for example:

    DISABLE_PROXY=1 SSL_NO_VERIFY=1 \
    tails-upgrade-frontend-wrapper --override-baseurl \
    http://192.168.1.4/tails
    

Windows Camouflage

Enable Windows XP camouflage via the Tails Greeter checkbox and:

• Tails OpenPGP Applet's context menu should look readable
• iceweasel should use a Internet Explorer theme

Unsafe Web Browser

• On start, if no DNS server was configured in NetworkManager (e.g. if there's no network connection), there must be an error.
• Once started, check that:
  • it has no scary red theme when Windows Camouflage is activated.
  • the iceweasel instance runs as the clearnet user.
  • it has no proxy configured.
  • no extensions are installed.
  • there are no bookmarks.
• On exit, check that:
  • make sure that its chroot gets properly teared down on exit (there should be nothing mounted inside /var/lib/unsafe-browser).

Real (non-VM) hardware

[can't-automate]

• Boot on bare-metal on USB.
• Boot on bare-metal on DVD.
• Measure boot time (from syslinux menu the GNOME dektop ready - quickly press enter in the greeter), then on some reference bare metal hardware, and compare with previous version. The new one should not be significantly slower to start.

Documentation

• Check that links to the online website (Mirror:) at the bottom of bundled static web pages (/usr/share/doc/tails/website/) are working. Else, it probably means the wiki was not built with a recent enough ikiwiki.
• Browse around in the documentation shipped in the image. Internal links should be fine.

Internationalization

Boot and check basic functionality is working for every supported language.

• The chosen keyboard layout must be applied.
• The virtual keyboard must work and be auto-configured to use the same keyboard layout as the X session.

• The Startpage search engine must be localized for the languages we ship a searchplug for:

  find /etc/iceweasel/searchplugins/locale/ -iname startpage-*.html
  

• The Wikipedia search engine must be localized for all languages.

Spellchecking

• Check that every supported language is listed in the list of languages for spell checking.
  • Visit https://translate.google.com/.
  • Right-click and choose "Check spelling".
  • Right-click and check the list of available languages.
• For a few languages, check the spell checking:
  • Type something in the textarea.
  • Right-click and select a language.
  • Verify that the spelling suggestion are from that language.
• Once ticket #5962 is fixed, the browser spelling dictionary must be localized (for languages that are supported by our branding extension).

Misc

• Check that Tails Greeter's "more options" screen displays properly on a display with 600 px height.
• Check that all seems well during init (mostly that all services start without errors), and that dmesg seems ok.
• MAT should be able to clean a PDF file, such as: http://examples.itextpdf.com/results/part3/chapter12/pdf_metadata.pdf
• Browsing (by hostname) a FTP server on the Internet should be possible using GNOME's Connect to server feature (once ticket #6096 is fixed)
• The Tails signing key shipped should be up-to-date (that is, neither it nor one its subkeys must have expired, or be about to expire any time soon).
  • gpg --list-keys 1202821CBE2CD9C1
• The "Report an error" desktop launcher should open the support page (automate: ticket #6904).
• One should be able to refresh the GnuPG keyring in Seahorse (once ticket #7051 is resolved).