Read this document from the branch used to prepare the release.

Changes

Keeping an eye on the changes between released versions is one of the many safeguards against releasing crap.

Packages

Compare the list of bundled packages and versions with the one shipped last time. .packages are usually attached to the email announcing the image is ready.

/usr/bin/diff -u \
    wiki/src/torrents/files/tails-amd64-3.1.packages \
    tails-amd64-3.2.packages \
    | wdiff --diff-input  --terminal

Check the output for:

  • new packages that may cause harm or make the images unnecessarily big
  • packages that could be erroneously removed
  • new versions of software we might not have audited yet (including: does the combination of our configuration with software X version Y+1 achieve the same wished results as with software X version Y?)

Image size

Check the images size has not changed much since the last release.

In a directory with many Tails ISO and USB images:

find \( -iname "tails*.iso" -o -iname "tails*.img" \) \
     -type f -exec ls -l --block-size=M '{}' \; | sort -rhk 5

Reproducibility

This section can not be done by the RM.

  1. Check that the https://tails.boum.org/contribute/calendar/ documents who is the Trusted Reproducer for this release. If this is not the case, ask the RM (this is the only exception to "do not trust anything said by the RM about this process").

  2. Download the ISO and USB images.

  3. Clear-sign the hashes of all products using your OpenPGP key and gzip the output (otherwise the signed text could be mangled at some point in the email chain):

     DEST_DIR=$(mktemp -d)
     sha512sum *.iso *.img \
       | gpg --clear-sign \
       | gzip \
       > "$DEST_DIR/TR-bits.gz"
    
  4. Locate the file generated by the command above. To display its location, execute the following command:

     echo "$DEST_DIR/TR-bits.gz"
    
  5. Send the aforementioned generated file as an attachment to the Trusted Reproducer.

  6. If the Trusted Reproducer is around, ask them:

    • if they've received your email
    • if they could successfully decompress the attachment
    • if they could check the inline signature of the attachment once decompressed
    • if the signed text contains all the information they need

Automated test suite

See the automated test suite documentation.

Running the automated test suite

See setup and usage.

Do point --old-iso to the ISO image of the previous stable release.

Automated test suite migration progress

The manual test suite below either contains tests that cannot be automated, has no automated test implemented yet, or has a test implemented, but it either hasn't been reviewed, had a confirmed pass by someone other than the test author, or has issues. The latter is tracked by subtasks of #10250.

Tor Browser

Miscellaneous functionality

  • Test if uBlock works:
    • The uBlock icon must be visible.
    • Visit a website that normally displays ads, such as https://www.nytimes.com/. The ads should not be displayed and the uBlock icon should display a strictly positive number of blocked elements.

Security and fingerprinting

  • Download the same version of Tor Browser and compare its fingerprint (running on Linux outside of Tails) with Tor Browser in Tails, using at least https://panopticlick.eff.org/. Click "Show full results for fingerprinting" to see the details we're interested in.
    • The exposed User-Agent should match the latest Tor Browser's one.
    • Ignore the result of the "blocking tracking ads" and "blocking invisible trackers" tests, which seem unreliable (we've seen different results for the very same version of Tor Browser).
    • Update the Browser fingerprint section of the known issues if needed.
  • In about:config check that media.peerconnection.enabled is set to false.

Thunderbird

EHLO/HELO

Only perform this test if one of these conditions is met:

  • We're testing a release candidate, such as Tails 7.42~rc1.
  • The release under testing upgrades to a major Thunderbird version. For example, the last Tails release shipped Thunderbird 87.9, and the one you're testing includes Thunderbird 88.1.

To check that the EHLO/HELO SMTP message is not leaking anything at the application level:

  1. Start Thunderbird using the GNOME Applications menu.
  2. Configure the outgoing SMTP server for an email account so it uses STARTTLS on port 587 (Thunderbird will send two EHLO/HELO: one before TLS is initiated; one after; the assumption here is that Thunderbird will send the same both times).
  3. Run sudo tcpdump -n -i lo -w dump while sending an email to capture the packets before Tor encrypts it, then close tcpdump. Note that the packet containing EHLO/HELO will be sent really early, so even if the email failed (e.g. because the mail server doesn't support plaintext SMTP on port 587) we are ok.
  4. Check the dump for the HELO/EHLO message and verify that it only contains 127.0.0.1: sudo tcpdump -A -r dump | grep EHLO

WhisperBack

  • I should be able to send a bug report with WhisperBack.
  • When we receive this bug report on the tails-bugs mailing list, Schleuder tells us that it was sent encrypted.

Virtualization support

  • Test that Tails starts and the browser launches in VirtualBox.

Incremental upgrades

(automate: #18004)

  1. Install from scratch on a USB stick the previous Tails stable release (that is, at this time, the current published one). A system that was upgraded to that version will not do.

  2. Start from that USB stick.

  3. Set an administration password in the Welcome Screen.

  4. Upgrade to the version we're testing:

     sudo sh -c 'sed -i /^TAILS_CHANNEL=/d /etc/os-release &&
                 echo TAILS_CHANNEL=\"test\" >> /etc/os-release' && \
     systemctl --user restart tails-upgrade-frontend.service
    
  5. Once the upgrade has been applied and you're suggested to restart Tails, do that.

  6. Verify that the resulting, upgraded system "works fine":

    • it boots
    • it pretends to be the correct version
    • Tor works fine
    • Tor Browser works fine
    • the Unsafe Browser starts

Tails Verification

  • The goal is to check that Tails Verification works in Tor Browser in the version of Tails we are testing here. Tails Verification only supports verifying the current release so for example, when doing tests for the Tails 3.13 release, we use it in the tentative Tails 3.13 to verify the Tails 3.12 ISO and USB images.

    1. Start the Tails that you are testing.

    2. Start Tor Browser.

    3. Visit https://tails.boum.org/install/download-iso/.

    4. If you already have the ISO image for the last, already released version of Tails, i.e. the one that's advertised in the sidebar of our production website, then click "I already downloaded Tails $N". Otherwise follow the instructions on that web page to download the ISO image.

    5. Install Tails Verification.

    6. Click the Verify Tails $N button.

    7. The verification should be successful.

    8. Repeat for the USB image, except the URL for step (3) is: https://tails.boum.org/install/download/

Torrents

Make sure that the .torrent files we advertise for this release can be used to download the full images:

  • ISO

  • USB image

Real (non-VM) hardware

[can't-automate]

Note that for release candidates, we do not always optimize the ordering of files in the SquashFS, which might make them boot somewhat slower.

On a 64-bit computer that has Secure Boot enabled and that is configured to prefer UEFI:

  1. Freshly install the version of Tails being tested to a USB stick.
  2. Boot this USB stick on bare-metal a first time to trigger re-partitioning. You should the "GNU GRUB" title, as opposed to a syslinux menu.
  3. Boot this USB stick a second time, measuring the boot time (from the GRUB menu until the GNOME desktop is ready -- quickly press ENTER in the Welcome Screen).
  4. Compare with the boot time of the previous Tails version. The new one should not be significantly slower to start.

On a computer configured to prefer legacy BIOS boot:

  1. Freshly install the version of Tails being tested to a USB stick.
  2. Boot this USB stick on bare-metal a first time to trigger re-partitioning. You should see the "SYSLINUX" title in the bootloader, as opposed to GRUB.
  3. Boot this USB stick a second time.
  4. You should see the Welcome Screen appear.

Misc

  • Check that the output of sudo journalctl should seem OK. For a quick overview, something like sudo journalctl -p 0..3 can be helpful. [can't-automate]