1. Requirements
  2. Environment
    1. Version numbers
    2. Other variables
  3. Pre-freeze
    1. Coordinate with Debian security updates
  4. Sanity check
  5. Freeze
    1. Major release
    2. Bugfix release
    3. Common steps for bugfix and major releases
  6. Update included files
    1. Upgrade Tor Browser
    2. Upgrade Tor Browser AppArmor profile
    3. Upgrade bundled binary Debian packages
    4. Upgrade custom packages for VeraCrypt integration
    5. Update PO files
  7. When preparing an actual release
    1. Major release
    2. Bugfix release
  8. Update other base branches
  9. Update more included files
    1. Changelog
    2. Included website
    3. Signing key downloaded by the Upgrader
    4. Website translations
  10. Call for translation
  11. Enable OpenPGP signing
  12. Build the almost-final images
  13. Tag the release in Git
  14. Prepare the versioned APT suites
  15. Build images
    1. Sanity check
    2. SquashFS file order
    3. Build the final images
    4. Verify that Jenkins reproduced your images
    5. Initialize the website release branch
  16. Generate the OpenPGP signatures and Torrents
  17. Prepare incremental upgrades
    1. Prepare the environment
    2. Sanity checks
    3. Build the Incremental Upgrade Kits locally
    4. ISO history
    5. Build the Incremental Upgrade Kits on Jenkins
    6. Verify that Jenkins reproduced your IUKs
    7. Prepare upgrade-description files
    8. Prepare the image description file for Tails Verification
  18. Done with OpenPGP signing
  19. Upload images
    1. Sanity check
    2. Publish the ISO, IMG, and IUKs over HTTP
    3. Announce, seed and test the Torrents
  20. Testing
  21. Prepare announcements and blog posts
    1. If preparing a final release
    2. If preparing a release candidate
    3. In any case
    4. Draft the Tor blog post
  22. Go wild!
    1. Wait for the HTTP mirrors to catch up
    2. Sanity checks
    3. Push
    4. Bug tracker
    5. Twitter
    6. Publish the Tor blog post
    7. Amnesia news
  23. Prepare for the next development cycle
    1. If you just released a final release
    2. If you just released an RC
  24. Related pages

See the release schedule.

Read the remainder of this document from the branch used to prepare the release after having merged the current master branch into it!

Requirements

To release Tails you'll need some packages installed:

  • tidy mktorrent transmission-cli
  • squashfs-tools 1:4.4-1+0.tails1 from our custom iukbuilder-stretch APT suite.
  • iuk dependencies
  • perl5lib dependencies
  • po4a from Stretch: the version in testing/sid extracts Markdown headings in a different way, which makes tons of strings fuzzy.

Environment

To be able to copy'n'paste the code snippets found on this page, you need to set a bunch of environment variables.

Unless the release process explicitly instructs you to change the value of one such variable, treat it as a constant: else, inconsistency will surely arise, which can cause trouble later on.

Version numbers

Note:

  • Regarding version numbers, what follows supports just fine the case when we do something else than alternating bugfix and major releases consistently. For example, if the next two releases are bugfix ones, do not set $NEXT_PLANNED_MAJOR_VERSION to one of these bugfix releases. Instead, set it to the version number of the next major release.
  • The $NEXT*VERSION constants are used only for two types of operations: preparing upgrade-description files and adding changelog entries. This two types of operations have to be consistent with each other: for example, if one adds a dummy entry for version X in a changelog, an UDF must exist for version X as well… hence the use of shared constants to encode the values that must be the same on both sides :)

Export the following environment variables:

  • version numbers (see release schedule):

      export VERSION=$(dpkg-parsechangelog -SVersion)
  export TAG=$(echo "${VERSION:?}" | sed -e 's,~,-,')
  export PREVIOUS_VERSION=$(dpkg-parsechangelog --offset 1 --count 1 -SVersion)
  export PREVIOUS_TAG=$(echo "${PREVIOUS_VERSION:?}" | sed -e 's,~,-,')

  • NEXT_PLANNED_MAJOR_VERSION: set to the version number of the next major Tails release; if you're preparing a RC for a major release, use that major release; otherwise, use whatever the next planned major release is

  • SECOND_NEXT_PLANNED_MAJOR_VERSION: if you're preparing the RC for a major release, set this to the version number of the second next major Tails release; e.g. if preparing the RC for the 3.9 major release, then set this to 3.12 (3.9 is the next major release, 3.10 and 3.11 are bugfix releases, 3.12 is a major release).
  • NEXT_PLANNED_BUGFIX_VERSION: set to the version number of the next scheduled bugfix Tails release
  • NEXT_POTENTIAL_EMERGENCY_VERSION: set to the version number we'll give to the next emergency release if we have to put one out
  • NEXT_STABLE_CHANGELOG_VERSION: if $NEXT_PLANNED_BUGFIX_VERSION is the next scheduled release, use it; otherwise, use $NEXT_POTENTIAL_EMERGENCY_VERSION

Other variables

Also export the following environment variables:

  • MAJOR_RELEASE: set to 1 if preparing a major release or a release candidate for a major release, to 0 otherwise
  • ISOS: the directory where one stores tails-amd64-* sub-directories like the ones downloaded with BitTorrent.
  • ARTIFACTS: the directory where build artifacts (e.g. the .packages file) land.
  • MASTER_CHECKOUT: a checkout of the master branch of the main Tails Git repository.
  • RELEASE_BRANCH=$(if [ "$MAJOR_RELEASE" = 1 ]; then echo -n testing; else echo -n stable; fi)
  • RELEASE_CHECKOUT: a checkout of the branch of the main Tails Git repository used to prepare the release (stable or testing).
  • TAILS_SIGNATURE_KEY=A490D0F4D311A4153E2BB7CADBB802B258ACD84F
  • TAILS_SIGNATURE_KEY_LONG_ID=$(echo "${TAILS_SIGNATURE_KEY:?}" | perl -nE 'say substr($_, -17)')
  • DIST: either 'alpha' (for RC:s) or 'stable' (for actual releases)
  • export WEBSITE_RELEASE_BRANCH="web/release-${TAG:?}"
  • export IUKS_DIR="${ISOS:?}/iuks/v2"
  • export IUKS_HASHES="${IUKS_DIR:?}/to_${VERSION}.sha256sum"

Pre-freeze

The release manager role documentation has more tasks that should be done early enough.

Coordinate with Debian security updates

See Debian security updates.

Sanity check

Visit the Jenkins RM view and check that the jobs for the release branch have good enough results.

Freeze

Major release

If we are at freeze time for a major release:

  1. Merge the master Git branch into devel:

     git checkout devel && git fetch origin && git merge --no-ff origin/master

  2. Merge each APT overlay suite listed in the devel branch's config/APT_overlays.d/ into the devel APT suite.

  3. Merge the devel Git branch into the testing one:

     git checkout testing && git merge devel

    ... and check that the resulting config/APT_overlays.d/ in the testing branch is empty.

  4. Hard reset the testing custom APT suite to the current state of the devel one.

  5. Freeze the time-based APT repository snapshots that shall be used during the freeze.

  6. Make it so the time-based APT repository snapshots are kept around long enough, by bumping their Valid-Until to 10 days after the next major release (the one after the one you're preparing)'s scheduled date: time-based snapshots

Bugfix release

If we are at freeze time for a bugfix release:

  1. Merge the master Git branch into stable:

     git checkout stable && git fetch && git merge --no-ff origin/master

  2. Merge each APT overlay suite listed in the stable branch's config/APT_overlays.d/ into the stable APT suite.

Common steps for bugfix and major releases

Reset the release branch's config/base_branch:

    echo "${RELEASE_BRANCH:?}" > config/base_branch && \
       git commit config/base_branch \
           -m "Restore ${RELEASE_BRANCH:?}'s base branch."

Bootstrap manual testing coordination:

  1. Create a pad.
  2. Copy the manual test suite into it.
  3. Send the pad URL in an encrypted mail to the manual testers: tails-manual-testers@boum.org

Update included files

Upgrade Tor Browser

See the dedicated page: tor-browser

Upgrade Tor Browser AppArmor profile

See the dedicated page: tor-browser AppArmor patch

Upgrade bundled binary Debian packages

Skip this section if you are preparing a bugfix release.

The goal here is to make sure the bundled binary Debian packages contain up-to-date localization files, so:

  • If you are preparing a release candidate, build at least the packages that change user-visible strings, so that translators can use the RC to check the status of their work and identify what's left to do.
  • If you are preparing a major release, build at least the packages that got translation updates since the RC: we've sent a call for translation while releasing the RC so the least we can do is to incorporate the work that ensued into our final release :)

For each bundled Debian package, cd into the package's root directory (e.g. a checkout of the whisperback repository), import translations from Transifex and sanity-check them:

cd whisperback && \
git checkout master && \
git pull && \
"${RELEASE_CHECKOUT:?}"/import-translations && \
"${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/lint_po

Then, for every PO file that has issues:

  1. Rollback changes to that file: git checkout po/LL.po
  2. Run lint_po again. It should pass this time.

And finally, commit:

git add po && git commit \
    -m "Update POT and PO files, pull updated translations from Transifex."

Then see the relevant release processes, and upload the packages to the release branch's custom APT suite:

Upgrade custom packages for VeraCrypt integration

See the dedicated page: veracrypt

Update PO files

Pull updated translations for languages translated in Transifex, refresh the code PO files, and commit the result, including new PO files:

cd "${RELEASE_CHECKOUT:?}" && \
./import-translations  && \
./refresh-translations && \
./submodules/jenkins-tools/slaves/lint_po && \
git add po && git commit -m 'Update PO files.'

If lint_po complains:

  • rollback the offending PO files and retry; worst case, delete it
  • send a note to tails-l10n@boum.org [public] so that they get in touch with whoever can fix them.

When preparing an actual release

If we're about to prepare the images for a final (non-RC) release, then follow these instructions:

Major release

Merge each APT overlay suite listed in the testing branch's config/APT_overlays.d/ into the testing custom APT suite.

Bugfix release

For bugfix releases, we generally do not put any RC out, so freeze time is the same as preparing the actual release. Hence, the following steps have already been done above, and this section is a noop in the general case.

Merge each APT overlay suite listed in the stable branch's config/APT_overlays.d/ into the stable custom APT suite.

Update other base branches

  1. Merge the release branch into devel following the instructions for merging base branches.

  2. Thaw, on the devel branch, the time-based APT repository snapshots that were used during the freeze. It's fine if that results in a no-op (it depends on how exactly previous operations were performed).

  3. Merge devel into feature/bullseye (if it exists), without following the instructions for merging base branches. (For now feature/bullseye is handled as any other topic branch forked off devel: its base branch is set to devel.) If the merge conflicts don't look like something you feel confident resolving properly, abort this merge and let the Foundations Team know.

  4. Ensure that the release, devel and feature/bullseye (if it exists) branches have the expected content in config/APT_overlays.d/: e.g. it must not list any overlay APT suite that has been merged already.

  5. Push the modified branches to Git:

     git push origin                          \
    "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
    $(if git describe feature/bullseye >/dev/null 2>&1; then echo feature/bullseye:feature/bullseye ; fi) \
    devel:devel

Update more included files

Changelog

Remove the placeholder entry for next release in debian/changelog, and then:

git checkout "${RELEASE_BRANCH:?}" && \
DEBEMAIL='tails@boum.org' DEBFULLNAME='Tails developers' \
./release ${VERSION:?} ${PREVIOUS_TAG:?}

This populates the Changelog with the Git log entries.

Then, launch an editor for the needed cleanup of the result:

dch -e

Changelog entries can be dispatched into those usual sections:

  * Major changes

  * Security fixes

  * Bugfixes

  * Minor improvements and updates

  * Build system

  * Test suite

Then, gather other useful information from:

  • every custom bundled package's own Changelog (Tails Installer, etc.)
  • the diff between the previous version's .packages file and the one from the to-be-released images; look for:
    • security fixes
    • new upstream releases of applications mentioned in features
    • new upstream releases of other important components such as the Linux kernel
  • the Completed for the next release Redmine view.

Finally, sanity check the version and commit:

if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
else
    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
fi

Included website

Merge master

Merge master into the branch used for the release:

git fetch origin && git merge origin/master

version number

If preparing a RC, skip this part.

In the branch used to build the release, update the wiki/src/inc/* files to match the version number and date of the new release. Set the date at least 24 hours in the future! Between tests and mirror synchronization, the build will not be released on the same day. Try to make sure it matches the date of the future signature.

RELEASE_DATE='2015-11-03'

echo "${VERSION:?}"      > wiki/src/inc/stable_amd64_version.html && \
echo -n "${RELEASE_DATE:?}" > wiki/src/inc/stable_amd64_date.html && \
for type in img iso; do
   basename="tails-amd64-${VERSION:?}"
   filename="${basename:?}.${type:?}"
   echo "TZ=UTC gpg --no-options --keyid-format long --verify ${filename:?}.sig ${filename:?}" \
        > wiki/src/inc/stable_amd64_${type:?}_gpg_verify.html && \
   echo "http://dl.amnesia.boum.org/tails/stable/${basename:?}/${filename:?}" \
        > wiki/src/inc/stable_amd64_${type:?}_url.html && \
   echo "https://tails.boum.org/torrents/files/${filename:?}.sig" \
        > wiki/src/inc/stable_amd64_${type:?}_sig_url.html && \
   echo "https://tails.boum.org/torrents/files/${filename:?}.torrent" \
        > wiki/src/inc/stable_amd64_${type:?}_torrent_url.html
done && \
./build-website --rebuild && \
git commit wiki/src/inc/ -m "Update version and date for ${VERSION:?}."

Signing key downloaded by the Upgrader

TMP_GNUPG_HOME=$(mktemp -d)
gpg --homedir "${TMP_GNUPG_HOME:?}" --import wiki/src/tails-signing.key && \
gpg --homedir "${TMP_GNUPG_HOME:?}" --export-options export-minimal \
    --armor --export "${TAILS_SIGNATURE_KEY:?}" \
    > wiki/src/tails-signing-minimal.key && \
git commit wiki/src/tails-signing-minimal.key \
    -m "Update signing key used by the Upgrader"
rm -rf "${TMP_GNUPG_HOME:?}"

Website translations

Refresh the website PO files and commit the ones corresponding to pages that were added or changed accordingly to changes coming with the new release. This e.g. ensures that the RC call for translation points translators to up-to-date PO files:

./build-website && \
git add wiki/src && \
git commit -m 'Update website PO files.'
git push origin "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}"

Call for translation

If at freeze time for a major release, send a call for translations to tails-l10n, making it clear what Git branch the translations must be based on, and what are the priorities.

To get a list of changes on the website:

git diff --stat ${PREVIOUS_TAG:?}.. -- \
    wiki/src/'*'.{mdwn,html} \
    ':!wiki/src/blueprint*' \
    ':!wiki/src/contribute*' \
    ':!wiki/src/inc' \
    ':!wiki/src/news*' \
    ':!wiki/src/security*'

Enable OpenPGP signing

If you have an OpenPGP smart card

If you have an OpenPGP smart card (i.e. if you are one of the usual release managers) go fetch it. Remember to only plug it when needed! A pro tip is to never plug it unless prompted which gpg will do for you.

Otherwise: importing the signing key

This is only relevant when the master key has been reassembled, e.g. for signing a Tails emergency release where none of the usual release managers are available.

You should never import the Tails signing key into your own keyring, and a good practice is to import it to a tmpfs to limit the risks that the private key material is written to disk:

export GNUPGHOME=$(mktemp -d)
sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
sudo chmod 0700 "${GNUPGHOME:?}"
gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our signatures, like the defaults we set in Tails:

cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"

Build the almost-final images

  1. Build ISO and USB images from the release branch, with $TAILS_BUILD_OPTIONS set like this:
    • Set defaultcomp, so we can more accurately optimize our SquashFS file ordering.
    • Do not set keeprunning nor rescue.
  2. Carefully read the build logs to make sure nothing bad happened.
  3. Keep the resulting build artifacts until the end of this release process.

  4. Record where the manifest of needed packages is stored:

     export BUILD_MANIFEST=XXX ; \
 [ -f "${BUILD_MANIFEST:?}" ] || echo "ERROR: BUILD_MANIFEST is incorrect"
 echo "${BUILD_MANIFEST:?}" | grep -E -qs '\.build-manifest$' \
    || echo "ERROR: BUILD_MANIFEST does not have the .build-manifest extension"

Tag the release in Git

git tag -u "${TAILS_SIGNATURE_KEY:?}" \
  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
git push origin "${TAG:?}" "${RELEASE_BRANCH:?}"

(Pushing the tag is needed so that the APT repository is updated, and the Tails APT configuration works at build and boot time. It might be premature, as testing might reveal critical issues, but this is a signed tag, so it can be overridden later. Yes, there is room for improvement here.)

XXX: From this push of a tag, the builds in Jenkins fail because we prevent it to continue if the last changelog entry is unreleased but corresponds to an existing tag. There are workarounds we need to decide and implement.

Prepare the versioned APT suites

  • Prepare the versioned APT suite in our custom APT repository.

  • Prepare tagged snapshots of upstream APT repositories:

        ./bin/tag-apt-snapshots "${BUILD_MANIFEST:?}" "${TAG:?}"

    Note:

    • This command can take a while (about a dozen minutes).
    • It's expected that the packages that were pulled from our custom APT repository are listed under "some packages were not found anywhere" (because we are currently not using time-based snapshots for our custom APT repository). However, no other package should be on that list. Now, we have a "safety" net, in case you don't notice such a problem: if other packages are missing, the next build (that will use the newly created partial, tagged APT repository) will fail.

Build images

Sanity check

Verify that the Tor Browser release used in Tails still is the most recent. Also look if there's a new -buildX tag (e.g. tor-browser-60.3.0esr-8.0-1-build1) for the Firefox version the Tor Browser we want to ship is based on in these Git repositories:

A new tag may indicate that a new Tor Browser release or rebuild is imminent.

Better catch this before people spend time doing manual tests.

SquashFS file order

  1. Install the almost final USB image to a USB stick.
  2. Boot this USB stick a first time to trigger re-partitioning.
  3. Shut down this Tails.
  4. Boot this USB stick on bare metal again.
  5. Add profile to the kernel command-line.
  6. Login with the default settings in the Welcome Screen (e.g. do not configure an Administration Password).
  7. Wait for the "Tor is ready" notification.
  8. Start Tor Browser.
  9. A few minutes later, once the boot-profile process has been killed, retrieve the new sort file from /var/log/boot-profile.
  10. Backup the old sort file: cp config/binary_rootfs/squashfs.sort{,.old}
  11. Copy the new sort file to config/binary_rootfs/squashfs.sort.

  12. Remove runtime-generated files that don't exist in the rootfs, in order to avoid confusing noise in the build output:

        perl -ni -E 'chomp; say unless m{(?:
           [.]pyc\s+\d+\z
        | \Alib/live/mount/medium/live/(?:filesystem[.]squashfs|initrd[.]img)\s
        | \Alib/live/mount/overlay/rw/etc/fstab\s
        | \Alib/live/mount/overlay/rw/etc/console-setup/cached_\S+[.](?:gz|sh)\s
        | \Alib/live/mount/overlay/rw/etc/machine-id\s
        | \Alib/live/mount/overlay/rw/etc/network/interfaces\s
        | \Alib/live/mount/overlay/rw/var/log/wtmp\s
        | \A(?:lib/live/mount/overlay/rw/)?etc/apparmor[.]d/cache/[.]features\s
        | \A(?:lib/live/mount/overlay/rw/)?etc/(?:group|gshadow|passwd|shadow)-\s
        | \A(?:lib/live/mount/overlay/rw/)?etc/resolv-over-clearnet[.]conf\s
        | \A(?:lib/live/mount/overlay/rw/)?etc/skel/[.]config/autostart/end-profile[.]desktop\s
        | \Arun/
        | \Avar/lib/AccountsService/users/Debian-gdm\s
        | \Avar/lib/gdm3/[#]\d+\s
        | \Avar/log/live/config[.]pipe\s
    )}xms' config/binary_rootfs/squashfs.sort

  13. Remove the bits about kill-boot-profile at the end: they're only useful when profiling the boot.

  14. Inspect the Git diff (including diff stat), apply common sense:

     diff -NaurB \
     <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort.old | sort ) \
     <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort     | sort ) \
     | less

  15. git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort

  16. Clean up: rm -f config/binary_rootfs/squashfs.sort.old

Build the final images

Then all included files should be up-to-date and the versioned APT suite should be ready, so it is time to:

  1. Mark the version as "released" in the changelog:

     dch --release --no-force-save-on-release --maintmaint && \
 git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog

  2. Export SOURCE_DATE_EPOCH:

     export SOURCE_DATE_EPOCH=$(date --utc --date="$(dpkg-parsechangelog --show-field=Date)" '+%s')

  3. tag the release again, with all included files in:

     git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
         -m "tagging version ${VERSION:?}" "${TAG:?}" && \
 git push --force origin "${TAG:?}" && \
 git push origin "${RELEASE_BRANCH:?}"

    Note: for Jenkins to build the release you must push the release branch with its tip tagged. I.e. if you deviate from the above commands by e.g. committing a commit in between git tag and the first git push then Jenkins won't build from the tag -- please avoid that!

  4. build the final images! Do not set keeprunning nor rescue in $TAILS_BUILD_OPTIONS. Our build system will apply the correct compression settings automatically so don't bother setting it yourself.

  5. Compare the new build manifest with the one from the previous, almost final build:

     diff -Naur \
    "${BUILD_MANIFEST:?}" \
    "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"

    They should be identical, except that the debian-security serial might be higher.

  6. To ensure we publish the final build's .build-manifest, run:

     export BUILD_MANIFEST="${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"

Verify that Jenkins reproduced your images

to verify that Jenkins reproduced your images:

  1. Visit the URL printed by this command:

    echo "https://jenkins.tails.boum.org/job/build_Tails_ISO_${RELEASE_BRANCH}/"

  2. Find the job (probably the last one) and make sure the ISO and USB images built by Jenkins have the same hash (in the .shasum file) as the images you built.

  3. Then:

    • If the ISO and USB images hashes match: yay, we're good to go! The .build-manifest may differ — that's OK.

      Set the $MATCHING_JENKINS_IMAGES_BUILD_ID environment variable to the ID of this job (an integer).

    • If there is a hash mismatch for one of the images: ouch! Now we are in a tricky situation: on the one hand it seems like a poor idea to block users from benefiting from this release's security updates, but on the other hand the failure might imply that something nefarious is going on. At this stage, no matter what, immediately fetch Jenkins' image, compare it with your, and try to rule out build system compromise:

       sudo diffoscope \
     --text diffoscope.txt \
     --html diffoscope.html \
     --max-report-size 262144000 \
     --max-diff-block-lines 10000 \
     --max-diff-input-lines 10000000 \
         path/to/your/tails-amd64-${VERSION:?}.iso \
         path/to/jenkins/tails-amd64-${VERSION:?}.iso

      Do the same for the USB image as well.

      Then carefully investigate the diffoscope report:

      • If you cannot rule out that the difference is harmful: let's take a step back; we might be compromised, so we are in no position to release. Halt the release, involve the rest of tails@boum.org, and then try to re-establish trust in all build machines and infra involved, etc. Have fun!

      • Otherwise, if the change is definitely harmless:

        • If the source of non-determinism is identified quickly and is easy and fast to fix, and the QA of the current images has not gone very far (so at least that time is not wasted), then you should consider abandoning the current version, and immediately start preparing an emergency release with:

          • the reproducibility fix,
          • a new changelog entry,
          • adjustments to the release notes so they are re-purposed for this emergency release (the abandoned release gets none, since it effectively never will be released publicly).

        • Otherwise, if the fix looks time-consuming or difficult, let's release anyway. But let's add a known issue about "This Tails release does not build reproducibility" to the release notes, linking to the ticket where the nature of the reproducibility failure is clearly described.

Initialize the website release branch

From now on, we don't want to push new commits on $RELEASE_BRANCH until the new release is out. Otherwise, this would break its build and the build of every branch based on it, which would effectively block other development work. So the final steps towards publishing the release will be done in a new, dedicated branch.

If preparing anything but a final release (e.g. an alpha, beta or RC):

    git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" origin/master && \
    git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

Else, if preparing a final release:

    git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" "${TAG:?}" && \
    git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

Generate the OpenPGP signatures and Torrents

Create a directory with a suitable name, go there, move the built images to this brand new directory, generate detached OpenPGP signatures for the images to be published (in the same directory as the images and with a .sig extension), then go up to the parent directory, create a .torrent file and check the generated .torrent files metadata:

mkdir -p "${ISOS:?}/tails-amd64-${VERSION:?}" && \
for type in iso img ; do
   cd "${ISOS:?}/tails-amd64-${VERSION:?}" && \
   mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.${type:?}" . && \
   gpg --armor --default-key "${TAILS_SIGNATURE_KEY:?}" --detach-sign *".${type:?}" && \
   rename 's,\.asc$,.sig,' *.asc && \
   tmp="$(mktemp -d)" && \
   mkdir -p "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
   cd "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
   for x in "${ISOS:?}/tails-amd64-${VERSION:?}"/*.${type:?}*; do
       ln -s ${x} .
   done && \
   mktorrent \
      -o "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}.torrent" \
      -a 'udp://tracker.torrent.eu.org:451'   \
      -a 'udp://tracker.coppersurfer.tk:6969' \
      "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
   transmission-show "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}.torrent" && \
   cd - && \
   rm -rf "${tmp:?}"
done

Lastly, let's set some variables to be used later:

ISO_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso"
ISO_SHA256SUM="$(sha256sum "${ISO_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
ISO_SIZE_IN_BYTES="$(stat -c %s "${ISO_PATH:?}")"
IMG_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.img"
IMG_SHA256SUM="$(sha256sum "${IMG_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
IMG_SIZE_IN_BYTES="$(stat -c %s "${IMG_PATH:?}")"

Prepare incremental upgrades

Since Tails 4.2, we use a new upgrade scheme, which fundamentally changes what the source version number of an upgrade means: it's now the version that was initially installed and not the currently running version. If this is news to you, see:

The main practical implications at release time are:

  • The Release Manager has to publish more IUKs than they used to. But they can now publish IUKs (reproducibly) built on Jenkins, instead of having to upload those they've built locally.

  • The Release Manager has to sign more UDFs than they used to.

  • Computing $IUK_SOURCE_VERSIONS is now straightforward enough that it was automated :)

Prepare the environment

Compute the list of initial version install to build IUKs for:

cd "${RELEASE_CHECKOUT:?}" && \
export IUK_SOURCE_VERSIONS=$(./bin/iuk-source-versions ${VERSION:?})
echo "${IUK_SOURCE_VERSIONS?:}"

Even if it's computed automatically, remember to store it in the file holding environment variables, it will be used in various places below.

Sanity checks

Check that you have the correct version of squashfs-tools installed:

[ "$(dpkg-query --showformat '${Version}\n' --show squashfs-tools)" \
  = '1:4.4-1+0.tails1' \
] || echo 'ERROR! Your squashfs-tools is not the required version, so any generated IUKs will *not* be reproducible!'

Check that you have the password for https://iso-history.tails.boum.org (it will be used by wrap_tails_create_iuks below, in order to download any ISO that is not present locally yet):

if ! grep -F --line-regexp -qs 'machine iso-history.tails.boum.org' ~/.netrc; then
    echo "ERROR! Add a section for iso-history.tails.boum.org' to your ~/.netrc"
    echo "The corresponding login and password are in the RMs' keyringer."
fi

Build the Incremental Upgrade Kits locally

(
   set -eu
   WORK_DIR=$(mktemp -d)
   TAILS_REMOTE="$(git -C "$RELEASE_CHECKOUT" remote get-url origin)"
   PUPPET_TAILS_REMOTE=$(echo -n "$TAILS_REMOTE" | perl -p -E 's,:tails(:?[.]git)?\z,:puppet-tails,')
   cd "$WORK_DIR"
   git clone "$PUPPET_TAILS_REMOTE"
   ./puppet-tails/files/jenkins/slaves/isobuilders/wrap_tails_create_iuks \
       --tails-git-remote "file://${RELEASE_CHECKOUT}/.git"             \
       --tails-git-commit "$TAG"                                      \
       --source-date-epoch "$SOURCE_DATE_EPOCH"                       \
       --local-isos-dir "$ISOS"                                       \
       --tmp-dir "${TMPDIR:-/tmp}"                                        \
       --output-dir "$IUKS_DIR"                                       \
       --source-versions "$IUK_SOURCE_VERSIONS"                       \
       --new-version "$VERSION"                                       \
       --verbose
   cd "$IUKS_DIR"
   sha256sum Tails_amd64_*_to_${VERSION}.iuk > "$IUKS_HASHES"

)

This command takes a long time. In parallel, while it is running, you can follow the next 2 steps:

  • Push the images to our ISO history git-annex repo
  • Build the Incremental Upgrade Kits on Jenkins

ISO history

Push the released ISO and USB images and their artifacts (.buildlog, .build-manifest, and .packages files) to our Tails ISO history git-annex repo, so that:

  • The Jenkins build_IUKs job can fetch them.
  • Our isotesters can fetch them from there for their testing.

How to do so is described in the ISO_history.mdwn document in the RM team's Git repo.

Then, wait (a few minutes) until the images appear on https://iso-history.tails.boum.org/.

Build the Incremental Upgrade Kits on Jenkins

  1. On https://jenkins.tails.boum.org/job/build_IUKs/build?delay=0sec, fill the form with these values:

    • TAILS_GIT_COMMIT: the value of $TAG in your release environment
    • SOURCE_DATE_EPOCH: the value of $SOURCE_DATE_EPOCH in your release environment
    • SOURCE_VERSIONS: the value of $IUK_SOURCE_VERSIONS in your release environment
    • NEW_VERSION: the value of $VERSION in your release environment
    • EXTRA_ARGS: leave it blank

  2. Click the Build button

  3. After a few seconds, a new build appears on top of the Build History sidebar. Click on the progress bar of this new build.

  4. Set the $CANDIDATE_JENKINS_IUKS_BUILD_ID environment variable to the ID of this job (an integer).

  5. Wait until this build_IUKs job completes successfully. It should take about 10-15 minutes for each member of the $IUK_SOURCE_VERSIONS list.

Verify that Jenkins reproduced your IUKs

"${RELEASE_CHECKOUT:?}"/bin/copy-iuks-to-rsync-server-and-verify \
   --hashes-file "${IUKS_HASHES:?}" \
   --work-dir /srv/tmp \
   --jenkins-build-id "${CANDIDATE_JENKINS_IUKS_BUILD_ID:?}"

If this verification succeeds, move on to the next section.

Else, if this verification fails, then:

  1. Visit the web page whose URL is printed by the following command:

     echo "https://jenkins.tails.boum.org/job/build_IUKs/${CANDIDATE_JENKINS_IUKS_BUILD_ID:?}/parameters/"

    It tells you which parameters you've passed to the Jenkins job.

  2. Double-check that you've passed the correct parameters.

    If you notice you made a mistake, build the IUKs on Jenkins again, and do the verification again.

    Else, if the parameters where correct, then follow the next steps.

  3. File a ticket about this problem.

    Specify:

    • Which set of parameters you've passed to the build_IUKs job, so that the person who'll investigate the problem can reproduce it.
    • The ID of the build that failed to reproduce your locally-built IUKs.

  4. Later on, after you're done with OpenPGP signing, you will upload the IUKs you've built locally.

Prepare upgrade-description files

  1. Prepare upgrade-description files (see the upgrade-description files specification for details).

    In most cases the example command below is exactly the one you should run. But in order to tell whether you're in one of the exceptional cases when you have to adjust that command, it's important that you understand what follows.

    At this step, we use the tails-iuk-generate-upgrade-description-files tool in order to:

    1. Create a new upgrade-description for the version being released and for the next one, that expresses that no upgrade is available for these ones yet.

      This is what the --version and --next_version arguments in the example command below do. You do not need to modify them.

    2. For every recent previous release that's not listed in $IUK_SOURCE_VERSIONS (i.e. those based on an older Debian release): tell the users of that old version they need to manually upgrade to the version being released.

      To do so, pass this previous version to --previous_version. You need to do this manually as the example command below won't do it automatically.

    Run this command, after adjusting it if needed as explained above:

     ${RELEASE_CHECKOUT:?}/config/chroot_local-includes/usr/src/iuk/bin/tails-iuk-generate-upgrade-description-files \
     --version "${VERSION:?}" \
     $( \
         if [ "${DIST:?}" = stable ]; then
             echo \
                 --next_version "${NEXT_PLANNED_MAJOR_VERSION:?}" \
                 --next_version "${NEXT_PLANNED_MAJOR_VERSION:?}~rc1" \
                 --next_version "${NEXT_PLANNED_BUGFIX_VERSION:?}" \
                 --next_version "${NEXT_POTENTIAL_EMERGENCY_VERSION:?}"
         else
             echo \
                 --next_version $(echo ${VERSION:?} | sed -e 's,~.*$,,') \
                 --next_version "${SECOND_NEXT_PLANNED_MAJOR_VERSION:?}"
         fi
     ) \
     $( \
         for version in $(echo ${IUK_SOURCE_VERSIONS:?}); do
            echo "--previous_version ${version:?}"
         done \
     ) \
     --iso "${ISO_PATH:?}" \
     --iuks "${IUKS_DIR:?}" \
     --release_checkout "${RELEASE_CHECKOUT:?}" \
     --major_release "${MAJOR_RELEASE:?}" \
     --channel "${DIST:?}"

  2. Create an armoured detached signature for each created or modified upgrade-description file.

     cd "${RELEASE_CHECKOUT:?}" && \
 find "${RELEASE_CHECKOUT:?}/wiki/src/upgrade/" \
    -type f -name upgrades.yml | \
    while read udf; do
        if [ -n "$(git status --porcelain "${udf:?}")" ]; then
            for x in 1 2 3; do
                gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor \
                    --detach-sign "${udf:?}" \
                && break
            done
            mv --force "${udf:?}.asc" "${udf:?}.pgp"
            ${RELEASE_CHECKOUT:?}/config/chroot_local-includes/usr/src/iuk/bin/tails-iuk-check-upgrade-description-file "${udf:?}" \
               || break
        fi
    done

  3. Add and commit the upgrade-description files and their detached signatures to the Git branch used to prepare the release ($WEBSITE_RELEASE_BRANCH):

     ( \
   cd "${RELEASE_CHECKOUT:?}" && git add wiki/src/upgrade && \
   git commit -m "Update upgrade-description files." && \
   git push origin ${WEBSITE_RELEASE_BRANCH:?} \
 )

  4. Copy the generated UDFs for the previous releases to the test channel in $MASTER_CHECKOUT, modify their content accordingly, sign them, commit and push:

     ( \
   cd ${MASTER_CHECKOUT:?} && \
   git fetch && \
   git merge origin/master && \
   for old_version in $(echo ${IUK_SOURCE_VERSIONS:?} ${VERSION:?}); do
     release_udf="wiki/src/upgrade/v2/Tails/${old_version:?}/amd64/${DIST:?}/upgrades.yml" && \
     test_udf="wiki/src/upgrade/v2/Tails/${old_version:?}/amd64/test/upgrades.yml" && \
     mkdir -p "$(dirname "$test_udf")" && \
     git show origin/${WEBSITE_RELEASE_BRANCH:?}:${release_udf:?} \
       | sed -e "s/channel: ${DIST:?}/channel: test/" > ${test_udf:?} && \
     echo "Signing ${test_udf:?}" && \
     gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor --detach-sign ${test_udf:?} && \
     mv ${test_udf:?}.asc ${test_udf:?}.pgp && \
     git add ${test_udf:?}* ; \
   done && \
   git commit -m "Add incremental upgrades on the test channel for Tails ${VERSION:?}" && \
   git push origin master:master \
 )

    XXX: ideally, we should also copy the UDFs that only advertise a full upgrade path to the version we're releasing, so that we can manually test that these upgrade paths work as expected. Our manual test suite already has everything in place for this.

Prepare the image description file for Tails Verification

If preparing a RC, skip this part.

Update the image description file (IDF) used by the browser extension:

./bin/idf-content \
   --version "${VERSION:?}" \
   --iso "${ISO_PATH:?}" \
   --img "${IMG_PATH:?}" \
   > "${RELEASE_CHECKOUT:?}"/wiki/src/install/v2/Tails/amd64/stable/latest.json && \
( cd "${RELEASE_CHECKOUT:?}" && \
  git add wiki/src/install/v2/Tails/amd64/stable/latest.json && \
  git commit -m "Update IDF file for Tails Verification." && \
  git push origin "${WEBSITE_RELEASE_BRANCH:?}" )

Done with OpenPGP signing

Unplug your OpenPGP smartcard and store it away, so you don't plug it back semi-mechanically later on.

Beware! If your have to plug your OpenPGP smart card or reassemble the key again after this point it invalidates everything done for the reproduction of this release so it has to be started from the beginning:

  • the original text is restored on the pad, and
  • some tester follows it from scratch, and
  • the Trusted Reproducer follows awaits the new input from said tester and then starts from scratch.

So please try to avoid this!

Upload images

Sanity check

Verify once more that the Tor Browser we ship is still the most recent (see above).

Publish the ISO, IMG, and IUKs over HTTP

If the IUKs reproducibility check you did earlier has failed, then upload the IUKs you've built to our rsync server (we trust your machine more than our Jenkins):

for source_version in $(echo ${IUK_SOURCE_VERSIONS:?}); do
   rsync --partial --inplace --progress -v \
      "${IUKS_DIR:?}/Tails_amd64_${source_version:?}_to_${VERSION:?}.iuk" \
      rsync.lizard:/srv/tmp/
done

While waiting for the IUKs to be uploaded, you can proceed with the next steps.

Upload the ISO and USB image signatures to our rsync server:

scp "${ISO_PATH:?}.sig" "${IMG_PATH:?}.sig" rsync.lizard:

Copy the ISO and USB images to our rsync server, verify their signature, move them in place with proper ownership and permissions and update the time in project/trace file on our rsync server and on the live website (even for a release candidate):

cat "${RELEASE_CHECKOUT:?}/wiki/src/tails-signing.key" \
   | ssh rsync.lizard gpg --import
ssh rsync.lizard << EOF
   wget --quiet \
      "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_IMAGES_BUILD_ID:?}/archive/build-artifacts/tails-amd64-${VERSION:?}.iso" \
      "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_IMAGES_BUILD_ID:?}/archive/build-artifacts/tails-amd64-${VERSION:?}.img" && \
   gpg --verify tails-amd64-${VERSION:?}.iso{.sig,} && \
   gpg --verify tails-amd64-${VERSION:?}.img{.sig,}
EOF

ssh rsync.lizard << EOF
  sudo install -o root -g rsync_tails -m 0755 -d \
     /srv/rsync/tails/tails/${DIST:?}/tails-amd64-${VERSION:?} && \
  sudo chown root:rsync_tails tails-amd64-${VERSION:?}.{iso,img}* && \
  sudo chmod u=rwX,go=rX tails-amd64-${VERSION:?}.{iso,img}* && \
  sudo mv tails-amd64-${VERSION:?}.{iso,img}* \
          /srv/rsync/tails/tails/${DIST:?}/tails-amd64-${VERSION:?}
EOF

TRACE_TIME=$(date +%s) &&
ssh rsync.lizard "echo ${TRACE_TIME:?} | sudo tee /srv/rsync/tails/tails/project/trace" && \
[ -n "${MASTER_CHECKOUT:?}" ] && \
echo ${TRACE_TIME:?} > "${MASTER_CHECKOUT:?}/wiki/src/inc/trace" &&
(
   cd "${MASTER_CHECKOUT:?}" && \
   git commit wiki/src/inc/trace \
      -m "Updating trace file after uploading the ISO and USB images for ${VERSION:?}." && \
   git push origin master
)

At this stage, either IUKs were reproduced by Jenkins and left on the rsync server in a temporary location, or one should wait until the synchronization of local files (started in the first part of this section) to the same temporary location has completed.

Move the IUKs in place with proper ownership and permissions and update the time in project/trace file on our rsync server and on the live website (even for a release candidate):

ssh rsync.lizard << EOF
  sudo chown root:rsync_tails /srv/tmp/Tails_amd64_*_to_${VERSION:?}.iuk && \
  sudo chmod u=rwX,go=rX /srv/tmp/Tails_amd64_*_to_${VERSION:?}.iuk && \
  sudo mv /srv/tmp/Tails_amd64_*_to_${VERSION:?}.iuk \
          /srv/rsync/tails/tails/${DIST:?}/iuk/v2/
EOF

TRACE_TIME=$(date +%s) &&
ssh rsync.lizard "echo ${TRACE_TIME:?} | sudo tee /srv/rsync/tails/tails/project/trace" && \
[ -n "${MASTER_CHECKOUT:?}" ] && \
echo ${TRACE_TIME:?} > "${MASTER_CHECKOUT:?}/wiki/src/inc/trace" &&
(
   cd "${MASTER_CHECKOUT:?}" && \
   git commit wiki/src/inc/trace \
      -m "Updating trace file after uploading the IUKs for ${VERSION:?}." && \
   git push origin master
)

Announce, seed and test the Torrents

Check if there's enough space on our Bittorrent seed to import the new ISO and USB images:

ssh bittorrent.lizard df -h /var/lib/transmission-daemon/downloads

If not, list already running Torrents:

ssh bittorrent.lizard transmission-remote --list

… set $ID to the oldest one and delete it (do this both for the ISO and USB images):

ssh bittorrent.lizard transmission-remote -t "${ID:?}" --remove-and-delete

… and finally check disk space again:

ssh bittorrent.lizard df -h /var/lib/transmission-daemon/downloads

Now you can announce and seed the Torrents for the release you're preparing:

cat "${RELEASE_CHECKOUT:?}/wiki/src/tails-signing.key" \
   | ssh bittorrent.lizard gpg --import
for type in iso img ; do
   image_filename="tails-amd64-${VERSION:?}.${type:?}"
   torrent_dirname="tails-amd64-${VERSION:?}-${type:?}"
   scp \
      "${ISOS:?}/${image_filename:?}.torrent" \
      "${ISOS:?}/tails-amd64-${VERSION:?}/${image_filename:?}.sig" \
      bittorrent.lizard: && \
   ssh bittorrent.lizard << EOF
      mkdir --mode 0775 "${torrent_dirname:?}" && \
      cd "${torrent_dirname:?}" && \
      mv "../${image_filename:?}.sig" . && \
      wget --quiet \
         "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_IMAGES_BUILD_ID:?}/archive/build-artifacts/${image_filename:?}" && \
      gpg --verify ${image_filename:?}{.sig,} && \
      cd && \
      chgrp -R debian-transmission "${torrent_dirname:?}" && \
      chmod -R go+rX,g+w "${torrent_dirname:?}" && \
      mv \
         "${torrent_dirname:?}" \
         /var/lib/transmission-daemon/downloads/ && \
      transmission-remote --add ${image_filename:?}.torrent \
                          --find /var/lib/transmission-daemon/downloads/
EOF
done

Test that you can start downloading the ISO and USB images with a BitTorrent client.

Testing

  1. Using check-mirrors, choose a fast mirror that already has the tentative ISO and USB images. E.g. https://mirrors.kernel.org/tails/ or https://mirrors.wikimedia.org/tails/ are reliable and have plenty of bandwidth.

     ./check-mirrors.rb --allow-multiple --fast --channel ${DIST:?} \
     --ip $(dig +short mirrors.kernel.org | tail -n1) \
     tails-amd64-${VERSION:?}

  2. Email tails-testers@boum.org to ask them to test the tentative ISO and USB images, pointing them to the up-to-date mirror you've found previously. This is a public list, don't point to the pad.

  3. Email tails@boum.org and tails-manual-testers@boum.org that tests may start:
    • point them to the up-to-date mirror you've found previously
    • make it clear what's the deadline
    • make it clear where and how you expect to get feedback
    • attach the Torrents
    • attach the .packages file
  4. Make sure someone is committed to run the automated test suite.
  5. Make sure that enough people are here to run the tests, that they report their results in due time, and that they make it clear when they're leaving for good.
  6. Fill the holes and make sure that the automated and manual test suites are done in due time. Clock and report this work separately from your RM'ing work.
  7. Triage test results, reproduce bugs as needed, decide what the next step is and make sure it happens: add to known issues? file ticket? release blocker? improve the test description (steps, expected outcome)?

Prepare announcements and blog posts

What follows in this section happens on the $WEBSITE_RELEASE_BRANCH branch in ${RELEASE_CHECKOUT:?}:

cd "${RELEASE_CHECKOUT:?}" && \
   git checkout "${WEBSITE_RELEASE_BRANCH:?}"

If preparing a final release

Skip this part if preparing a RC.

Rename, copy, garbage collect and update various files:

cp "${ISO_PATH:?}.sig" \
   "${IMG_PATH:?}.sig" \
   "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest" \
   "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.packages" \
   "${ISOS:?}/tails-amd64-${VERSION:?}".{iso,img}.torrent \
   "${RELEASE_CHECKOUT:?}/wiki/src/torrents/files/" && \
git rm \
   "${RELEASE_CHECKOUT:?}/wiki/src/torrents/files/tails-amd64-${PREVIOUS_VERSION:?}."{build-manifest,iso.sig,img.sig,packages,iso.torrent,img.torrent} && \
LC_NUMERIC=C ls -l -h -L ${ISO_PATH:?} | \
   cut -f 5 -d ' ' | sed -r 's/(.+)([MG])/\1 \2B/' \
   > "${RELEASE_CHECKOUT:?}/wiki/src/inc/stable_amd64_iso_size.html" && \
LC_NUMERIC=C ls -l -h -L ${IMG_PATH:?} | \
   cut -f 5 -d ' ' | sed -r 's/(.+)([MG])/\1 \2B/' \
   > "${RELEASE_CHECKOUT:?}/wiki/src/inc/stable_amd64_img_size.html" && \
gpg --check-trustdb && \
LANG=C TZ=UTC gpg --no-options --keyid-format long --trusted-key "${TAILS_SIGNATURE_KEY_LONG_ID:?}" --verify "${ISO_PATH:?}.sig" "${ISO_PATH:?}" 2>&1 | \
   perl -pE 's/\[ultimate\]$/[full]/' | \
   sed 's/ /\&nbsp;/g;s/</\&lt;/;s/>/\&gt;/;s/$/<br\/>/g' > \
   "${RELEASE_CHECKOUT:?}/wiki/src/inc/stable_amd64_iso_gpg_signature_output.html" && \
LANG=C TZ=UTC gpg --no-options --keyid-format long --trusted-key "${TAILS_SIGNATURE_KEY_LONG_ID:?}" --verify "${IMG_PATH:?}.sig" "${IMG_PATH:?}" 2>&1 | \
   perl -pE 's/\[ultimate\]$/[full]/' | \
   sed 's/ /\&nbsp;/g;s/</\&lt;/;s/>/\&gt;/;s/$/<br\/>/g' > \
   "${RELEASE_CHECKOUT:?}/wiki/src/inc/stable_amd64_img_gpg_signature_output.html"

Then, build the website and commit this last set of changes:

./build-website && \
git add wiki/src/{inc,torrents} && \
git commit -m "Update various website source files for ${VERSION:?}"

Ensure our technical writer has written the announcement for the release in wiki/src/news/version_${TAG:?}.mdwn.

Write an announcement listing the security bugs affecting the previous version in wiki/src/security/Numerous_security_holes_in_${PREVIOUS_VERSION:?}.mdwn in order to let the users of the old versions know that they have to upgrade. Date it a few days before the images to be released were built. Including:

Remove obsolete bits from wiki/src/home/testing.html. For example, remove any text that's specific to a release candidate we may have published for the version you are preparing.

If preparing a release candidate

Skip this part if preparing a final release.

Copy the signatures and the Torrents into the website repository:

cp "${ISO_PATH:?}.sig" \
   "${IMG_PATH:?}.sig" \
   "${ISOS:?}/tails-amd64-${VERSION:?}".{iso,img}.torrent \
   "${RELEASE_CHECKOUT:?}/wiki/src/torrents/files/"

Write the announcement for the release in ${RELEASE_CHECKOUT:?}/wiki/src/news/test_${TAG:?}.mdwn, including:

  • Update the meta title directive.
  • Update the meta date directive.
  • Document important config changes that persistence users have to do themselves (e.g. the Pidgin proxy settings change in 9925321 breaks all existing persistent profiles).
  • Document known issues.

  • This snippet can help to convert the copied changelog's ticket references to links:

    sed -i 's@#\([0-9]\{4,5\}\)@<a href="https://redmine.tails.boum.org/code/issues/\1">#\1</a>@g' \
    wiki/src/news/test_${TAG:?}.mdwn

In wiki/src/home/testing.html, add a bullet point with a pointer to the call for testing for this release candidate (https://tails.boum.org/news/test_$TAG/, that you just wrote).

In any case

Generate PO files for the pages added or updated earlier in this section, and publish them on the $WEBSITE_RELEASE_BRANCH branch:

./build-website && \
git add wiki/src && \
git commit -m "Releasing version ${VERSION:?}" && \
git push origin "${WEBSITE_RELEASE_BRANCH:?}"

Then, if you are preparing a final release, email tails-l10n@boum.org, pointing them to:

  • the $WEBSITE_RELEASE_BRANCH branch
  • the PO files for the release notes

… so that these pages get translated shortly, perhaps even soon enough to merge the translations before you publish the release.

Draft the Tor blog post

Skip this if you are preparing anything but a final, major release.

  • Generate a Tor Blog-friendly post; please go through it manually, and look at the previews, to make sure it looks sane!

    ikiwiki --setup ikiwiki.setup \
        --render wiki/src/news/version_${VERSION:?}.mdwn | \
    tidy --wrap 99999 | \
    sed '0,/^<div id="content" role="main">$/d' | \
    sed '/^<div id="footer" class="pagefooter" role="contentinfo">$/,$d' | \
    sed '/^<div class="toc">$/,+7d' | \
    sed '/^<p><img [^<>]*\/><\/p>$/d' | \
    sed '/^<div[^<>]*\>/d' | \
    sed '/^<\/div>$/d' | \
    sed 's@<a name[^<>]*></a>@@g' | \
    sed 's@href="\.\./@href="https://tails.boum.org/@g' | \
    sed 's@src="\./@src="https://tails.boum.org/news/@g' | \
    sed 's@\(\.en\)\?.html@/@g' \
    > /tmp/tor-blog-post.html
    cat >> /tmp/tor-blog-post.html <<EOF
<h1>Support and feedback</h1>
<p>For support and feedback, visit the <a href="https://tails.boum.org/support/">Support section</a> on the Tails website.</p>
EOF

  • login to the Tor blog

  • click Content → Add content → Blog Post
  • add these tags, each in a separate text entry field (use Add another item):
    • tails
    • anonymous operating system
    • tails releases
  • set Title to "New Release: Tails $VERSION"
  • choose Filtered HTML as the Text format in the blog post editor
  • copy the text you have prepared into the Post Body textarea of the blog post editor
  • open Comment Settings and verify that comments are Closed
  • open Promotion Options and check Promoted to front page
  • click Preview and ensure everything is OK
  • click Back to content editing
  • ensure the Publish checkbox is not checked
  • click Save to save the draft
  • Let steph@torproject.org know when you intend to publish this blog post.

Go wild!

Wait for the HTTP mirrors to catch up

Test downloading the ISO, USB image, and IUK over HTTP.

Make sure every active mirror in the pool has the new version:

./check-mirrors.rb --channel ${DIST:?} --allow-multiple --fast \
    tails-amd64-${VERSION:?}

Ask tails-mirrors@boum.org [encrypted] to drop those that are lagging behind and notify their administrators.

Sanity checks

  • Check the outcome of the "Testing" section above.

  • Wait for the Mozilla security advisory (MFSA) to be published.

    Below you find a shell function that will exit once a new MFSA is published, so you can wait_for_new_mfsa ; notify-send "New MFSA!" or similar to automate the wait:

    current_mfsa() {
    local current="$(
        torsocks --isolate curl --silent https://www.mozilla.org/en-US/security/advisories/ | \
        sed --regexp-extended -n 's@.*<a href="/en-US/security/advisories/(mfsa[0-9]+-[0-9]+)/".*>@\1@p' | \
        sort -n | \
        tail -n 1
    )"
    echo "$(date --rfc-3339=s): got ${current}" >&2
    echo "${current}"
}
wait_for_new_mfsa() {
    local initial="$(current_mfsa)"
    while true; do
        local new="$(current_mfsa)"
        [ -z "${new}" ] && continue
        [ "${new}" != "${initial}" ] && break
        sleep 60
    done
}

  • Add the relevant MFSA to wiki/src/security/Numerous_security_holes_in_${PREVIOUS_VERSION:?}.mdwn and commit.

  • Verify once more that the Tor Browser we ship is still the most recent (see above).

Push

Git

Push the last commits to our Git repository and put master in the following state:

( cd "${RELEASE_CHECKOUT:?}" && \
  git push origin \
      "${WEBSITE_RELEASE_BRANCH:?}:${WEBSITE_RELEASE_BRANCH:?}" \
      devel:devel \
) && \
( cd "${MASTER_CHECKOUT:?}" && \
  git fetch && \
  git merge origin/master && \
  git merge "origin/${WEBSITE_RELEASE_BRANCH:?}" && \
  echo "stable" > config/base_branch && \
  git commit config/base_branch \
      -m "Restore master's base branch." \
)

Finally, push the master branch to make the changes go live on our website:

( cd "${MASTER_CHECKOUT:?}" && \
  git push origin master:master \
)

The release is now public! Woo!

Bug tracker

Skip this part if preparing a release candidate.

Edit the Completed for the next release Redmine view: set the Target version filter to the next scheduled release.

Postpone to next release any remaining open issue for the version you've just released. Use the right-click contextual menu to do so in one single batch.

Then, mark the just-released Redmine milestone as done: go to the target version page, click Edit, and set Status to Closed.

Tickets linked from the website

Go through the tickets linked from the documentation and support sections of the website and point our lead technical writer (sajolida) to the tickets that might be resolved in this release.

cd "${MASTER_CHECKOUT:?}" && \
find wiki/src/{doc,support} -name "*.mdwn" -o -name "*.html" | xargs cat | \
    ruby -e 'puts STDIN.read.scan(/\[\[!tails_ticket\s+(\d+)[^\]]*\]\]/)' | \
    while read ticket; do
        url="https://redmine.tails.boum.org/code/issues/${ticket:?}"
        url_content=$(curl --fail --silent ${url:?})
        if [ "${?}" -ne 0 ] || [ -z "${url_content:-}" ]; then
            echo "Failed to fetch ${url:?} so manually investigate #${ticket:?}" >&2
            continue
        fi
        ticket_status="$(echo "${url_content:?}" | \
            sed -n 's,^.*<div class="status attribute"><div class="label">Status:</div><div class="value">\([^<>]\+\)</div></div>.*$,\1,p')"
        if [ -z "${ticket_status:-}" ]; then
            echo "Failed to find the status of #${ticket:?}" >&2
            continue
        fi
        if [ "${ticket_status:?}" != "New" ] && \
           [ "${ticket_status:?}" != "Confirmed" ] && \
           [ "${ticket_status:?}" != "In Progress" ]; then
            echo "It seems ticket #${ticket:?} has been fixed (Status: ${ticket_status:?}) so please find all instances in the wiki and fix them. Ticket URL: ${url:?}"
        fi
    done

Remember that ticket expressions, e.g. <a href="https://redmine.tails.boum.org/code/issues/1234">#1234</a>, can span several lines, so finding the ones reported by the above code might be harder than git grep "tails_ticket 1234".

Twitter

Check in the comments of the ticket for the release notes if the technical writers have prepared a tweet. Otherwise tweet a simple link to the release notes:

Tails x.y is out: https://tails.boum.org/news/version_x.y/

Publish the Tor blog post

  1. Open the draft Tor blog post you have prepared earlier
  2. Adjust Authoring Information → Authored on to make the publication date correct
  3. Click Save and publish

Amnesia news

  1. Subscribe to amnesia-news@. It's the only reliable way to check if the next step has worked (the online web archive is not refreshed immediately).

  2. Log in and accept the release announcement, that's been automatically sent to amnesia-news@ on an hourly basis, and is stuck in the moderation queue.

Prepare for the next development cycle

If you just released a final release

  1. If you just released a new stable release, remove the previous stable release from:
    • our rsync server: ssh rsync.lizard sudo rm -rf /srv/rsync/tails/tails/stable/tails-amd64-${PREVIOUS_VERSION:?}/
    • our Bittorrent seed: get the previous release's Transmission IDs (ISO and USB image) with ssh bittorrent.lizard transmission-remote --list and then delete them with ssh bittorrent.lizard transmission-remote -t "${PREVIOUS_VERSION_TRANSMISSION_ID:?}" --remove-and-delete then re-run ssh bittorrent.lizard transmission-remote --list and make sure everything looks good
  2. Remove any remaining RC for the just-published release from rsync.lizard:/srv/rsync/tails/tails/alpha/

  3. If you've published a final release, remove IUKs that upgrade to an older version as they were superseded by this release:

    • first check that it's not going to remove anything we want to keep:

      ssh rsync.lizard /bin/sh -c \
    \"find /srv/rsync/tails/tails/alpha/iuk/v2  \
           /srv/rsync/tails/tails/stable/iuk/v2 \
           -type f -name '*.iuk' \
           -not -name "*_to_${VERSION:?}.iuk" \
           -not -name '*~test_*~test.iuk' \
           -not -name '*~testoverlayfs_*~testoverlayfs.iuk' \
           -ls \
    \"

    • then actually delete the files:

      ssh rsync.lizard /bin/sh -c \
    \"find /srv/rsync/tails/tails/alpha/iuk/v2  \
           /srv/rsync/tails/tails/stable/iuk/v2 \
           -type f -name '*.iuk' \
           -not -name "*_to_${VERSION:?}.iuk" \
           -not -name '*~test_*~test.iuk' \
           -not -name '*~testoverlayfs_*~testoverlayfs.iuk' \
           -delete \
    \"

  4. Check how much space our mirrors need:

     ssh rsync.lizard du -sh /srv/rsync/tails

    Compare it to the minimum disk space we ask of our mirror operators (30 GiB) and determine if any further action is needed to either reduce our usage by deleting stuff, or asking them to give us more space.

  5. Delete Git branches that were merged:

     cd "${MASTER_CHECKOUT:?}" && \
 git checkout master && \
 git fetch && \
 git merge origin/master && \
 git submodule update --init && \
 bare_repo=$(mktemp -d) && \
 git clone --bare --reference "${MASTER_CHECKOUT:?}" \
    gitolite@d53ykjpeekuikgoq.onion:tails \
    "${bare_repo:?}" && \
 PYTHONPATH=lib/python3 ./bin/delete-merged-git-branches \
    --repo "${bare_repo:?}" && \
 rm -rf "${bare_repo:?}"

  6. On the stable and devel branches, remove all old versions that were never released from wiki/src/upgrade/v2/Tails and debian/changelog. Explanation: the post-release APT repository steps from the previous stable release will usually have had us prepare for an emergency release that was never made.

  7. Thaw the packages that were granted freeze exceptions.
  8. Pull master back and merge it into stable, and in turn into devel
  9. Follow the post-release custom APT repository documentation. This includes some git operations, like creating an appropriate "dummy changelog entry" in the debian/changelog file.

  10. Verify that the snapshots used in the release branch are ok, e.g. they use the correct snapshots, and they were bumped appropriately (they should expire after the next planned major release date). Look carefully at the output of this command:

     cd "${RELEASE_CHECKOUT:?}" && \
 git checkout "${RELEASE_BRANCH:?}" && \
 for dir in config/APT_snapshots.d vagrant/definitions/tails-builder/config/APT_snapshots.d; do
 (
     echo "${dir:?}:"
     cd "${dir:?}" && \
     for ARCHIVE in * ; do
         SERIAL="$(cat ${ARCHIVE:?}/serial)"
         if [ "${SERIAL:?}" = 'latest' ]; then
             EXPIRY='never'
             if [ "${ARCHIVE:?}" != 'debian-security' ]; then
                 echo "Warning: origin '${ARCHIVE:?}' is using the 'latest' snapshot, which is unexpected" >&2
             fi
         else
             case "${ARCHIVE:?}" in
                 'debian-security')
                     DIST='buster/updates'
                     ;;
                 'torproject')
                     DIST='buster'
                     ;;
                 *)
                     DIST='stable'
                     ;;
             esac
             EXPIRY="$(curl --silent "http://time-based.snapshots.deb.tails.boum.org/${ARCHIVE:?}/dists/${DIST:?}/snapshots/${SERIAL:?}/Release" | sed -n 's/^Valid-Until:\s\+\(.*\)$/\1/p')"
         fi
         echo "* Archive '${ARCHIVE:?}' uses snapshot '${SERIAL:?}' which expires on: ${EXPIRY:?}"
     done
     echo ---
 )
 done

  11. Push the resulting branches.

  12. Make sure Jenkins manages to build all updated major branches: https://jenkins.tails.boum.org/.
  13. Make sure you pushed all changes in every of our Git repo (including our Debian packages ones).
  14. Delete the Release Manager View for ${VERSION:?}_ Redmine custom query.
  15. Ensure the next two releases have their own Release Manager View.
  16. On the roadmap, update the Due date for the Holes in the Roof so that this section appears after the next release.
  17. In calendar, remove the entries about the version that you've just released.

  18. Check the Mozilla release calendars:

    If the upcoming release date in the Mozilla calendar does not match what we have in calendar, then alert tails-dev@boum.org, tails-rm@boum.org, and explicitly Cc the designated RM for that upcoming release.

  19. Announce the date of the next release on tails-dev@boum.org and tails-l10n@boum.org.

  20. If you are the release manager for the next release too, look at the tasks that must be done at the beginning of your shift in the release manager role page. Otherwise, kindly remind the next release manager about this :)

If you just released an RC

  1. On the devel branch, thaw all packages (if any) that were granted freeze exceptions.
  2. On the devel branch, thaw the time-based APT repository snapshots. This should generally be a no-op but if there was some hiccup earlier it could be needed.
  3. Follow the post-release custom APT repository documentation. This includes some git operations, like creating an appropriate "dummy changelog entry" in the debian/changelog file.

  4. Merge testing into devel, and push them:

     cd "${RELEASE_CHECKOUT:?}" && \
 git checkout devel && \
 git merge testing && \
 git push origin testing devel

  5. Make sure you pushed all changes in every of our Git repo (including our Debian packages ones).

  6. Follow the "Verify that the snapshots used in the release branch are ok" step for final releases, above.
  7. Make sure Jenkins manages to build all updated major branches: https://jenkins.tails.boum.org/.
  8. In calendar, remove the entries about the version that you've just released.

Related pages