1. Context
    1. Release types
    2. Plan your journey
  2. Prepare your checklist
  3. TRAP: whenever you go AFK for long times
  4. Requirements
    1. Packages
    2. System configuration
    3. Configuration files
  5. Environment
  6. Pre-freeze
    1. Check the Release Manager role
    2. Coordinate with Debian security updates
  7. Sanity check
  8. Freeze
    1. Only for Major release
    2. Only for Bugfix release
  9. Set base branch
  10. Manual testing coordination
  11. Update included files
    1. Upgrade Tor Browser
    2. Update PO files
  12. If preparing a final release
    1. Only for Major release
    2. Only for Bugfix release
  13. Update other base branches
  14. Update more included files
    1. Changelog
    2. Included website
  15. Enable OpenPGP signing
  16. Build the almost-final images
  17. Tag the release in Git
  18. Prepare the versioned APT suites
    1. Versioned APT suite in our custom APT repository
    2. Tagged snapshots of upstream APT repositories
  19. Build images
    1. Sanity check
    2. SquashFS file order
    3. Build the final images
    4. Verify that Jenkins reproduced your images
    5. Initialize the website release branch
  20. Generate the OpenPGP signatures and Torrents
  21. Get the internal manual testing started
  22. Prepare incremental upgrades
    1. Sanity checks
    2. Build the Incremental Upgrade Kits locally
    3. ISO history
    4. Build the Incremental Upgrade Kits on Jenkins
    5. Verify that Jenkins reproduced your IUKs
    6. Prepare upgrade-description files
    7. Prepare the image description file for the download verification
  23. Done with OpenPGP signing
  24. Upload images
    1. Sanity check
    2. Delete obsolete IUKs
    3. Publish the ISO, IMG, and IUKs over HTTP
    4. Announce, seed and test the Torrents
  25. Run the automated test suite
  26. Public testing
  27. Lead the internal manual testing to conclusion
  28. Prepare announcements and blog posts
    1. If preparing a major or a bugfix release
    2. If preparing a release candidate
    3. In any case
    4. Draft the Tor blog post
  29. Go wild!
    1. Wait for the HTTP mirrors to catch up
    2. Sanity checks
    3. Push
    4. Bug tracker
    5. Twitter
    6. Publish the Tor blog post
    7. Amnesia news
  30. Prepare for the next development cycle
    1. If you just released a final release
    2. If you just released an RC
  31. Related pages

See the release schedule.

Context

Release types

There are exactly 3 types of releases:

  • A bugfix release
  • A major release
  • A release candidate. A release candidate can only be made for a major release

We will not cover beta release in this document.

Both a bugfix release and a major release are called "final". A release candidate is not a final release.

Sections that only apply to some of those types of release, we'll have it written clearly in the title.

If you want to know whether the release you are doing is a major or not, you need to check calendar, and use these criteria:

  • if nothing is specified about being a major, then it's most probably a bugfix release
  • if it is a major, there has to be a release candidate before the final

Plan your journey

The release process is long. With proper planning and some luck, it's almost enjoyable.

Here are some useful tips that will hopefully help you have a pleasant release process:

  • if time allows, try to unblock other people's work. For example, reaching the Get the internal manual testing started on Monday would be great, because it allows manual testers to plan their job better.

  • when you are clocking off, or just spending some time afk, consider keeping your CPU busy with some long-running jobs. Here are some good candidates:

    • Building the final image
    • Building IUKs
    • Running the automated test suite

  • cross your fingers 🤞

Prepare your checklist

  1. Merge the current master branch into the branch used to prepare the release (stable for a bugfix release, testing for a major release, devel for a RC).
  2. Copy this very file from the branch used to prepare the release to some place outside of Git.
  3. Close the version of this file that's in Git.
  4. From now on, follow the instructions in your own local copy, using it as a checklist: delete steps once you've successfully completed them. This ensures you won't mistakenly skip a step, without relying on memory, so "oops, I forgot X" cannot happen.

TRAP: whenever you go AFK for long times

Keep this section at the top of your pad, so you don't forget

When you clock off, or take any other kind of long break, please comunicate this on XMPP. Clarify:

  • when do you expect to come back

  • what's the status right now

  • try to give an ETA (or clarify that you cannot) for reaching this steps:

    • When will the final image be available?

    • When will manual testers be able to start their work?

Requirements

Packages

To release Tails you'll need some packages installed:

  • gitlab-cli jq tidy mktorrent podman python3-bs4 python3-debian python3-gitlab python3-jinja2 python3-voluptuous transmission-cli
    • Note: if the Debian version you are running does not include all required Go packages, such as podman, you should be able to install them from a more recent Debian release with APT pinning.
  • perl5lib dependencies
  • iuk dependencies
  • squashfs-tools-ng and libsquashfs1 from Bullseye
  • po4a 0.55:
    • Different versions extract Markdown headings in a different way, which makes tons of strings fuzzy.
    • Different versions emit different sets of message flags, some of which are not supported by i18nspector yet.
  • i18nspector 0.26 or newer
  • packages to build a local version of the website

System configuration

  • Enable unprivileged user namespaces

    If you're running a Debian kernel older than 5.10.1-1~exp1, set the kernel.unprivileged_userns_clone sysctl to 1:

      echo 'kernel.unprivileged_userns_clone=1' \
     | sudo tee /etc/sysctl.d/unprivileged-user-namespaces.conf && \
  sudo sysctl --system

Configuration files

To release Tails you need:

  • The correct version of python3-gitlab.

  • ~/.python-gitlab.cfg

    You need at least this content:

      [global]
  ssl_verify = true

  [TailsRM]
  url = https://gitlab.tails.boum.org
  per_page = 100
  private_token = XXX

  [Tails]
  url = https://gitlab.tails.boum.org
  per_page = 100
  private_token = XXX

    Then:

    • In the TailsRM section, set the value of the private_token option to the role-release-automation GitLab user's API token, which you'll find in rm.git's keyringer.

    • In the Tails section, set the value of the private_token option to a GitLab API token for your own user.

  • ~/.config/tails/release_management/local.yml

    If you have no such file yet, copy config/release_management/examples/local.yml there and adjust to your local environment.

Environment

To be able to copy'n'paste the code snippets found on this page, you need to set a bunch of environment variables.

Generate the base environment

  1. To generate the ~/.config/tails/release_management/current.yml template, run:

     ./bin/rm-config generate-boilerplate

  2. Edit ~/.config/tails/release_management/current.yml and replace FIXME:s:

      "${EDITOR:?}" ~/.config/tails/release_management/current.yml

  3. Ensure there's a iuks/v2 directory under the directory you specified for isos in ~/.config/tails/release_management/local.yml.

     python3 <<EOF
 import yaml,os
 from pathlib import Path
 fpath = Path(os.path.expanduser("~/.config/tails/release_management/local.yml"))
 isos = Path(yaml.safe_load(fpath.open())["isos"])
 iuks = isos / "iuks/v2"
 if not iuks.exists(): raise Exception("IUKS dir '%s' not found" % iuks)
 EOF

  4. Generate the resulting environment variables and export them into your environment:

     . $(./bin/rm-config generate-environment)

Notes

  • Unless the release process explicitly instructs you to change the value of one such variable, treat it as a constant: else, inconsistency will surely arise, which can cause trouble later on.
  • Regarding version numbers, what follows supports just fine the case when we do something else than alternating bugfix and major releases consistently. For example, if the next two releases are bugfix ones, do not set $NEXT_PLANNED_MAJOR_VERSION to one of these bugfix releases. Instead, set it to the version number of the next major release.
  • The $NEXT*VERSION constants are used only for two types of operations: preparing upgrade-description files and adding changelog entries. This two types of operations have to be consistent with each other: for example, if one adds a dummy entry for version X in a changelog, an UDF must exist for version X as well… hence the use of shared constants to encode the values that must be the same on both sides :)

Pre-freeze

Check the Release Manager role

Check the release manager role documentation: it has more tasks that should be done early enough.

Coordinate with Debian security updates

Look at the list of upcoming Debian Security Advisories (DSA) found in the Debian security tracker's Git repository and in the debian-security-announce mailing list.

Rationale: we don't want to release Tails and learn 2 days later about important security issues fixed by Debian security updates. In some cases, it may be worth delaying a Tails release a bit, while waiting for a DSA to happen.

Sanity check

Visit the Jenkins RM view and check that the jobs for the release branch have good enough results:

  • the last build and reproducibility testing builds have succeeded for:
    • if you're preparing a final release: the release branch
    • else, if you're preparing a release candidate: the devel branch
  • look at the last build of the corresponding test_Tails_ISO_* job; if anything looks unusual or fishy, check the few previous builds to see if it's a one-off robustness issue, a serious regression, or what not.

Freeze

This section ensures freezing is done. Freezing is relevant for Release Candidates and for bugfix releases. You can skip this if you're releasing a final major version.

Only for Major release

Follow this part ONLY if you are preparing the release candidate for a major release.

  1. Merge the master Git branch into devel:

     git checkout devel && git fetch origin && git merge origin/devel && git merge --no-ff origin/master

  2. Merge each APT overlay suite listed in the devel branch's config/APT_overlays.d/ into the devel APT suite:

     ./bin/merge-APT-overlays devel

  3. Merge the devel Git branch into the testing one:

     git checkout testing && git merge origin/testing && git merge devel

    ... and check that the resulting config/APT_overlays.d/ in the testing branch is empty.

  4. Hard reset the testing custom APT suite to the current state of the devel one:

     ./bin/reset-custom-APT-suite testing devel

  5. Freeze the time-based APT repository snapshots that shall be used during the code freeze, and those used by our builder VM:

     ./bin/freeze-all-APT-snapshots

  6. Make it so the time-based APT repository snapshots are kept around long enough, by bumping their Valid-Until half a year from now:

     git checkout "$RELEASE_BRANCH" && \
 ./bin/bump-APT-snapshots-expiration-date 183

Only for Bugfix release

Follow this part ONLY if you are preparing a bugfix release.

  1. Merge the master Git branch into stable:

     git checkout stable && git fetch && git merge origin/stable && git merge --no-ff origin/master

  2. Merge each APT overlay suite listed in the stable branch's config/APT_overlays.d/ into the stable APT suite:

     ./bin/merge-APT-overlays stable

Set base branch

Reset the release branch's config/base_branch:

    echo "${RELEASE_BRANCH:?}" > config/base_branch && \
       git commit config/base_branch \
           -m "Restore ${RELEASE_BRANCH:?}'s base branch."

Manual testing coordination

Bootstrap internal manual testing coordination:

  1. Create a pad.
  2. Set the PAD environment variable to the URL of this pad.
  3. Copy the manual test suite into this pad.

  4. Generate boilerplate for the call for manual testing:

     ./bin/generate-call-for-manual-testers \
    --dist "${DIST:?}" \
    --pad "${PAD:?}" \
    --version "${VERSION:?}"

  5. Send the call for manual testing:

    • encrypt and sign your email
    • use the headers and body generated above
    • adjust the body as needed

Update included files

Upgrade Tor Browser

If you did not import the new Tor Browser yet, do so now.

Update PO files

Pull updated translations for languages translated in Transifex, refresh the code PO files, and commit the result, including new PO files:

(cd "${RELEASE_CHECKOUT:?}"
 set -x
 err() { echo "Error $*"; exit 1 }
 ./import-translations || err "importing translations"
 ./refresh-translations || err "refreshing translations"
 ./submodules/jenkins-tools/slaves/lint_po || err "in lint"
 git add po && git commit -m 'Update PO files.'
)

If lint_po complains:

  • rollback the offending PO files and retry; worst case, delete it
  • send a note to tails-l10n@boum.org [public] so that they get in touch with whoever can fix them.

If preparing a final release

If we're about to prepare the images for a final (non-RC) release, then follow these instructions:

Only for Major release

Merge each APT overlay suite listed in the testing branch's config/APT_overlays.d/ into the testing custom APT suite:

    ./bin/merge-APT-overlays testing

Only for Bugfix release

For bugfix releases, we generally do not put any RC out, so freeze time is the same as preparing the actual release. Hence, the following steps have already been done above, and this section is a noop in the general case.

Merge each APT overlay suite listed in the stable branch's config/APT_overlays.d/ into the stable custom APT suite:

    ./bin/merge-APT-overlays stable

Update other base branches

  1. Merge the release branch into devel:

     ./bin/merge-main-branch "${RELEASE_BRANCH:?}" devel

  2. Thaw, on the devel branch, the time-based APT repository snapshots being used during the freeze:

     git checkout devel && \
 ./auto/scripts/apt-snapshots-serials thaw && \
 git commit \
     -m "Thaw APT snapshots." \
     config/APT_snapshots.d/*/serial || :

    It's fine if that results in a no-op (it depends on how exactly previous operations were performed).

  3. Try to merge the devel Git branch into feature/bookworm (if it exists), without merging the corresponding APT suite (for now feature/bookworm is handled as any other topic branch forked off devel: its base branch is set to devel):

     if git describe feature/bookworm >/dev/null 2>&1; then
    git checkout feature/bookworm && \
    git pull && \
    git merge devel
 fi

    If the merge conflicts don't look like something you feel confident resolving properly, abort this merge and let the Foundations Team know.

  4. Ensure that the $RELEASE_BRANCH, devel and feature/bookworm (if it exists) branches have the expected content in config/APT_overlays.d/: e.g. it must not list any overlay APT suite that has been merged already.

  5. Push the modified branches to Git:

     git push origin                          \
    "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
    $(if git describe feature/bookworm >/dev/null 2>&1; then echo feature/bookworm:feature/bookworm ; fi) \
    devel:devel

Update more included files

Changelog

Update the Changelog entry for the release you're preparing:

git checkout "${RELEASE_BRANCH:?}" && \
./bin/update-changelog --version "${MILESTONE:?}"

If you're preparing a release candidate, set the version for the new entry in debian/changelog to $VERSION: the above code has incorrectly set it to $MILESTONE.

Then, gather other useful information from the diff between the previous version's .packages file and the one from the to-be-released images:

  1. Generate the diff.

    if <span class="createlink"><a href="/ikiwiki.cgi?do=create&amp;from=contribute%2Frelease_process&amp;page=___34____36____123__DIST__58____63____125____34_____61___stable_" rel="nofollow">?</a> &#34;&#36;&#123;DIST&#58;&#63;&#125;&#34; &#61; stable </span>; then
    NEW_BRANCH="$RELEASE_BRANCH"
else
    NEW_BRANCH=devel
fi

diff --color=never -u \
    "wiki/src/torrents/files/tails-amd64-${PREVIOUS_STABLE_VERSION}.packages" \
    <(curl --silent "https://nightly.tails.boum.org/build_Tails_ISO_${NEW_BRANCH}/lastSuccessful/archive/latest.packages") \
 | wdiff --diff-input  --terminal

  2. In the diff, look for:

    • new upstream releases of applications mentioned in features
    • new upstream releases of other important components such as the Linux kernel

    And mention those changes in debian/changelog

  3. Sanity check the version, commit, and push:

    if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then git commit debian/changelog -m "Update changelog for ${VERSION:?}." else echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog' fi && \ git push origin "${RELEASE_BRANCH:?}"

  4. Let Technical Writers know that the changelog is ready

    • Open the GitLab issue about the release notes

    • Notify @technical-writers that the changelog is ready:

      echo "Dear @technical-writers, you can find the changelog on the ${RELEASE_BRANCH:?} branch"

    • Provide them a link to latest relevant Jenkins built image, so they can use that to make screenshots or write documentation.

Included website

./bin/prepare-included-website-for-release

Expect this to take 5-15 minutes. While the website is building, if this is not a good time for taking a break, you can follow the "Enable OpenPGP signing" instructions.

Enable OpenPGP signing

If you have an OpenPGP smart card

If you have an OpenPGP smart card with a Tails signing subkey, go fetch it.

Only plug it when gpg asks you to.

Otherwise: importing the signing key

This is only relevant when the master key has been reassembled, e.g. for signing a Tails emergency release where none of the usual release managers are available.

You should never import the Tails signing key into your own keyring, and a good practice is to import it to a tmpfs to limit the risks that the private key material is written to disk:

export GNUPGHOME=$(mktemp -d)
sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
sudo chmod 0700 "${GNUPGHOME:?}"
gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our signatures, like the defaults we set in Tails:

cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"

Build the almost-final images

  1. Build ISO and USB images from the release branch, with $TAILS_BUILD_OPTIONS set like this:

    • Set defaultcomp, so we can more accurately optimize our SquashFS file ordering.
    • Do not set keeprunning, nor ignorechanges, nor rescue. Expected time: 15-30 minutes depending on your computer.

  2. Keep the resulting build artifacts until the end of this release process.

  3. Record where the manifest of needed packages is stored:

    1. To update the ~/.config/tails/release_management/current.yml template with newly required variables, run:

      ./bin/rm-config generate-boilerplate --stage built-almost-final

    2. Edit ~/.config/tails/release_management/current.yml and replace FIXME:s:

      "${EDITOR:?}" ~/.config/tails/release_management/current.yml

    3. Generate the resulting environment variables and export them into your environment:

      . $(./bin/rm-config generate-environment --stage built-almost-final)

Tag the release in Git

git tag -u "${TAILS_SIGNATURE_KEY:?}" \
  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
git push origin "${TAG:?}" "${RELEASE_BRANCH:?}"

Known limitations:

  • Pushing the tag is needed so that the APT repository is updated, and the Tails APT configuration works at build and boot time. It might be premature, as testing might reveal critical issues, but this is a signed tag, so it can be overridden later.

  • From this push of a tag, the release branch will fail to build because the last changelog entry is unreleased but corresponds to an existing tag. Don't worry about it, this will be fixed shortly.

Prepare the versioned APT suites

Expected time: 15 minutes

Versioned APT suite in our custom APT repository

Within a few minutes after pushing the new release's Git tag, a cronjob creates a new APT suite in our custom APT repository. This new APT suite is called $TAG and is initially empty.

Wait for this APT suite to be created and initialize it with the packages currently found in the APT suite corresponding to the branch used to prepare the release:

  while ! ssh reprepro@incoming.deb.tails.boum.org reprepro list "${TAG:?}" >/dev/null 2>&1; do
     sleep 5
  done && \
  ssh reprepro@incoming.deb.tails.boum.org \
     tails-merge-suite "${RELEASE_BRANCH:?}" "${TAG:?}"

Tagged snapshots of upstream APT repositories

Create tagged snapshots of upstream APT repositories:

  ./bin/tag-apt-snapshots "${ALMOST_FINAL_BUILD_MANIFEST:?}" "${TAG:?}"

Note:

  • This command takes a while (about a dozen minutes). While it's running, if you don't feel the need to take a break, you can proceed with the "SquashFS file order" section below.
  • It's expected that the packages that were pulled from our custom APT repository are listed under "some packages were not found anywhere" (because we are currently not using time-based snapshots for our custom APT repository). However, no other package should be on that list. Now, we have a "safety" net, in case you don't notice such a problem: if other packages are missing, the next build (that will use the newly created partial, tagged APT repository) will fail.

Build images

Expected time: 30-75 minutes depending on your computer and the available Jenkins workers.

Sanity check

Verify that the Tor Browser release used in Tails still is the most recent. Also look if there's a new -buildX tag (e.g. tor-browser-60.3.0esr-8.0-1-build1) for the Firefox version the Tor Browser we want to ship is based on in these Git repositories:

A new tag may indicate that a new Tor Browser release or rebuild is imminent.

Better catch this before people spend time doing manual tests.

SquashFS file order

Expected time: 20 minutes

Feel free to skip this section when preparing an emergency release.

  1. Install the almost-final USB image to a USB stick.
  2. Boot this USB stick a first time to trigger re-partitioning.
  3. Shut down this Tails.
  4. If possible, set up a wired connection to avoid having to deal with wireless settings.
  5. Boot this USB stick on bare metal again.
  6. Add profile to the kernel command-line.
  7. Login with the default settings in the Welcome Screen (e.g. do not configure an Administration Password).
  8. Unless you've set up a wired connection, connect to Wi-Fi.
  9. Automatically connect to Tor with Tor Connection.
  10. Wait for Tor Connection to report success.
  11. Start Tor Browser.
  12. A few minutes later, once the boot-profile process has been killed, retrieve the new sort file from /var/log/boot-profile.
  13. Backup the old sort file: cp config/binary_rootfs/squashfs.sort{,.old}
  14. Copy the new sort file to config/binary_rootfs/squashfs.sort.

  15. To remove:

    • runtime-generated files that don't exist in the rootfs, in order to avoid confusing noise in the build output
    • the bits about kill-boot-profile at the end: they're only useful when profiling the boot
    • some hardware-dependent files

    Run this command:

        ./bin/clean-squashfs-sort-file config/binary_rootfs/squashfs.sort

  16. Inspect the Git diff (including diff stat), apply common sense:

     diff -NaurB \
     <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort.old | sort ) \
     <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort     | sort ) \
     | less

  17. git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort

  18. Clean up: rm -f config/binary_rootfs/squashfs.sort.old

Build the final images

Expected time: 30-75 minutes depending on your computer and the available Jenkins workers.

Then all included files should be up-to-date and the versioned APT suite should be ready, so it is time to:

  1. Mark the version as "released" in the changelog:

     dch --release --no-force-save-on-release --maintmaint && \
 git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog

  2. Export SOURCE_DATE_EPOCH:

     export SOURCE_DATE_EPOCH=$(date --utc --date="$(dpkg-parsechangelog --show-field=Date)" '+%s')

  3. Take note of the value of $SOURCE_DATE_EPOCH: you'll need it to "Build the Incremental Upgrade Kits on Jenkins" until #18318 is solved.

  4. tag the release again, with all included files in:

     git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
         -m "tagging version ${VERSION:?}" "${TAG:?}" && \
 git push --force origin "${TAG:?}" && \
 git push origin "${RELEASE_BRANCH:?}"

  5. build the final images! Do not set keeprunning nor rescue in $TAILS_BUILD_OPTIONS. Our build system will apply the correct compression settings automatically so don't bother setting it yourself.

  6. Make sure the Jenkins build starts as soon as possible. If needed, remove other builds from the Build Queue.

  7. Make sure the Jenkins build is right: you can recognize this is the case, when you see Jenkins starts building the website. Jenkins will take 20-65 minutes to complete the build, depending on the isobuilder that is used.

  8. This could be a good time for break!

  9. Compare the new build manifest with the one from the previous, almost-final build:

     diff -Naur \
    "${ALMOST_FINAL_BUILD_MANIFEST:?}" \
    "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"

    They should be identical, except that the debian-security serial might be higher.

Verify that Jenkins reproduced your images

To verify that Jenkins reproduced your images:

  1. Visit the URL printed by this command:

    echo "https://jenkins.tails.boum.org/job/build_Tails_ISO_${RELEASE_BRANCH}/"

  2. Find the relevant build (probably the last one) and compare the SHA512 hashes of the ISO and USB images built by Jenkins (in the tails-build-artifacts.shasum build artifact) with the ones of the images you built.

  3. Then:

    • If the ISO and USB images hashes match: yay, we're good to go! The .build-manifest may differ — that's OK. Then:

      1. To update the ~/.config/tails/release_management/current.yml template with newly required variables, run:

        ./bin/rm-config generate-boilerplate --stage reproduced-images

      2. Edit ~/.config/tails/release_management/current.yml and set the matching_jenkins_images_build_id value to the ID of this job (an integer):

        "${EDITOR:?}" ~/.config/tails/release_management/current.yml

      3. Generate the resulting environment variables and export them into your environment:

        . $(./bin/rm-config generate-environment --stage reproduced-images)

    • Else, if there is a hash mismatch for one of the images: ouch! see next section

If there is a hash mismatch

Now we are in a tricky situation: on the one hand it seems like a poor idea to block users from benefiting from this release's security updates, but on the other hand the failure might imply that something nefarious is going on. At this stage, no matter what, immediately fetch Jenkins' image, compare it with your, and try to rule out build system compromise:

  sudo diffoscope \
      --text diffoscope.txt \
      --html diffoscope.html \
      --max-report-size 262144000 \
      --max-diff-block-lines 10000 \
      --max-diff-input-lines 10000000 \
          path/to/your/tails-amd64-${VERSION:?}.iso \
          path/to/jenkins/tails-amd64-${VERSION:?}.iso

Do the same for the USB image as well.

Then carefully investigate the diffoscope report:

  • If you cannot rule out that the difference is harmful: let's take a step back; we might be compromised, so we are in no position to release. Halt the release, involve the rest of the Board, and then try to re-establish trust in all build machines and infra involved, etc. Have fun!

  • Otherwise, if the change is definitely harmless:

    • If the source of non-determinism is identified quickly and is easy and fast to fix, and the QA of the current images has not gone very far (so at least that time is not wasted), then you should consider abandoning the current version, and immediately start preparing an emergency release with:

      • the reproducibility fix,
      • a new changelog entry,
      • adjustments to the release notes so they are re-purposed for this emergency release (the abandoned release gets none, since it effectively never will be released publicly).

    • Otherwise, if the fix looks time-consuming or difficult, let's release anyway. But let's add a known issue about "This Tails release does not build reproducibility" to the release notes, linking to the issue where the nature of the reproducibility failure is clearly described.

Initialize the website release branch

From now on, we don't want to push new commits on $RELEASE_BRANCH until the new release is out. Otherwise, this would break its build and the build of every branch based on it, which would effectively block other development work. So the final steps towards publishing the release will be done in a new, dedicated branch.

If preparing anything but a final release (e.g. an alpha, beta or RC):

    git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" origin/master && \
    git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

Else, if preparing a final release:

    git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" "${TAG:?}" && \
    git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

Generate the OpenPGP signatures and Torrents

Run the following command to create a directory with a suitable name, go there, move the built images to this brand new directory, generate detached OpenPGP signatures for the images to be published (in the same directory as the images and with a .sig extension), then go up to the parent directory, create a .torrent file and check the generated .torrent files metadata:

./bin/generate-images-signatures-and-torrents

Get the internal manual testing started

Email tails-board@boum.org and tails-manual-testers@boum.org that tests may start:

  • reply to the email you've sent them earlier

  • encrypt and sign your email

  • point them to the images that Jenkins has built:

     echo "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_IMAGES_BUILD_ID:?}/archive/build-artifacts/"

    • if any of the expectations set in your initial email are outdated, update them now

    • make it clear that the Torrents don't work yet

    • attach Torrents, .packages, .iso.sig and .img.sig:

      ls -1 "$ISOS/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.packages" "$ISOS/tails-amd64-${VERSION:?}/"tails-amd64-${VERSION:?}.{iso,img}.sig "$ISOS/tails-amd64-${VERSION:?}".{iso,img}.torrent

Prepare incremental upgrades

Reference: design documentation

Sanity checks

Check that you have the correct version of squashfs-tools-ng and libsquashfs1 installed:

[ "$(dpkg-query --showformat '${Version}\n' --show squashfs-tools-ng)" \
  = '1.0.4-1' \
] || echo 'ERROR! Your squashfs-tools-ng is not the required version, so any generated IUKs will *not* be reproducible!'
[ "$(dpkg-query --showformat '${Version}\n' --show libsquashfs1)" \
  = '1.0.4-1' \
] || echo 'ERROR! Your libsquashfs1 is not the required version, so any generated IUKs will *not* be reproducible!'

Add to $IGNORED_TAGS, in ./bin/iuk-source-versions, any recent alpha or beta release that gets no security support, and as such, shall not get automatic upgrades. Here is a useful heuristics: if we did not do our manual test suite on a release, it does not get security support.

Build the Incremental Upgrade Kits locally

Verify there's enough free disk space in $IUKS_DIR:

MIN_DISK_SPACE=30000
FREE_DISK_SPACE=$(/bin/df --block-size=M --output=avail "${IUKS_DIR:?}" \
   | tail -n1 | sed --regexp-extended 's,M$,,')
if [ "$FREE_DISK_SPACE" -lt "$MIN_DISK_SPACE" ]; then
   echo "ERROR! Not enough free space in ${IUKS_DIR:?}"
fi

To build the IUKS, run this command:

(
   set -eu
   WORK_DIR=$(mktemp -d)
   TAILS_REMOTE="$(git -C "${RELEASE_CHECKOUT?:}" remote get-url origin)"
   PUPPET_TAILS_REMOTE=$(echo -n "${TAILS_REMOTE?:}" | perl -p -E 's,:tails/tails(?:[.]git)?\z,:tails/puppet-tails,')
   cd "${WORK_DIR?:}"
   git clone "$PUPPET_TAILS_REMOTE"
   time                                                                   \
   sudo                                                                   \
   ./puppet-tails/files/jenkins/slaves/isobuilders/wrap_tails_create_iuks \
       --tails-git-remote "file://${RELEASE_CHECKOUT?:}/.git"             \
       --tails-git-commit "${TAG?:}"                                      \
       --source-date-epoch "${SOURCE_DATE_EPOCH?:}"                       \
       --local-isos-dir "${ISOS?:}"                                       \
       --tmp-dir "${TMPDIR:-/tmp}"                                        \
       --output-dir "${IUKS_DIR?:}"                                       \
       --source-versions "${IUK_SOURCE_VERSIONS?:}"                       \
       --new-version "${VERSION?:}"                                       \
       --verbose                                                          \
       --jobs "$(grep '^core id' /proc/cpuinfo | sort -u | wc -l)"
) && \
 cd "${IUKS_DIR?:}" && \
 sha256sum Tails_amd64_*_to_${VERSION?:}.iuk > "${IUKS_HASHES?:}"

Take note of the time output above and add it to the table in #17750.

This command takes a long time. In parallel, while it is running, you can follow the next steps:

  • ISO history
  • Build the Incremental Upgrade Kits on Jenkins
  • If you expect the need to run some manual tests and you have one spare computer, you can prepare a USB stick and do so.

ISO history

In this section, you will push the released ISO and USB images and their artifacts (.buildlog, .build-manifest, and .packages files) to our Tails ISO history git-annex repo, so that:

  • The Jenkins build_IUKs job can fetch them.
  • Our isotesters can fetch them from there for their testing.

Make sure that Jenkins has finished building the images so you have set MATCHING_JENKINS_IMAGES_BUILD_ID (see above), then run:

cd "${RELEASE_CHECKOUT:?}" && \
./bin/add-release-to-iso-history \
    --version "${VERSION:?}" \
    --isos "${ISOS:?}" \
    --release-branch "${RELEASE_BRANCH:?}" \
    --matching-jenkins-images-build-id "${MATCHING_JENKINS_IMAGES_BUILD_ID:?}"

Then, wait (a few minutes, */15 crontab) until the images appear on https://iso-history.tails.boum.org/?C=M&O=D.

Build the Incremental Upgrade Kits on Jenkins

  1. Make sure the push to ISO history (started in the previous section) has finished, and that images have appeared on the web server: https://iso-history.tails.boum.org/?C=M&O=D

  2. On https://jenkins.tails.boum.org/job/build_IUKs/configure, adjust the SOURCE_VERSION axis to list all versions in $IUK_SOURCE_VERSIONS, and save the updated configuration.

  3. On https://jenkins.tails.boum.org/job/build_IUKs/build?delay=0sec, fill the form with these values:

    • TAILS_GIT_COMMIT: the value of $TAG in your release environment
    • SOURCE_DATE_EPOCH: the value of $SOURCE_DATE_EPOCH in your release environment, i.e. the same value you are using to "Build the Incremental Upgrade Kits locally"
    • NEW_VERSION: the value of $VERSION in your release environment
    • EXTRA_ARGS: leave it blank

  4. Click the Build button

  5. After a few seconds, a new build appears on top of the Build History sidebar. Click on the progress bar of this new build.

  6. Record the ID of the Jenkins build_IUKs build:

    1. To update the ~/.config/tails/release_management/current.yml template with newly required variables, run:

      ./bin/rm-config generate-boilerplate --stage built-iuks

    2. Edit ~/.config/tails/release_management/current.yml and set the candidate_jenkins_iuks_build_id value to the ID of that build_IUKs build (an integer):

      "${EDITOR:?}" ~/.config/tails/release_management/current.yml

    3. Generate the resulting environment variables and export them into your environment:

      . $(./bin/rm-config generate-environment --stage built-iuks)

  7. Wait until the build_IUKs build completes successfully. It should take about 20-25 minutes for each member of the $IUK_SOURCE_VERSIONS list, distributed across isobuilderN workers.

    If you have a fast computer, you'll be waiting for Jenkins to be done, while your CPU is idle. Consider starting the automated test suite in the meanwhile to optimize this: running it does not depend on any of the IUK-related steps.

  8. Add the time it took for build_IUKs to the table of #17750.

Verify that Jenkins reproduced your IUKs

Run this command to verify that Jenkins reproduced your IUKs:

"${RELEASE_CHECKOUT:?}"/bin/copy-iuks-to-rsync-server-and-verify \
   --hashes-file "${IUKS_HASHES:?}" \
   --work-dir /srv/tmp \
   --jenkins-build-id "${CANDIDATE_JENKINS_IUKS_BUILD_ID:?}"

The download step takes a few minutes.

If this verification succeeds, good! Otherwise, see the next section

If IUK verification fails

if the last command failed:

  1. Visit this page: https://jenkins.tails.boum.org/job/build_IUKs/lastBuild/parameters/

    It tells you which parameters you've passed to the upstream Jenkins job, which spawned the many workers. Make sure to use the build history browser to find the right build_IUKs job in case there were several attempts; the URL above assumes the last build is what you're interested in (see lastBuild part).

  2. Double-check that you've passed the correct parameters.

    If you notice you made a mistake, build the IUKs on Jenkins again, and do the verification again.

    Else, if the parameters where correct, then follow the next steps.

  3. Leave the IUKs built by Jenkins where they already are (in rsync.lizard:/srv/tmp): at least in some cases, this will speed up uploading your own IUKs later by a 500+ factor, thanks to rsync --partial --inplace.

  4. File an issue about this problem.

    Specify:

    • Which set of parameters you've passed to the build_IUKs job, so that the person who'll investigate the problem can reproduce it.
    • The ID of the build that failed to reproduce your locally-built IUKs.

    Set:

    • /label ~"To Do" ~"Core Work:Foundations Team" ~"C:Continuous Integration" ~"C:Upgrader" ~"C:Release management"
    • milestone: the next release

  5. Later on, after you're done with OpenPGP signing, you will upload the IUKs you've built locally.

Prepare upgrade-description files

  1. Prepare upgrade-description files (see the upgrade-description files specification for details).

    In most cases the example command below is exactly the one you should run. But in order to tell whether you're in one of the exceptional cases when you have to adjust that command, it's important that you understand what follows.

    At this step, we use the tails-iuk-generate-upgrade-description-files tool in order to:

    1. Create a new upgrade-description for the version being released and for the next one, that expresses that no upgrade is available for these ones yet.

      This is what the --version and --next_version arguments in the example command below do. You do not need to modify them.

    2. For every recent previous release that's not listed in $IUK_SOURCE_VERSIONS (for example those based on an older Debian release): tell the users of that old version they need to manually upgrade to the version being released.

      To do so, pass this previous version to --previous_version. You need to do this manually as the example command below won't do it automatically.

    Run this command, after adjusting it if needed as explained above:

     ${RELEASE_CHECKOUT:?}/config/chroot_local-includes/usr/src/iuk/bin/tails-iuk-generate-upgrade-description-files \
     --version "${VERSION:?}" \
     $( \
         if [ "${DIST:?}" = stable ]; then
             echo \
                 --next_version "${NEXT_PLANNED_MAJOR_VERSION:?}" \
                 --next_version "${NEXT_PLANNED_MAJOR_VERSION:?}~rc1" \
                 --next_version "${NEXT_PLANNED_BUGFIX_VERSION:?}" \
                 --next_version "${NEXT_POTENTIAL_EMERGENCY_VERSION:?}"
         else
             echo \
                 --next_version "${NEXT_PLANNED_MAJOR_VERSION:?}" \
                 --next_version "${SECOND_NEXT_PLANNED_MAJOR_VERSION:?}"
         fi
     ) \
     $( \
         for version in $(echo ${IUK_SOURCE_VERSIONS:?}); do
            echo "--previous_version ${version:?}"
         done \
     ) \
     --iso "${ISO_PATH:?}" \
     --iuks "${IUKS_DIR:?}" \
     --release_checkout "${RELEASE_CHECKOUT:?}" \
     --major_release "${MAJOR_RELEASE:?}" \
     --channel "${DIST:?}"

  2. Create an armoured detached signature for each created or modified upgrade-description file.

     ./bin/sign-updated-udfs

  3. Add and commit the upgrade-description files and their detached signatures to the Git branch used to prepare the release ($WEBSITE_RELEASE_BRANCH):

     cd "${RELEASE_CHECKOUT:?}" && \
 git add wiki/src/upgrade && \
 git commit -m "Update upgrade-description files." && \
 git push origin ${WEBSITE_RELEASE_BRANCH:?}

  4. Copy the generated UDF for the previous stable release to the test channel in $MASTER_CHECKOUT, modify their content accordingly, sign them, commit and push:

     ./bin/publish-test-udfs

Prepare the image description file for the download verification

If preparing a RC, skip this part.

Update the image description file (IDF) used by the verification JavaScript:

cd "${RELEASE_CHECKOUT?:}" && \
./bin/idf-content \
   --version "${VERSION:?}" \
   --iso "${ISO_PATH:?}" \
   --img "${IMG_PATH:?}" \
   > wiki/src/install/v2/Tails/amd64/stable/latest.json && \
git add wiki/src/install/v2/Tails/amd64/stable/latest.json && \
git commit -m "Update IDF for download verification" && \
git push origin "${WEBSITE_RELEASE_BRANCH:?}"

Done with OpenPGP signing

Unplug your OpenPGP smartcard and store it away, so you don't plug it back semi-mechanically later on.

Beware! If your have to plug your OpenPGP smart card or reassemble the key again after this point it invalidates everything done for the reproduction of this release so it has to be started from the beginning:

  • the original text is restored on the pad, and
  • some tester follows it from scratch, and
  • the Trusted Reproducer follows awaits the new input from said tester and then starts from scratch.

So please try to avoid this!

Upload images

Sanity check

Verify once more that the Tor Browser we ship is still the most recent (see above).

Delete obsolete IUKs

Skip this section if you're preparing a non-final release (beta, RC).

Delete the IUKs that upgrade to any beta or RC corresponding to the version you are preparing:

  • first check that it's not going to remove anything we want to keep:

     ssh rsync.lizard /bin/sh -c \
     \"find /srv/rsync/tails/tails/alpha/iuk/v2  \
            /srv/rsync/tails/tails/stable/iuk/v2 \
            -type f \
            -name "*_to_${VERSION:?}~*.iuk" \
            -not -name '*~test_*~test.iuk' \
            -not -name '*~testoverlayfs_*~testoverlayfs.iuk' \
            -not -name '*~testoverlayfsng_*~testoverlayfsng.iuk' \
            -ls \
     \"

  • then actually delete the files:

     ssh rsync.lizard /bin/sh -c \
     \"find /srv/rsync/tails/tails/alpha/iuk/v2  \
            /srv/rsync/tails/tails/stable/iuk/v2 \
            -type f \
            -name "*_to_${VERSION:?}~*.iuk" \
            -not -name '*~test_*~test.iuk' \
            -not -name '*~testoverlayfs_*~testoverlayfs.iuk' \
            -not -name '*~testoverlayfsng_*~testoverlayfsng.iuk' \
            -print -delete \
     \"

Rationale: avoid the need for mirrors to store 3 concurrent sets of IUKs (#17944). It's OK to delete them now, because we're past the deadline we've announced in the call for testing.

Publish the ISO, IMG, and IUKs over HTTP

If the IUKs reproducibility check you did earlier has failed, then upload the IUKs you've built to our rsync server (we trust your machine more than our Jenkins):

for source_version in $(echo ${IUK_SOURCE_VERSIONS:?}); do
   rsync --partial --inplace --progress -v \
      "${IUKS_DIR:?}/Tails_amd64_${source_version:?}_to_${VERSION:?}.iuk" \
      rsync.lizard:/srv/tmp/
done

While waiting for the IUKs to be uploaded, you can proceed with the next steps.

Upload the ISO and USB image signatures to our rsync server:

scp "${ISO_PATH:?}.sig" "${IMG_PATH:?}.sig" rsync.lizard:

Copy the ISO and USB images to our rsync server, verify their signature, move them in place with proper ownership and permissions:

cd "${RELEASE_CHECKOUT:?}" && \
./bin/copy-images-to-rsync-server-and-verify \
    --version "${VERSION:?}" \
    --dist "${DIST:?}" \
    --release-branch "${RELEASE_BRANCH:?}" \
    --matching-jenkins-images-build-id "${MATCHING_JENKINS_IMAGES_BUILD_ID:?}"

Update the time in project/trace file on our rsync server and on the live website (even for a release candidate):

./bin/update-trace-time \
   "Updating trace file after uploading the ISO and USB images for ${VERSION:?}."

At this stage, either IUKs were reproduced by Jenkins and left on the rsync server in a temporary location, or you must wait until the upload of the locally built IUKs (started in the first part of this section) is completed.

Finally, move the IUKs in place with proper ownership and permissions and update the time in project/trace file on our rsync server and on the live website (even for a release candidate):

./bin/publish-iuks

Announce, seed and test the Torrents

Check if there's enough space on our Bittorrent seed to import the new ISO and USB images:

ssh bittorrent.lizard df -h /var/lib/transmission-daemon/downloads

If not, list already running Torrents:

ssh bittorrent.lizard transmission-remote --list

… set $ID to the oldest one and delete it (do this both for the ISO and USB images):

ssh bittorrent.lizard transmission-remote -t "${ID:?}" --remove-and-delete

… and finally check disk space again:

ssh bittorrent.lizard df -h /var/lib/transmission-daemon/downloads

Now you can announce and seed the Torrents for the release you're preparing:

./bin/announce-and-seed-torrents

Test that you can start downloading the ISO and USB images with a BitTorrent client. It can take a few dozen minutes before this works.

Run the automated test suite

This is not a required step: check the manual testers pad and see if anyone volunteered for that. But if noone did, start the automated test suite. The sooner, the better.

You probably already know how to do that, but keep this in mind:

  • you need to use the --old-iso option

  • you probably want to specify --tmpdir to point to a non-volatile directory. The default lives in /tmp/, which might lead to unpleasant surprises if, for whatever reasons, you need to turn off your computer.

Public testing

  1. Using check-mirrors, choose a fast mirror that already has the tentative ISO and USB images. E.g. https://mirrors.edge.kernel.org/tails/ or https://mirrors.wikimedia.org/tails/ are reliable and have plenty of bandwidth:

     (
     success=false
     while true; do
         for mirror in mirrors.wikimedia.org mirrors.edge.kernel.org; do
             url="https://$mirror/tails/"
             if [[ -z "$(./check-mirrors.rb --allow-multiple --fast --channel ${DIST:?} \
                 --url-prefix "$url" \
                 tails-amd64-${VERSION:?} )" ]]
             then
                 echo "WORKING: $url"
                 success=true
             fi
         done
         $success && exit 0
         sleep 1m
     done
 )

  2. Email tails-testers@boum.org to ask them to test the tentative images:

    • subject: "Call for testing: $VERSION"
    • Point them to the up-to-date mirror you've found previously.
    • This is a public list, don't point to the pad.
    • Make a deadline clear: until when is feedback useful?

Lead the internal manual testing to conclusion

  1. Manual testers can now do the "Reproducibility" part of the manual test suite.

    To communicate this to them, delete "⚠ Wait until the Release Manager […]" in the "Reproducibility" section on the pad.

  2. Make sure that enough people are here to run the tests, that they report their results in due time, and that they make it clear when they're leaving for good.

  3. Ensure the "Incremental upgrades" part of the manual test suite is done:

    1. Wait until most mirrors in the pool have the new version:

      ./check-mirrors.rb --channel ${DIST:?} \ --allow-multiple --fast --stats \ tails-amd64-${VERSION:?}

    2. Delete "⚠ Wait until the Release Manager […]" in the "Incremental upgrades" section on the pad.

  4. Fill the holes and make sure that the automated and manual test suites are done in due time.

  5. If you were the one running the automated test suite, check its results.

  6. Triage test results, reproduce bugs as needed, decide what the next step is and make sure it happens: add to known issues? file an issue? release blocker? improve the test description (steps, expected outcome)?

Prepare announcements and blog posts

What follows in this section happens on the $WEBSITE_RELEASE_BRANCH branch in ${RELEASE_CHECKOUT:?}, making sure we catch up with the technical writers who have likely prepared release notes:

cd "${RELEASE_CHECKOUT:?}" && \
   git checkout "${WEBSITE_RELEASE_BRANCH:?}" && \
   git fetch && \
   git merge "origin/${WEBSITE_RELEASE_BRANCH?:}"

If no release notes where prepared, we add one with placeholders that the technical writes can update post-release:

cp wiki/src/news/version_${PREVIOUS_STABLE_VERSION:?}.mdwn \
   wiki/src/news/version_${VERSION:?}.mdwn &&
${EDITOR:?} wiki/src/news/version_${VERSION:?}.mdwn

Make sure to:

  1. adjust the !meta date at the top.
  2. adjust all version numbers everywhere appropriately.
  3. add placeholders to the "Changes and updates" and "Fixed problems", e.g. "This section will be updated soon. Sorry for the inconvenience!".
  4. add any known issues to the "Known issues" section.

If preparing a major or a bugfix release

Skip this part if preparing a RC.

Rename, copy, garbage collect and update various files:

./bin/add-release-files-to-website

Then, build the website and commit this last set of changes:

if [ "$(./bin/po4a-version)" = 0.55 ]; then
    ./build-website && \
    git add wiki/src/{inc,torrents} && \
    git commit -m "Update various website source files for ${VERSION:?}"
else
    echo "Wrong po4a version"
fi

Ensure our technical writer has written the announcement for the release in wiki/src/news/version_${TAG:?}.mdwn.

Generate an announcement mentioning the security vulnerabilities affecting the previous version in wiki/src/security/known_security_vulnerabilities_in_${PREVIOUS_STABLE_VERSION:?}.mdwn, in order to let the users of the old versions know that they have to upgrade:

    ./bin/generate-security-advisory \
       --previous-version "${PREVIOUS_STABLE_VERSION:?}" \
       --version "${VERSION:?}" \
       --tag "${TAG:?}" \
    > "wiki/src/security/known_security_vulnerabilities_in_${PREVIOUS_STABLE_VERSION:?}.mdwn"

Remove obsolete bits from wiki/src/home/testing.html. For example, remove any text that's specific to a release candidate we may have published for the version you are preparing.

If preparing a release candidate

Skip this part if preparing a final release.

Copy the signatures and the Torrents into the website repository:

cp "${ISO_PATH:?}.sig" \
   "${IMG_PATH:?}.sig" \
   "${ISOS:?}/tails-amd64-${VERSION:?}".{iso,img}.torrent \
   "${RELEASE_CHECKOUT:?}/wiki/src/torrents/files/"

Write the announcement for the release:

  1. Set this environment variable, in the format YYYY-MM-DD:

    • FINAL_RELEASE_DATE: the date when we plan to publish the final release you're preparing a RC for

  2. Generate the boilerplate contents from the template:

     ./bin/generate-call-for-testing \
    --version "${VERSION:?}" \
    --tag "${TAG:?}" \
    --date "${RELEASE_DATE:?}" \
    --final-version "${NEXT_PLANNED_MAJOR_VERSION:?}" \
    --final-date "${FINAL_RELEASE_DATE:?}" \
 > ${RELEASE_CHECKOUT:?}/wiki/src/news/test_${TAG:?}.mdwn

  3. Edit ${RELEASE_CHECKOUT:?}/wiki/src/news/test_${TAG:?}.mdwn:

    • Mention major user-visible improvements.

    • Document important config changes that users of the Persistent Storage have to do themselves (e.g. the Pidgin proxy settings change in 9925321 breaks all existing persistent profiles).

    • Document known issues.

    • Look for FIXME.

In wiki/src/home/testing.html, in the "Current calls for testing" section, add a bullet point with a pointer to the call for testing for this release candidate (https://tails.boum.org/news/test_$TAG/, that you just wrote).

In any case

Generate PO files for the pages added or updated earlier in this section, and publish them on the $WEBSITE_RELEASE_BRANCH branch:

if [ "$(./bin/po4a-version)" = 0.55 ]; then
    ./build-website && \
    git add wiki/src && \
    git commit -m "Releasing version ${VERSION:?}" && \
    git push origin "${WEBSITE_RELEASE_BRANCH:?}"
else
    echo "Wrong po4a version"
fi

Draft the Tor blog post

Skip this if you are preparing anything but a final, major release.

  • Generate the draft blog post:

      ./bin/generate-Tor-blog-post "${VERSION:?}"

  • Proof-read the draft blog post

  • login to the Tor blog. The credentials are in keyringer in rm.git.
  • click Content → Add content → Blog Post
  • add these tags, each in a separate text entry field (use Add another item):
    • tails
    • anonymous operating system
    • tails releases
  • set Title to "New Release: Tails $VERSION"
  • set URL alias to "/new-release-tails-XYZ" (without dots)
  • choose Filtered HTML as the Text format in the blog post editor
  • copy the text you have prepared into the Post Body textarea of the blog post editor
  • open Comment Settings and verify that comments are Closed
  • open Promotion Options and check Promoted to front page
  • click Preview and ensure everything is OK
  • click Back to content editing
  • ensure the Publish checkbox is not checked
  • click Save to save the draft

Go wild!

Wait for the HTTP mirrors to catch up

Test downloading the ISO, USB image, and one of the new IUKs over HTTP.

Check if every active mirror in the pool has the new version:

./check-mirrors.rb --channel ${DIST:?} \
    --allow-multiple --fast --stats \
    tails-amd64-${VERSION:?}

To disable mirrors that are lagging behind:

  1. Set their weight to 0 in the mirrors pool.
  2. Push to the tails/mirror-pool master branch.
  3. Verify that the CI pipeline passed on tails/mirror-pool/-/pipelines.
  4. Notify tails-mirrors@boum.org, so they communicate with the operators of the outdated mirrors.

Sanity checks

  • Check for any feedback in the Internal testing pad.
  • Check for any feedback on tails-testers@boum.org.
  • Verify once more that the Tor Browser we ship is still the most recent (see above).

Push

Push the last commits to our Git repository and put master in the following state:

( cd "${RELEASE_CHECKOUT:?}" && \
  git push origin \
      "${WEBSITE_RELEASE_BRANCH:?}:${WEBSITE_RELEASE_BRANCH:?}" \
      devel:devel \
) && \
( cd "${MASTER_CHECKOUT:?}" && \
  git fetch && \
  git merge origin/master && \
  git merge --edit "origin/${WEBSITE_RELEASE_BRANCH:?}" && \
  git submodule update --init && \
  echo "stable" > config/base_branch && \
  git commit config/base_branch \
      -m "Restore master's base branch." \
)

Close the "Write release notes" ticket manually since Closes: statements are not sufficient when updating master.

Finally, push the master branch to make the changes go live on our website:

( cd "${MASTER_CHECKOUT:?}" && \
  git push origin master:master \
)

Wait for the release to be fully published, that is for the website to have been fully refreshed. To do so:

Bug tracker

Skip this part if preparing a release candidate.

  1. Verify there's a missed:$VERSION label on groups/tails/-/labels; if there's none, submit a MR against tails/gitlab-config to create one.

  2. Ensure you have an up-to-date tails:gitlab-triage-stable container image:

     cd "${MASTER_CHECKOUT:?}" && \
 git checkout master && \
 ./config/gitlab-triage/bin/ensure-up-to-date-container-image

  3. Postpone to the next scheduled release any remaining open issue and merge request whose milestone is the version you've just released:

     cd "${MASTER_CHECKOUT:?}" && \
 git checkout master && \
 ./bin/gitlab-triage-post-release \
     --host-url "$(bin/gitlab-url TailsRM)" \
     --token "$(bin/gitlab-api-token TailsRM)"

  4. Finally, submit a MR against tails/gitlab-config to close the just-released milestone. See this as an example: tails/gitlab-config!18

    A cheap way to do this, could be to use Gitlab web interface to create the MR

    Assign the MR to the on-duty sysadmin (see link to "Semi-public calendar" in https://gitlab.tails.boum.org/tails/summit/-/wikis/Teams#sysadmins).

Issues linked from the website

  1. Go through the issues linked from the documentation and support sections of the website to identify the issues that might be resolved in this release.

    cd "${MASTER_CHECKOUT:?}" && \
find wiki/src/{doc,support} ! -path wiki/src/support/known_issues/graphics.mdwn -name "*.mdwn" -o -name "*.html" | xargs cat | \
    ruby -e 'puts STDIN.read.scan(/\[\[!tails_ticket\s+(\d+)[^\]]*\]\]/)' | \
    while read ticket; do
        ticket_status="$(python-gitlab --gitlab TailsRM -o json --fields state project-issue get --project-id 2 --iid "$ticket" | jq --raw-output .state)"
        if [ -z "${ticket_status:-}" ]; then
            echo "Failed to find the status of #${ticket:?}" >&2
            continue
        fi
        if [ "${ticket_status:?}" = "closed" ]; then
            echo "It seems issue #${ticket:?} has been fixed. Please find all instances on the website and adjust them as needed. Ticket URL: https://gitlab.tails.boum.org/tails/tails/-/issues/${ticket:?}"
        fi
    done

  2. Create a GitLab issue to list them.

    Title: Issues linked from the website for Tails $VERSION Labels: Core Work:Technical writing and To Do

Twitter

Check in the comments of the issue for the release notes if the technical writers have prepared a tweet. Otherwise tweet a simple link to the release notes:

echo "Tails ${VERSION:?} is out: https://tails.boum.org/news/version_${TAG:?}/"

The credentials are in keyringer in rm.git.

Publish the Tor blog post

If you prepared a draft Tor blog post earlier, then:

  1. Open the draft Tor blog post you have prepared earlier
  2. Adjust Authoring Information → Authored on to make the publication date correct
  3. Click Save and publish

Amnesia news

Log in and accept the release announcement (it's been automatically sent to amnesia-news@ every half hour and is stuck in the moderation queue).

The password is in keyringer in rm.git.

Prepare for the next development cycle

If you just released a final release

  1. If you just released a new stable release, remove the previous stable release from:

    • our rsync server: ssh rsync.lizard sudo rm -rf /srv/rsync/tails/tails/stable/tails-amd64-${PREVIOUS_STABLE_VERSION:?}/

    • our Bittorrent seed: get the previous release's Transmission IDs (ISO and USB image, use comma-separated values) with:

      ssh bittorrent.lizard transmission-remote --list | grep --fixed-strings "tails-amd64-${PREVIOUS_VERSION:?}-i"

      Then delete them with:

      ssh bittorrent.lizard transmission-remote \
    -t "${PREVIOUS_VERSION_TRANSMISSION_ID:?}" --remove-and-delete

      Finally, check that everything looks good:

      ssh bittorrent.lizard transmission-remote --list

  2. Remove any remaining RC for the just-published release from rsync.lizard:/srv/rsync/tails/tails/alpha/.

    1. Check if there is anything:

      ssh rsync.lizard find /srv/rsync/tails/tails/alpha/ -type f

    2. Decide what to do!

  3. If you've published a final release, remove IUKs that upgrade to an older version as they were superseded by this release:

    • first check that it's not going to remove anything we want to keep:

      ssh rsync.lizard /bin/sh -c \
    \"find /srv/rsync/tails/tails/alpha/iuk/v2  \
           /srv/rsync/tails/tails/stable/iuk/v2 \
           -type f -name '*.iuk' \
           -not -name "*_to_${VERSION:?}.iuk" \
           -not -name '*~test_*~test.iuk' \
           -not -name '*~testoverlayfs_*~testoverlayfs.iuk' \
           -not -name '*~testoverlayfsng_*~testoverlayfsng.iuk' \
           -ls \
    \"

    • then actually delete the files:

      ssh rsync.lizard /bin/sh -c \
    \"find /srv/rsync/tails/tails/alpha/iuk/v2  \
           /srv/rsync/tails/tails/stable/iuk/v2 \
           -type f -name '*.iuk' \
           -not -name "*_to_${VERSION:?}.iuk" \
           -not -name '*~test_*~test.iuk' \
           -not -name '*~testoverlayfs_*~testoverlayfs.iuk' \
           -not -name '*~testoverlayfsng_*~testoverlayfsng.iuk' \
           -print -delete \
    \"

  4. Check how much space our mirrors need:

     ssh rsync.lizard du -sh /srv/rsync/tails

    Compare it to the minimum disk space we ask of our mirror operators (80 GiB) and determine if any further action is needed to either reduce our usage by deleting stuff, or asking them to give us more space.

  5. Delete Git branches that were merged:

     cd "${MASTER_CHECKOUT:?}" && \
 git checkout master && \
 git fetch && \
 git merge origin/master && \
 git submodule update --init && \
 bare_repo=$(mktemp -d) && \
 git clone --bare --reference "${MASTER_CHECKOUT:?}" \
    git@gitlab-ssh.tails.boum.org:tails/tails \
    "${bare_repo:?}" && \
 PYTHONPATH=lib/python3 ./bin/delete-merged-git-branches \
    --repo "${bare_repo:?}" && \
 rm -rf "${bare_repo:?}"

  6. On the stable branch, remove all old versions that were never released from wiki/src/upgrade/v2/Tails:

     cd "${RELEASE_CHECKOUT:?}" && \
 git checkout stable && \
 ./bin/remove-unused-udfs --before-version "${VERSION:?}"

    Explanation: the post-release APT repository steps from the previous stable release will usually have had us prepare for an emergency release that was never made.

  7. Pull master back and merge it into stable:

     cd "${RELEASE_CHECKOUT:?}" && \
 git fetch origin && \
 git checkout stable && \
 git merge origin/master

    Note: After a final major release is published, it's normal to leave the testing branch alone: it will only become relevant again when the next freeze starts.

  8. Put the stable Git branch and the corresponding APT suite in the desired shape:

     if [ "${MAJOR_RELEASE:?}" -eq 1 ]; then
    ./bin/reset-custom-APT-suite stable testing && \
    git checkout stable && \
    if [ $(find config/APT_overlays.d -maxdepth 1 -type f | wc -l) -gt 1 ]; then
       git rm config/APT_overlays.d/* && \
       git commit config/APT_overlays.d/ \
          -m "Empty the list of APT overlays: they were merged"
    fi
 fi

  9. Merge the release branch and its APT suite into devel:

     ./bin/merge-main-branch "${RELEASE_BRANCH:?}" devel

    When there's a merge conflict, the script aborts to let you resolve the conflict. Then:

    1. Resolve the conflict, git add and git merge --continue.
    2. Re-run the exact same merge-main-branch command.

  10. Check if the version number of the last entry in devel's debian/changelog matches the next major release, so that next builds from the devel branch do not use the APT suite meant for the last release; if this is not the case yet, run:

     cd "${RELEASE_CHECKOUT}" && \
 git checkout devel && \
 dch --newversion "${NEXT_PLANNED_MAJOR_VERSION:?}" \
    "Dummy entry for next release." && \
 git commit debian/changelog \
    -m "Add dummy changelog entry for ${NEXT_PLANNED_MAJOR_VERSION:?}."

  11. If this command outputs anything:

      git show devel:config/chroot_apt/preferences \
     | grep -E '^Explanation: freeze exception'

    Then checkout the devel branch, and there thaw the packages that were granted freeze exceptions.

  12. Increment the version number in stable's debian/changelog, so that next builds from the stable branch do not use the APT suite meant for the last release:

     cd "${RELEASE_CHECKOUT}" && \
 git checkout stable && \
 dch --newversion "${NEXT_STABLE_CHANGELOG_VERSION:?}" \
    "Dummy entry for next release." && \
 git commit debian/changelog \
    -m "Add dummy changelog entry for ${NEXT_STABLE_CHANGELOG_VERSION:?}."

  13. Verify that the snapshots used in the release branch are ok, e.g. they use the correct snapshots, and they were bumped appropriately (they should expire after the next planned major release date). Look carefully at the output of this command:

     cd "${RELEASE_CHECKOUT:?}" && \
 git checkout "${RELEASE_BRANCH:?}" && \
 ./bin/apt-snapshots-expiry

  14. Push the resulting branches:

     git push origin stable testing devel

  15. In calendar (master branch), remove the entries about the version that you've just released.

  16. Check the Mozilla release calendars:

    If the upcoming release dates in the Mozilla calendar does not match what we have in calendar, then alert tails-dev@boum.org, tails-rm@boum.org, and explicitly Cc the designated RM for that upcoming release.

  17. Announce the date of the next release on tails-dev@boum.org and tails-l10n@boum.org.

    • Cc tails-manual-testers@boum.org, asking them to privately tell the RM for that release how much manual testing they can do on release day.
    • Subject: "Release schedule for Tails NEXT_VERSION_NUMBER"
    • If a release candidate is planned, include it in the release schedule and the call for manual testers.

  18. If you are the release manager for the next release too, look at the tasks that must be done at the beginning of your shift in the release manager role page.

    Otherwise, kindly remind the next release manager about this :)

    • Send an email to the next RM with:
      • subject "RM for Tails NEXT_VERSION_NUMBER"
      • body: "This is a reminder that you will be RM for Tails NEXT_VERSION_NUMBER. Please see https://tails.boum.org/contribute/working_together/roles/release_manager/#shift".
    • If there's a release candidate scheduled before our next release, do the same for the release manager of that release candidate.

  19. Put aside the configuration file used for this release:

     mkdir -p ~/.config/tails/release_management/old && \
 mv ~/.config/tails/release_management/current.yml \
    ~/.config/tails/release_management/old/${VERSION:?}.yml.disabled

  20. You're done for today! Suggestion: enjoy some time AFK :)

  21. Tomorrow, make sure Jenkins manages to build all updated major branches: https://jenkins.tails.boum.org/view/RM/.

If you just released an RC

  1. If this command outputs anything:

     git show devel:config/chroot_apt/preferences \
    | grep -E '^Explanation: freeze exception'

    Then checkout the devel branch, and there thaw all packages (if any) that were granted freeze exceptions.

  2. On the devel branch, thaw the time-based APT repository snapshots:

     git checkout devel && \
 ./auto/scripts/apt-snapshots-serials thaw && \
 git commit \
     -m "Thaw APT snapshots." \
     config/APT_snapshots.d/*/serial || :

    This should generally be a no-op but if there was some hiccup earlier it could be needed.

  3. Increment the version number in debian/changelog on the branch used for the release, to match the upcoming non-RC release, so that the next builds from it do not use the APT suite meant for the RC:

     cd "${RELEASE_CHECKOUT}" && \
 git checkout "${RELEASE_BRANCH:?}" && \
 dch --newversion "${NEXT_PLANNED_MAJOR_VERSION:?}" \
    "Dummy entry for next release." && \
 git commit debian/changelog \
    -m "Add dummy changelog entry for ${NEXT_PLANNED_MAJOR_VERSION:?}."

  4. Increment the version number in devel's debian/changelog to match the second next major release, so that images built from there have the right version number:

     cd "${RELEASE_CHECKOUT}" && \
 git checkout devel && \
 dch --newversion "${SECOND_NEXT_PLANNED_MAJOR_VERSION:?}" \
    "Dummy entry for next release." && \
 git commit debian/changelog \
    -m "Add dummy changelog entry for ${SECOND_NEXT_PLANNED_MAJOR_VERSION:?}."

  5. Merge testing into devel (keeping a single changelog entry, for the second next major release), and push them:

     cd "${RELEASE_CHECKOUT:?}" && \
 git checkout devel && \
 git merge testing && \
 git push origin testing devel

  6. Follow the "Verify that the snapshots used in the release branch are ok" step for final releases, above.

  7. In calendar, remove the entries about the version that you've just released.

  8. Put aside the configuration file used for this release:

     mkdir -p ~/.config/tails/release_management/old && \
 mv ~/.config/tails/release_management/current.yml \
    ~/.config/tails/release_management/old/${VERSION:?}.yml.disabled

  9. Delete your local copy of the IUKs [XXX:automate]

  10. You're done for today! Suggestion: enjoy some time AFK :)

  11. Tomorrow, on https://jenkins.tails.boum.org/view/RM/, make sure Jenkins successfully:

    • builds all updated major branches
    • builds the website
    • checks PO files

Related pages