We use a custom APT repository to store our custom packages.


We use one single APT repository hosting multiple suites:

  • We have a (read-only) suite for every past release: 0.9, 0.10.1, etc.
  • We have a suite for each main branch: stable, testing, devel, feature-jessie
  • We have an overlay suite for each topic branch: bugfix/*, feature/*, etc. Note: the APT suite corresponding to a given Git topic branch contains only the packages this branch adds to the tag or main branch it diverged from. Think of it as an overlay.
  • We also have a less formal unstable suite, that should not be used by any Tails git branch; it can be used as hosting space for other packaging work we might do, e.g. acting as upstream or Debian maintainers.
  • We also have a builder-jessie suite, used to provide additional packages needed on a Jessie system to build Tails.

The suite(s) to use as sources for APT, during the build and inside the resulting system, are determined by the content of the config/base_branch and config/APT_overlays.d/* files. See details in the Build system section below.

Basically, a cronjob fetches the Tails Git repository every few minutes, detects new branches, and accordingly create new suites in the custom APT repository.

We manage our APT repository with reprepro. See the corresponding documentation for details.

Build system

The Tails ISO build system dynamically adds APT sources that will be used during the build, and inside the resulting ISO itself.

If the last version in debian/changelog was released already (i.e. a matching tag exists), then the build system adds the suite corresponding to this release (e.g. 1.5 or 3.0), and that's all.

Else, it adds:

  • one APT source for the base branch of the one being built, as found in config/base_branch;
  • one APT source for each suite listed in config/APT_overlays.d/*; note that only the name of such files matters, and their content is ignored.

In practice, config/APT_overlays.d/ should contain:

  • for a topic branch:
    • if needed, a file that is named like the branch's own overlay APT suite; e.g. for the bugfix/12345-whatever branch, it would be called config/APT_overlays.d/bugfix-12345-whatever
    • any file representing APT suites that came from merging its base branch into this topic branch, that is:
  • for a base branch (stable, testing, devel or feature/jessie): a file for each additional, overlay APT suite that came from topic branches that ship Debian packages and were merged into this base branch since last time it was used to prepare a release.

The code that implements this is auto/scripts/tails-custom-apt-sources. It has automated tests.

At release time, the release manager:

  1. merges into the release branch's APT suite all APT overlay suites found in config/APT_overlays.d/;
  2. empties config/APT_overlays.d/ in the release branch;
  3. merges the release branch into other base branches as needed, and ensures that all resulting config/APT_overlays.d/:s make sense.

Note that a branch like feature/jessie needs to be a base branch: we want to be able to work on topic branches forked off feature/jessie.

SSH access

Configure your SSH client to connect to the APT server:

Host incoming.deb.tails.boum.org
    Port 3003

HTTP access

The custom APT repository can be browsed at https://deb.tails.boum.org/.


Importing a new package

Creating a new branch

Push your branch to Git and wait a few minutes for the new APT suite to appear on https://deb.tails.boum.org/dists/. You can look up the name of that suite there.

Then, you probably want to drop a new file in config/APT_overlays.d/, named after the APT suite corresponding to your new branch:


Incidentally, this gives you the name of the APT suite you'll have to upload to later on.

For details, see the Build system section above.

Building a package

If you want to import a package from Debian instead of building one, instead see how to grant a freeze exception.

Make sure the Distribution: field in your .changes file matches the suite you want the package to land in (e.g. pass --changes-option=-DDistribution=feature-torbrowser to pdebuild's --debbuildopts).

Make sure to have the .changes file include the original source archive (.orig.tar.{gz,bz2,xz}) if it is not already in our APT repository; this can be done by passing -sa to pdebuild's --debbuildopts.

Configuring an upload tool

Configuring dupload

Add this configuration snippet to your dupload configuration:

$config::cfg{'tails'} = {
        fqdn => "incoming.deb.tails.boum.org",
        method => "scp",
        login => "reprepro",
        incoming => "/srv/reprepro/incoming/",
        dinstall_runs => 1,

Configuring dput

Add this to .dput.cf:

fqdn            = incoming.deb.tails.boum.org
method          = scp
login           = reprepro
incoming        = /srv/reprepro/incoming/
run_dinstall    = 0

If you don't have upload rights to Debian / don't use dput to upload to anything else but Tails, you might want to also add this section to .dput.cf to use the [tails] section by default:

default_host_main = tails

Uploading and importing process

Carefully prepare and build your package. Usual precautions, (Lintian etc.) apply.

Carefully check the .changes file (especially the Distribution control field, and the included files list; the former can be fixed with the changestool(1) command, from reprepro).

If the .orig.tar.{gz,bz2,xz} tarball is present neither in the .changes file nor in our custom APT repository, add it using:

$ changestool $CHANGES_FILE includeallsources

Make sure that the .changes file lists the binary packages too: Tails has no package auto-builder that would build them from source for you.

Sign the .changes file with a key that is in the uploaders list:

$ debsign $CHANGES_FILE

Upload the files to the incoming queue:

$ dupload --to tails $CHANGES_FILE

or, using dput:

$ dput tails $CHANGES_FILE

reprepro will automatically notice the new files and import them into the suite specified in your .changes file.

Check the result:

$ ssh reprepro@incoming.deb.tails.boum.org reprepro list $SUITE $PACKAGENAME

Merging a main branch

When a Git main branch (devel, testing, stable) is merged into another main branch, the corresponding operation must be done on the APT suites.

  1. Set some environment variables:

         WORKDIR=$(mktemp -d)
         # the branch you want to merge
         # the branch you want to merge _into_
  2. Save the list of packages currently present in the APT suite we want to merge into:

         ssh reprepro@incoming.deb.tails.boum.org \
              reprepro list "$DST" \
              > "$WORKDIR/$DST.orig.list"
  3. Make sure you are not going to overwrite newer packages with older ones:

         ssh reprepro@incoming.deb.tails.boum.org \
               tails-diff-suites "$SRC" "$DST"

    … and look for packages that are newer in $DST than in $SRC.

  4. Merge the APT suites:

    1. Merge in Git and APT:

      If you have to resolve a merge conflict in debian/changelog, ensure only the latest UNRELEASED entry is present.
        git checkout "$DST" && \
        git merge "origin/$DST" && \
        git merge "$SRC" && \
        ssh reprepro@incoming.deb.tails.boum.org \
             tails-merge-suite "$SRC" "$DST"
    2. Restore the config/base_branch if needed:

        echo "${DST}" > config/base_branch && \
        git commit config/base_branch -m "Restore ${DST}'s base branch." || :
    3. Push:

        git push origin "${DST}:${DST}"
  5. Make sure not to re-add, into the branch we merge into, any package that was removed from it, but still is in the branch we merge from: e.g. when merging stable into devel, it may be that devel had some packages removed (e.g. due to previously merging a topic branch into it, whose purpose is to remove custom packages). To this end, compare the resulting list of (package, version) in the devel APT suite with the one saved before the merge, check Git merges history if needed, apply common sense, and remove from devel the packages that were removed from it a while ago, and were just erroneously re-added by the merge operation.

Resetting a suite to the state of another one

a. First, set some environment variables:

    # the suite to reset
    # the final state it should be in

b. Then, empty the OLD suite:

    ssh reprepro@incoming.deb.tails.boum.org \
          reprepro removematched $OLD '\*'

c. Finally, merge NEW into OLD

    ssh reprepro@incoming.deb.tails.boum.org \
          tails-merge-suite $NEW $OLD

Merging APT overlays

This operation merges all APT overlays listed in the given branch's config/APT_overlays.d/ into its own APT suite, empties config/APT_overlays.d/ accordingly, then commits and pushes to Git.

  1. Set some environment variables:

     # The branch that should have its overlays merged
  2. Merge the APT overlays in reprepro:

     git checkout "$BRANCH" && \
     for overlay in $(ls config/APT_overlays.d/) ; do
        if ! ssh reprepro@incoming.deb.tails.boum.org \
                tails-merge-suite "$overlay" "$BRANCH" ; then
           echo "Failed to merge '$overlay' into '$BRANCH': $?" >&2
  3. Empty config/APT_overlays.d/:

     git checkout "$BRANCH" && \
     git rm config/APT_overlays.d/* && \
     git commit config/APT_overlays.d/ \
        -m "Empty the list of APT overlays: they were merged"
  4. Push the Git branch:

     git push origin "${BRANCH}:${BRANCH}"

Tagging a new Tails release

Once the new release's Git tag is pushed, a cronjob creates a new APT suite on the custom APT repository's side within a few minutes. This new APT suite is called the same as the new release version.

Wait for this new (empty) APT suite to be created and initialize it with the packages currently found in the APT suite corresponding to the branch that is used to prepare the release:

while ! ssh reprepro@incoming.deb.tails.boum.org reprepro list "${TAG:?}" >/dev/null 2>&1; do
   sleep 5
done && \
ssh reprepro@incoming.deb.tails.boum.org \
     tails-merge-suite "$RELEASE_BRANCH" "$TAG"

After a new Tails release is out

If you just put out a final release

If the release was a major one, then:

  1. Hard reset the stable APT suite to the state of the testing one.

  2. Empty config/APT_overlays.d in the stable branch:

     git checkout stable && \
     if [ $(find config/APT_overlays.d -maxdepth 1 -type f | wc -l) -gt 1 ]; then
        git rm config/APT_overlays.d/* && \
        git commit config/APT_overlays.d/ \
           -m "Empty the list of APT overlays: they were merged"

In any case:

  • merge stable or testing into devel
  • increment the version number in devel's debian/changelog to match the next major release, so that next builds from the devel branch do not use the APT suite meant for the last release:

      cd "${RELEASE_CHECKOUT}" && \
      git checkout devel && \
      dch --newversion "${NEXT_PLANNED_MAJOR_VERSION:?}" \
         "Dummy entry for next release." && \
      git commit debian/changelog \
         -m "Add dummy changelog entry for ${NEXT_PLANNED_MAJOR_VERSION:?}."
  • increment the version number in stable's debian/changelog, so that next builds from the stable branch do not use the APT suite meant for the last release:

      cd "${RELEASE_CHECKOUT}" && \
      git checkout stable && \
      dch --newversion "${NEXT_STABLE_CHANGELOG_VERSION:?}" \
         "Dummy entry for next release." && \
      git commit debian/changelog \
         -m "Add dummy changelog entry for ${NEXT_STABLE_CHANGELOG_VERSION:?}."

Else, if you just released a RC

  • increment the version number in debian/changelog on the branch used for the release, to match the upcoming non-RC release, so that the next builds from it do not use the APT suite meant for the RC:

      cd "${RELEASE_CHECKOUT}" && \
      git checkout "${RELEASE_BRANCH:?}" && \
      dch --newversion "${NEXT_PLANNED_MAJOR_VERSION:?}" \
         "Dummy entry for next release." && \
      git commit debian/changelog \
         -m "Add dummy changelog entry for ${NEXT_PLANNED_MAJOR_VERSION:?}."
  • increment the version number in devel's debian/changelog to match the second next major release, so that images built from there have the right version number:

      cd "${RELEASE_CHECKOUT}" && \
      git checkout devel && \
      dch --newversion "${SECOND_NEXT_PLANNED_MAJOR_VERSION:?}" \
         "Dummy entry for next release." && \
      git commit debian/changelog \
         -m "Add dummy changelog entry for ${SECOND_NEXT_PLANNED_MAJOR_VERSION:?}."

Giving access to a core developer

  1. Give SSH access to the reprepro user on the system that hosts reprepro (using the ssh_authorized_key Puppet resource, until #12346 is resolved).
  2. Import the developer's public GnuPG key into the reprepro user's GnuPG keyring -- should be doable using Puppet, some day
  3. Add the developer's OpenPGP key ID to $reprepro_uploaders in our tails::reprepro Puppet module. Deploy.

Contributing without privileged access

Non-core developers without access to the "private" APT infrastructure would add the .deb they want to their Git branch as we have been doing until now, push the result to their own Git repository... and at merge time, we would rewrite their history to remove the .deb, and import it into our APT repo.