We use a custom APT repository to store our custom packages.

Overview

We use one single APT repository hosting multiple suites:

  • We have a (read-only) suite for every past release: 0.9, 0.10.1, etc.
  • We have a suite for each main branch: stable, testing, devel, feature-jessie
  • We have an overlay suite for each topic branch: bugfix/*, feature/*, etc. Note: the APT suite corresponding to a given Git topic branch contains only the packages this branch adds to the tag or main branch it diverged from. Think of it as an overlay.
  • We also have a less formal unstable suite, that should not be used by any Tails git branch; it can be used as hosting space for other packaging work we might do, e.g. acting as upstream or Debian maintainers.
  • We also have a builder-jessie suite, used to provide additional packages needed on a Jessie system to build Tails.

The suite(s) to use as sources for APT, during the build and inside the resulting system, are determined by the content of the config/base_branch and config/APT_overlays.d/* files. See details in the Build system section below.

Basically, a cronjob fetches the Tails Git repository every few minutes, detects new branches, and accordingly create new suites in the custom APT repository.

We manage our APT repository with reprepro. See the corresponding documentation for details.

Build system

The Tails ISO build system dynamically adds APT sources that will be used during the build, and inside the resulting ISO itself.

If the last version in debian/changelog was released already (i.e. a matching tag exists), then the build system adds the suite corresponding to this release (e.g. 1.5 or 3.0), and that's all.

Else, it adds:

  • one APT source for the base branch of the one being built, as found in config/base_branch;
  • one APT source for each suite listed in config/APT_overlays.d/*; note that only the name of such files matters, and their content is ignored.

In practice, config/APT_overlays.d/ should contain:

  • for a topic branch:
    • if needed, a file that is named like the branch's own overlay APT suite; e.g. for the bugfix/12345-whatever branch, it would be called config/APT_overlays.d/bugfix-12345-whatever
    • any file representing APT suites that came from merging its base branch into this topic branch, that is:
  • for a base branch (stable, testing, devel or feature/jessie): a file for each additional, overlay APT suite that came from topic branches that ship Debian packages and were merged into this base branch since last time it was used to prepare a release.

The code that implements this is auto/scripts/tails-custom-apt-sources. It has automated tests.

At release time, the release manager:

  1. merges into the release branch's APT suite all APT overlay suites found in config/APT_overlays.d/;
  2. empties config/APT_overlays.d/ in the release branch;
  3. merges the release branch into other base branches as needed, and ensures that all resulting config/APT_overlays.d/:s make sense.

Note that a branch like feature/jessie needs to be a base branch: we want to be able to work on topic branches forked off feature/jessie.

SSH access

Configure your SSH client to connect to the APT server:

Host incoming.deb.tails.boum.org
    Port 3003

HTTP access

The custom APT repository can be browsed at https://deb.tails.boum.org/.

Workflow

Creating a new branch

Push your branch to Git and wait a few minutes for the new APT suite to appear on https://deb.tails.boum.org/dists/. You can look up the name of that suite there.

Then, you probably want to drop a new file in config/APT_overlays.d/, named after the APT suite corresponding to your new branch. See details in the Build system section above.

Importing a new package

Building a package

Make sure the Distribution: field in your .changes file matches the suite you want the package to land in (e.g. pass --changes-option=-DDistribution=feature-torbrowser to pdebuild's --debbuildopts).

Make sure to have the .changes file include the original source archive (.orig.tar.{gz,bz2,xz}) if it is not already in our APT repository; this can be done by passing -sa to pdebuild's --debbuildopts.

Configuring an upload tool

Configuring dupload

Add this configuration snippet to your dupload configuration:

$config::cfg{'tails'} = {
        fqdn => "incoming.deb.tails.boum.org",
        method => "scp",
        login => "reprepro",
        incoming => "/srv/reprepro/incoming/",
        dinstall_runs => 1,
};

Configuring dput

Add this to .dput.cf:

[tails]
fqdn            = incoming.deb.tails.boum.org
method          = scp
login           = reprepro
incoming        = /srv/reprepro/incoming/
run_dinstall    = 0

Uploading and importing process

Carefully prepare and build your package. Usual precautions, (Lintian etc.) apply.

Carefully check the .changes file (especially the Distribution control field, and the included files list; the former can be fixed with the changestool(1) command, from reprepro).

Remove all buildinfo and dbgsym lines from the .changes file. XXX: still needed? We now run reprepro from Stretch, that supports both in theory.

If the .orig.tar.{gz,bz2,xz} tarball is present neither in the .changes file nor in our custom APT repository, add it using:

$ changestool $CHANGES_FILE includeallsources

Make sure that the .changes file lists the binary packages too: Tails has no package auto-builder that would build them from source for you.

Sign the .changes file with a key that is in the uploaders list:

$ debsign $CHANGES_FILE

Upload the files to the incoming queue:

$ dupload --to tails $CHANGES_FILE

or, using dput:

$ dput tails $CHANGES_FILE

reprepro will automatically notice the new files and import them into the suite specified in your .changes file.

Check the result:

$ ssh reprepro@incoming.deb.tails.boum.org reprepro list $SUITE $PACKAGENAME

Merging a main branch

When a Git main branch (devel, testing, stable, feature/jessie) is merged into another main branch, the corresponding operation must be done on the APT suites.

  1. Save the list of packages currently present in the APT suite we want to merge into, e.g. reprepro list devel.

  2. Make sure you are not going to overwrite newer packages with older ones (hint: use the tails-diff-suites script).

  3. Merge the APT suites:

    1. Set some environment variables:

        # the branch you want to merge
        SRC=stable
        # the branch you want to merge _into_
        DST=devel
      
    2. Merge in Git and APT:

        git checkout "$DST" && \
        git merge "$SRC" && \
        ssh reprepro@incoming.deb.tails.boum.org \
             tails-merge-suite "$SRC" "$DST"
      
    3. Restore the config/base_branch if needed:

        echo "${DST}" > config/base_branch && \
        git commit config/base_branch -m "Restore ${DST}'s base branch." || :
      
    4. Push:

        git push origin "${DST}:${DST}"
      
  4. Make sure not to re-add, into the branch we merge into, any package that was removed from it, but still is in the branch we merge from: e.g. when merging stable into devel, it may be that devel had some packages removed (e.g. due to previously merging a topic branch into it, whose purpose is to remove custom packages). To this end, compare the resulting list of (package, version) in the devel APT suite with the one saved before the merge, check Git merges history if needed, apply common sense, and remove from devel the packages that were removed from it a while ago, and were just erroneously re-added by the merge operation.

Resetting a suite to the state of another one

  1. . First, set some environment variables:

    # the suite to reset
    OLD=testing
    # the final state it should be in
    NEW=devel
    
  2. . Then, empty the OLD suite:

    ssh reprepro@incoming.deb.tails.boum.org \
          reprepro removematched $OLD '\*'
    
  3. . Finally, merge NEW into OLD

    ssh reprepro@incoming.deb.tails.boum.org \
          tails-merge-suite $NEW $OLD
    

Merging APT overlays

This operation merges all APT overlays listed in the given branch's config/APT_overlays.d/ into its own APT suite, empties config/APT_overlays.d/ accordingly, then commits and pushes to Git.

  1. Set some environment variables:

     # The branch that should have its overlays merged
     BRANCH=devel
    
  2. Merge the APT overlays in reprepro:

     git checkout "$BRANCH" && \
     for overlay in $(ls config/APT_overlays.d/) ; do
        if ! ssh reprepro@incoming.deb.tails.boum.org \
                tails-merge-suite "$overlay" "$BRANCH" ; then
           echo "Failed to merge '$overlay' into '$BRANCH': $?" >&2
           break
        fi
     done
    
  3. Empty config/APT_overlays.d/:

     git checkout "$BRANCH" && \
     git rm config/APT_overlays.d/* && \
     git commit config/APT_overlays.d/ \
        -m "Empty the list of APT overlays: they were merged"
    
  4. Push the Git branch:

     git push origin "${BRANCH}:${BRANCH}"
    

Tagging a new Tails release

Once the new release's Git tag is pushed, a cronjob creates a new APT suite on the custom APT repository's side within a few minutes. This new APT suite is called the same as the new release version. One may check it has appeared in ~reprepro/conf/distributions.

Then, the APT suite corresponding to the branch that was used to prepare the release must be copied to the new empty APT suite that just appeared:

$ ssh reprepro@incoming.deb.tails.boum.org \
     tails-merge-suite "$RELEASE_BRANCH" "$TAG"

After a new Tails release is out

If you just put out a final release:

  • merge stable or testing into devel
  • increment the version number in devel's debian/changelog to match the next major release, so that next builds from the devel branch do not use the APT suite meant for the last release:

      cd "${RELEASE_CHECKOUT}" && \
      git checkout devel && \
      dch --newversion "${NEXT_PLANNED_MAJOR_VERSION:?}" \
         "Dummy entry for next release." && \
      git commit debian/changelog \
         -m "Add dummy changelog entry for ${NEXT_PLANNED_MAJOR_VERSION:?}."
    
  • increment the version number in stable's debian/changelog to match the next point release, so that next builds from the stable branch do not use the APT suite meant for the last release:

      cd "${RELEASE_CHECKOUT}" && \
      git checkout stable && \
      dch --newversion "${NEXT_PLANNED_MINOR_VERSION:?}" \
         "Dummy entry for next release." && \
      git commit debian/changelog \
         -m "Add dummy changelog entry for ${NEXT_PLANNED_MINOR_VERSION:?}."
    

If you just released a RC (XXX: please automate these steps during the 3.2~rc1 release process, based on the above commands):

  • add a dummy changelog entry (for the upcoming, non-RC version) in the branch used for the release (stable or testing), so that the next builds from it do not use the APT suite meant for the RC
  • add a dummy changelog entry (for the release after the one you released a RC for) in the branch used for the release (stable or testing), so that the next builds from it do not use the APT suite meant for the RC (XXX: I don't understand what this is about; is it instead about adding an entry for that release on the devel branch? -- intrigeri)

If the release was a major one, then:

  1. Hard reset the stable APT suite to the state of the testing one.

  2. Empty config/APT_overlays.d in the stable branch:

     git checkout stable && \
     git rm config/APT_overlays.d/* && \
     git commit config/APT_overlays.d/ \
        -m "Empty the list of APT overlays: they were merged"
    

Giving access to a core developer

  1. Give SSH access to the reprepro user on the system that hosts reprepro (using the ssh_authorized_key Puppet resource, until #12346 is resolved).
  2. Import the developer's public GnuPG key into the reprepro user's GnuPG keyring -- should be doable using Puppet, some day
  3. Add the developer's OpenPGP key ID to $reprepro_uploaders in our tails::reprepro Puppet module. Deploy.

Contributing without privileged access

Non-core developers without access to the "private" APT infrastructure would add the .deb they want to their Git branch as we have been doing until now, push the result on GitLab or whatever... and at merge time, we would rewrite their history to remove the .deb, and import it into our APT repo.