Tracking ticket: #12615

The big picture

We've been thinking for a while of being based on quarterly snapshots of Debian testing. We've used the "porting Tails to Debian Stretch" cycle to evaluate how it would feel like. See the initial plan & analysis.

Calendar

A two-years Debian release cycle could look like from our perspective, if we were based on quarterly snapshots of Debian testing:

  1. a new Debian stable is released
  2. 6 months during which Tails is based on Debian stable that was just released, and we can get ready for the next stage
  3. 12 months during which Tails is based on quarterly snapshots of (non-frozen) Debian testing taken at the beginning of each major Tails release cycle
  4. 6 months during which Tails is based on a frozen Debian testing
  5. go back to (1)

We would be based on a moving target half of the time; the remaining half of the time we are based on something that doesn't change much.

Why?

  • We'll need to track 2 Debian versions at the same time (and continuously forward-port development that was based on the oldest one) during 6-7 months maximum, compared to 19 months (Jessie cycle) and 10 months (Stretch cycle). We would save lots of time there, that could be re-invested in aspects of this proposal that will require additional work.
  • no need to produce and maintain backports of Debian packages anymore
  • support new hardware better
  • upstream bugfixes trickle down faster and for free to Tails users ⇒ greater incentive for us to fix bugs upstream instead of implementing Tails-specific workarounds
  • we help stabilizing Debian testing
  • the Foundations Team prefers doing the "porting to next Debian" work continuously than as a huge disruptive project every two years
  • no huge change every 2 years confusing users and creating huge burts in the Help Desk and Foundations Team post-release workload
  • new features / modern software

This idea is about doing a different kind of work, and doing it in different places than in the past. It probably won't lower the total amount of effort we have to do, but it likely will make such efforts generate less frustration, more happiness, and a warm feeling of being part of a broader community.

Fears, concerns & problems we need to address or mitigate

  • Security

    • How to keep track of security issues that affect us, and whether their fix has been uploaded and has migrated to testing yet? See e.g. how security support for Debian testing used to be (briefly) done, and in particular the Vulnerable source packages in the testing suite and the Candidates for DTSAs. At first glance it seems totally doable for Foundations Team members to run (e.g. locally) a lightweight fork of Debian's security tracker:
      • Problem to solve: list security issues that affect Tails (current stable and testing branches) but are fixed in current Debian testing and/or sid
      • Point it to our relevant APT snapshots (and perhaps current Debian testing or sid instead of Tails devel, to avoid having to resolve the "latest" unfrozen snapshot)
      • Add our own annotations about issues we decide to ignore/postpone, e.g. using the new <ignored> and <postponed> tags
      • Filter packages based on some relevant build manifest so we don't ever see packages we don't ship in Tails.
      • Automatically merge from Debian's repo and report to us in case of merge conflicts.
    • more freeze exceptions in order to address security issues
    • need to keep our stable branch in an always releasable state ⇒ the tracking & cherry-picking of security issues must be done continuously, and not only right before a planned freeze or release
    • when the updated package from Debian testing/sid is not installable on current Tails stable/testing, we'll have to backport security fixes
  • Transitions

    • How to deal with ongoing transitions that block migration of security updates from sid to testing?
    • How to select the right snapshot we shall take, e.g. during transitions?
    • Sometimes we'll need to rebuild on testing some packages we want to cherry-pick from sid.
  • Consequences for users

    • too many software regressions and not well tested new features
    • confused users due to constant incremental changes
    • bigger upgrades on average: according to research done on #14622, we should assume ~600MB large IUKs; there will be consequences in terms of:
      • RAM requirements: XXX
      • download time: XXX
      • download reliability ⇒ add a retry/continue mechanism to Tails Upgrader?
    • our users debug Debian testing on top of debugging Tails
  • Drawbacks for contributors & the Tails community

    • more frequent regressions in ISO reproducibility
    • harder to extract info from help desk
    • harder for help desk to deal with user requests: having to re-learn constantly how things work / what's broken
    • spend too much mental space dealing with always changing software
    • lack of Debian skills internally at the moment ⇒ we'll need to invest time learning
    • need to better study the delta between N & N+1 (for help desk, release notes, security audits, etc.)
  • Communication

    • needs new version numbering scheme
    • feeling of stalled/slower project for users because no big all-at-a-time changes
    • how do we deal with our (implicit) hardware support promise? i.e. "The hardware you purchase for Tails N.0 will work with N.x"
    • harder to get press write stuff as we'll lack releases with tons of changes like 2.0 or 3.0
  • Additional Software Packages: will they be pulled from current Debian testing or from our snapshot? Can we tell APT to install by default from Debian testing, unless it's impossible or problematic (e.g. would remove tons of packages) and then fall back to installing from our snapshot?

The plan

  • From now to the end of 2017-11: the Foundations Team tries to port the code & test suite during sprints. If the work doesn't fit into these sprints then we'll need to draw conclusions.
  • At the end of 2017-11:
    1. Decide whether we want to release Tails based on Debian testing in 2018-01, 2018-04, or give up and rather release it mid-2019. The following assumes "in 2018-01" and will need to be adjusted if we decide something else.
    2. The Foundations Team tells technical writers what needs to be updated (#14579)
  • November-January: technical writers update the documentation
  • January 16: first Tails release based on Debian testing

Zooming in on specific problems

How can the Foundations Team tell Technical Writers what part of our doc needs to be updated?

The first time we'll have to deal with this problem is #14579, but it will come back every quarter.

Current implementation:

We want to automate this as much as we can, but let's be clear: automation will not fully solve this problem unless we switch to executable documentation (there is research and tools for that but we're not there yet). So some amount of manual documentation testing will be needed. Let's start with strategies that will limit the amount and frequency of the needed manual documentation testing, and further below we'll discuss manual testing.

Changes in our code, test suite, and list of included packages

Whenever we have to adjust our code to keep working on Debian testing, it's plausible that the corresponding documentation needs an update as well. Developers could either proactively tell technical writers about it along the way, or look at git diff when requested to sum up what part of Tails is impacted by the changes.

Similarly, whenever we have to update the test suite, it's plausible that the corresponding documentation needs an update as well. Test suite maintainers can then add the updated scenario to a list of things the technical writers should check (again: either incrementally or upon request).

And finally, developers or technical writers could look at the diff between the old and new lists of included packages, to identify what part of Tails is impacted by the changes.

All this is fine but it requires someone to keep a mental index of what kind of change impacts what part of our documentation. Not only this is boring work, but nobody is in a good position to do it, as it requires a good knowledge of our code base and of our documentation. So I think we should build and maintain this index incrementally, in Git instead of in our minds, and use it automatically. Read on :)

Metadata that encodes the relationship between {source code, test suite, packages} and documentation pages

Whenever we notice that changes in a given file in our source code, automated test suite scenario, or package, have an effect on a given documentation page, we could encode this information somewhere, in a machine-readable format. Then, we could write a program whose input would be:

  • this metadata
  • 2 Git commitish
  • 2 package lists

This program would output the list of documentation pages that are affected by the changes.

This set of metadata shall be built and garbage-collected incrementally.

Remaining questions:

  • Where to store this metadata?

    At least the information about included packages lives nowhere in our Git tree, so encoding it close to the source is impossible (besides, I'd rather not have to extract metadata from comments in source code written in N different programming languages). So we have main options:

    1. Store it in the affected documentation pages themselves, (HTML comments? ikiwiki directives? any other idea?) Advantages:

      • No work required when renaming pages.
      • Easier to maintain by tech writers as they can see this info while working on the doc, which increases the chances they keep it up-to-date.
      • Lightweight: no need to explicitly write the doc page name.
    2. Store it externally, in a dedicated file. Advantages:

      • Probably nicer format to read and edit, both for humans and computers, than whatever we would embed in doc pages.
      • Does not affect translators: anything we would put in doc pages will be extracted to PO files; so storing this metadata there would waste translators' time (either because they uselessly translate it, or because they paste the English string to reach 100% translation status and keep their dashboard of pages needing attention usable).
      • Works for encoding which sources files impact screenshots.

    intrigeri & anonym discussed that and were torn. The advantages of the first option are very compelling, but the drawback it has wrt. translators seems to be a deal-breaker. So the second option appears to be the best one.

    XXX: sajolida?

  • Defining sets of sources?

    Tails user-facing features are built from a not-that-small number of source files. It could be useful to express the fact that changes to any of these source files (e.g. the ones that define the "Tor Browser" feature) may affect a given doc page, without having to list them all for every affected page.

  • Defining sets of doc pages?

    Similarly, it might be that some doc sections share most of their dependencies. It could be useful to define named groups of pages.

  • Who shall we optimize the data format for?

    In practice, developers are the ones who know where a given user-visible change comes from, so whenever a needed doc update was missed, most likely the tech writers will ask the Foundations Team either to add the missing metadata, or what exactly caused that user-visible change so they can add it themselves.

    I (intrigeri) am really not sure which of these two situations will be the most common one, so I'm wary of optimizing too much for one of these two groups at this point.

  • Mapping source → doc, or doc → source?

    Update: a format where we define a list of dependencies between X doc pages and Y source files, without explicitly mapping one to the other, would avoid having to choose too early, which would be nice. I'll simply do that.

    Internally, the code that uses this metadata will care about "source → doc", but there's no reason why the data format itself should be optimized for this program. Let's instead optimize it for the humans who will maintain the metadata :)

    "doc → source" expresses a "depends on" / "is affected by" relationship, which is well suited when fixing the metadata after a postmortem analysis of "why was that doc page not updated before the release?". OTOH "source → doc" expresses an "impacts" relationship, which is well suited to proactive adjustments of the metadata, e.g. when a developer realizes their proposed branch will impact the doc. I (intrigeri) am pretty sure that the first situation will be the most common one for a while until we get used to this metadata and the second one slowly takes over.

Example (YAML):

# doc → source oriented
- page: doc/first_steps/accessibility
  files:
    - config/chroot_local-includes/etc/dconf/db/local.d/00_Tails_defaults
  packages:
    - caribou
    - dasher
    - orca
- page: doc/first_steps/startup_options
  package: tails-greeter
  files:
    - config/binary_local-hooks/*grub*
    - config/binary_local-hooks/*syslinux*
    - config/chroot_local-includes/etc/dconf/db/local.d/00_Tails_defaults
- page: doc/first_steps/startup_options/administration_password
  package: tails-greeter
- page: doc/first_steps/startup_options/bridge_mode
  packages:
    - obfs4proxy
    - tor
  file: config/chroot_local-includes/usr/share/tails/tbb-sha256sums.txt
  tests: tor_bridges.feature

# "source → doc" oriented
- package: gnome-disk-utility
  pages:
    - doc/first_steps/persistence/change_passphrase
    - doc/first_steps/persistence/check_file_system
- package: tails-persistence-setup
  tests: persistence.feature
  pages:
    - doc/first_steps/persistence/configure
    - doc/first_steps/persistence/delete
- package: openpgp-applet
  tests: encryption.feature
- package: tails-greeter
  files:
    - features/images/TailsGreeter*
  pages:
    - doc/first_steps/persistence/use
    - doc/first_steps/startup_options
    - doc/first_steps/startup_options/*

Automated testing

As said above, changes in our automated test suite can already help identify the bits of documentation that need an update. To boost this capability we could:

  • Add steps to our existing tests to validate the screenshots we have in our documentation. This seems doable but not trivial:
    • Screenshots embed context, e.g. window placement and elements that change all the time; so we would have to extract the relevant part of each screenshot (for example based on coordinates encoded somewhere).
    • Credentials used when taking screenshots (e.g. Pidgin) must be the ones used by the isotesters.
    • Screenshots in our documentation are resized to 66% so we need to resize them back. This is quite easy, but the information lost in the double-resizing process may prevent Sikuli from finding it on the screen.
  • Add automated tests that specifically check things we document.
  • Try to make our Gherkin scenarios closer to end-user documentation.

Manual testing of the documentation

Which part of the documentation shall be tested, when, and by whom?

The sources of information listed above should help focus this effort on the relevant parts of our documentation.

In many cases, a quick look will be enough, and following the entire doc page to the letter won't be needed. But likely stricter testing will be needed from time to time.

Random ideas:

  • Aim for automation: feed the aforementioned metadata store with information gathered during manual testing, so that we need less and less manual testing as time goes.

  • Prioritize: focus on core pages.

  • Spread the load between different release, i.e. test some pages for some release and some others for another release.

Dismissed options

Manual testing of Tails releases

In theory our manual test suite could also help: some of the manual tests could be performed by following our end-user documentation to the letter. But I've not been able to find any single candidate test case for this, so let's forget it.

Users and Help Desk

Users and Help Desk are also in a good position to notice outdated bits and report them back, but they can only do so after the fact, while ideally we'd rather update the documentation before the corresponding changes are released to our users. So if we receive such notifications, e.g. at RC time, fine, but let's not count on it too much.