Currently automatic upgrades have the limitation that only some N number of them can be performed in a row before the system partition runs out of space and automatic upgrades cannot be done further, but a full, manual upgrade is needed.

Automatic upgrades should be possible forever. This would:

  • Improve UX as currently full upgrades are one of the most complicated and long scenario, requiring at least 2 USB sticks and 2 restarts.

  • Improve security by eliminating the need for recurring verifications of ISO images and reducing cross-contamination by limiting the number of times you might want to clone or rely on another operating system for downloading or upgrading.

Different approaches to endless upgrades

While examining these approaches, let's forget about IUKs for a while, and only consider full automatic upgrades (#7499):

Extract upgrade from ISO

Keeping our current partition layout and filesystems, a full upgrade of a running system can be done by extracting the required parts from the new version's .iso. Here we'd need at least 3 times the size of an installation available:

  1. one for the old, running installation
  2. one for the new ISO
  3. one for the extracted contents of the new ISO

We would unlink the old squashfs, kernel, initrd etc. then copying in their place new version's ones, so the new version will run next time we reboot.

Disk images instead of ISOs

Note that if we switch to using disk images as the primary way to install Tails (#8550, comment 9) we can similarly extract the needed parts just like above: replace "(ISO|.iso)" with "disk image" and it still works.

Streaming decompression

An improved version of the above idea would be to serve "upgrade packs" using some compression method friendly to streaming decompression, so the need to download a the full archive (ISO or disk image above) is avoided, which reduces the amount of temporary disk space needed from 3 to 2 times the size of an installation when performing the upgrade.

ISOBoot

Grub and syslinux supports booting directly from an ISO so we could switch to a strategy where Tails USB/SD installation means that we dump the ISO image into the system partition, and configure the bootloader to boot from it. If we make the system partition large enough to store two ISOs (= 2 times the size of an installation, when comparing o the above methods) we can keep the older, running version in place while downloading the new one, and then safely switch to the new one at next boot. Since the old version is kept, there's a fallback in case the new version doesn't boot. Older versions can be cleaned up at the next upgrade.

Depending on how Grub/syslinux does the ISOBoot, this may make all running Tails instances more consistent, i.e. we now do not have the three cases of what type of medium Tails has booted from: DVD; USB drive; or SD card -- in all cases we have a ISO9660 filesystem. OTOH, the magic employed for the ISOBoot may hide too much about underlying real device, which we still probably will need information about.

Some more details (some perhaps wrong) can be read in comment 16 on #7496.

Apply upgrade in initramfs

By storing the ISO (or disk image) that the upgrade is gonna be extracted from on a scratch partition, and then apply the upgrade in initramfs on next boot, we only need 2 times the size of an installation in total, and can safely download the ISO to the scratch partition while keeping the system partition read-only.

Ideally, let's also save the files used to authenticate the upgrade on the scratch partition, and make check the authenticity during initramfs, before starting the upgrade.

Also, with the scratch partition we will never have to mount it read-write outside of initramfs, it can be made read-only in a slightly stronger sense on the block dev level. This is not really a security feature, but more a foolproof things, but why not?

How others do i.e. non-NIH solutions

This survey will be updated in a while with #15277.

Not suitable

(but could be good sources of inspiration!)

  • EndlessOS' OSTree based upgrader
    • supports separate poll/fetch/apply operations (so presumably we could make the relevant partition read-write only after fetching, as long a we can fetch to a tmpfs)
    • daemon + D-Bus API
    • systemd integration
    • supports Flatpak
    • likely too many EndlessOS-specific assumptions… and written in C so difficult to adapt to our needs
  • mender: downloads full update
  • swabic's swupdate: downloads full update; binary delta updates are on the roadmap

Auxiliary ideas

Read-write system partition while the network is enabled

Currently we keep the system partition read-only as long as the network is enabled. E.g. we download IUKs into the tmpfs, and when it's finished we disable the network and only then remount the system partition read-write to apply the upgrade.

This is problematic for full upgrades, since we cannot assume that users' systems has over 1 GiB of free RAM available but need to download it to disk, but cannot use the system partition. In the "extract upgrade from ISO/Disk image" case we can simply introduce a scratch partition where the temporary download is stored without loss of disk usage efficiency (we still need 3 times an installation). But for the ISOBoot and streaming decompression cases we'd then go from 2 times to 3 times the size of an installation.

It might be worth having a discussion about whether preventing the network to be used while the system partition is mounted read-write actually brings us any security. After all, if the system is compromised while downloading the upgrade, it will still be compromised when the download is finished and the system partition is remounted read-write, and it can infect that part then.

Note that with A/B system updates this discussion is moot: the bootloader configuration update is the only part of the process that requires read-write access.

What about IUKs?

If we want to support IUKs at the same time as full upgrades, we need to allocate even more space to the system partition. Also, we need to carefully consider the UX part of this.

For instance, we could always give users the choice to do a full or incremental upgrade unless installing the incremental upgrade will result in too little disk space remaining for a full upgrade to be done next time -- in this case we only allow full upgrades, to prevent users from locking themselves in a situation where they cannot upgrade further. When both options are available, we could give users the suggestion that they should do a full upgrade if they currently have a lot of time and/or a fast connection.

Self-contained verification

An interesting option for Tails installation verification (#7496) is to contain all the bits needed to verify an installation's integrity. For instance, in the ISOBoot case we'd include the .sig for the ISO on as well as the .iso, so an external verifier simply can verify the .sig instead of needing an external known-good .iso to compare with.

The ISOBoot case is the simplest, probably, as the other methods would require signed manifests of files with the expected hashes (or similar), which of course is not such a big deal to provide.

The ISOBoot case would still require a manifest of files we extract from the ISO (bootloader code and configuration), but indeed that manifest could be in the ISO itself, and thus its authenticity and integrity would be signed/verified at the same time as the ISO.

Temporary stop gap: stack one single SquashFS diff when upgrading

(This idea was originally conceived in the comments of #11131.)

Without any major redesign of our current incremental upgrade system (i.e. comparatively cheap) we can allow users upgrade only through IUKs for much longer, probably throughout a whole Tails series, e.g. all of Tails 2.x, which we'll use for our examples below since we moved to 3.x already so we have all data we need to test this idea.

Basic idea

Inside Tails

  • Tails Upgrader fetches the UDF for the version of Tails that was originally installed (instead of the version of the running Tails),
  • We remove any previous IUK (and reference to it in Tails.module) when a new IUK is installed.

Release management

When we release version M.N, then, for all previous versions M.X since M.0, we generate a M.X_to_M.N IUK.

(Actually, it is not necessary to choose the first release of a series (like M.0) as the "synchronization point". We could add another one half way through the series, or make every tenth release a synchronization point, or we could do it when when the IUK simply grows too big. We have lots of options to balance this according to our needs, but the first release of a series seems like a technically reasonable choice according to the data we have for the 2.x series (see below), but also nice from the user's PoV if the only time a manual upgrade is needed is when upgrading between series.)

Example

  1. I install Tails 2.2 on a USB stick.
  2. Later when Tails 2.3 is released I upgrade to it via the 2.2_to_2.3 IUK (i.e. identical to our original scheme).
  3. When Tails 2.4 is released I upgrade to it via the 2.2_to_2.4 IUK (whereas in our original scheme we would stack 2.3_to_2.4 on top of 2.2_to_2.3) and remove the 2.2_to_2.3 IUK (whereas in our original scheme we never remove IUKs).

Notice how step 3 can be repeated for each new release, so that we in the end reach the final release of the series, Tails 2.12. And notice how we in step 1 could have installed any 2.x release.

Problems due to IUK size

We'll be dealing with larger IUKs (because generally more and more changes accumulate between different Tails versions) and only have one present at a time, with the exception being while we are installing a new IUK (then two will be present). Let's look at the IUK sizes towards the end of the 2.x series as if the last manual install was for Tails 2.0, which is the excepted worst case:

569M Tails_i386_2.0_to_2.9.iuk
678M Tails_i386_2.0_to_2.10.iuk
682M Tails_i386_2.0_to_2.11.iuk
634M Tails_i386_2.0_to_2.12.iuk

The IUK size is involved in at least four concerns:

  • Disk space needs on mirrors: for 2.x there were 14 non-rc/beta/alpha Tails releases, so there would be 13 IUKs for the last release (2.12). So using the worst-case as the average case we get the upper bound of the space needs on each mirror: 14*682 MB ~= 10 GB.
  • Disk space needs on the Tails system partition: we need to be able to store two large IUKs. Apparently the worst case for 2.x is the 2.10 to 2.11 upgrade if 2.0 was the version originally installed:
    • The 2.0 installation itself uses 1082 MB, so upgraded to 2.10 the total usage is 1082 MB + 678 MB = 1760 MB.
    • Due to the Upgrader requiring 3x the size of the IUK on the Tails system partition, the actual requirement is 1760 MB + 3*682 MB = 3806 MB of free disk space. That wouldn't have worked during the 2.x times, when the system partition was 2.5 GB, and it barely would work with our current 4.0 GB partition. OTOH, during the 2.x series we shipped two kernels, while we don't in 3.x, so we have that in our favor.
    • When both IUKs are present at the same time (i.e. before rebooting after having applied the upgrade) the disk usage is 1760 MB + 682 MB = 2442 MB.
    • After the old IUK is purged the disk usage is down to 2442 MB - 678 MB = 1764 MB.
  • Memory needs: while performing the upgrade, 2x the size of the IUK of memory is needed, so 2*682 MB = 1364 MB. On a system with 2 GB of memory, a fresh boot of Tails 3.5 (without networking to limit the amount of processes autostarting) has 1255 MB of free memory (according to check_free_memory()'s calculation in config/chroot_local-includes/usr/local/bin/tails-upgrade-frontend-wrapper) so the upgrade would fail. This is a regression for users with 2 GB memory: for all of 2.x and 3.x, all IUKs have been under 400 MB, which would work fine with 2 GB of memory. If that's a blocker, then we have to solve it by changing the format of the IUKs (#6876); we should coordinate this with other changes that will break automated upgrades from Tails N to N+1, such as Tails 4.0 and the migration to overlayfs (#9373) .
  • Bandwidth needs of the RM. Uploading 10 GB of IUKs can be a pain for some of us, but that can easily be solved by making it possible to generate IUKs on lizard (and then compare them with the ones you generated locally. Thanks reproducibility!).

Support upgrading very old Tails

We probably do not want to support upgrading very old (e.g. from six months ago) Tails installations because our signing key might have been updated in the meantime, and/or some critical update to Tails Upgrader (e.g. UDF version). So unless we want to spend a lot of resources on solving these problems, which roughly translates to "you can take long breaks from Tails", let's guarantee our current "you can at most skip one planned release" promise, and optimistically support older versions whose key and Tails Upgrader are still ok.

Download requirements

tl;dr: for 2.x the new scheme would have required downloading 50% more data than the original scheme.

2.x stats for original IUK scheme

Assumptions:

  • a 4 GB Tails system partition.

It would look like this:

1082M   tails-i386-2.0.iso
153M    Tails_i386_2.0_to_2.0.1.iuk
315M    Tails_i386_2.0.1_to_2.2.iuk
104M    Tails_i386_2.2_to_2.2.1.iuk
258M    Tails_i386_2.2.1_to_2.3.iuk
207M    Tails_i386_2.3_to_2.4.iuk
324M    Tails_i386_2.4_to_2.5.iuk
363M    Tails_i386_2.5_to_2.6.iuk
328M    Tails_i386_2.6_to_2.7.iuk
197M    Tails_i386_2.7_to_2.7.1.iuk
176M    Tails_i386_2.7.1_to_2.9.1.iuk
1153M   tails-i386-2.10.iso
300M    Tails_i386_2.10_to_2.11.iuk
296M    Tails_i386_2.11_to_2.12.iuk

Total: 5256 MB

2.x stats for 1BigIUK scheme

Assumptions:

  • a 4 GB Tails system partition.
  • original installing 2.0 creates the largest IUKs.

It would look like this:

1082M   tails-i386-2.0.iso
153M    Tails_i386_2.0_to_2.0.1.iuk
388M    Tails_i386_2.0_to_2.2.iuk
395M    Tails_i386_2.0_to_2.2.1.iuk
434M    Tails_i386_2.0_to_2.3.iuk
462M    Tails_i386_2.0_to_2.4.iuk
515M    Tails_i386_2.0_to_2.5.iuk
590M    Tails_i386_2.0_to_2.6.iuk
597M    Tails_i386_2.0_to_2.7.iuk
621M    Tails_i386_2.0_to_2.7.1.iuk
622M    Tails_i386_2.0_to_2.9.1.iuk
678M    Tails_i386_2.0_to_2.10.iuk
682M    Tails_i386_2.0_to_2.11.iuk
634M    Tails_i386_2.0_to_2.12.iuk

Total: 7853 MB

FWIW, IUK sizes for 3.x

200M    Tails_amd64_3.0_to_3.1.iuk
278M    Tails_amd64_3.1_to_3.2.iuk
260M    Tails_amd64_3.2_to_3.3.iuk
177M    Tails_amd64_3.3_to_3.4.iuk
213M    Tails_amd64_3.4_to_3.5.iuk