1. Goals
  2. Principles
    1. Infrastructure as code
    2. Free Software
    3. Relationships with upstream
  3. Duties
    1. In general
    2. During sysadmin shifts
    3. Outside of sysadmin shifts
  4. Tools
  5. Communication
  6. Services
    1. APT repositories
    2. Bitcoind
    3. BitTorrent
    4. DNS
    5. GitLab
    6. Gitolite
    7. git-annex
    8. Icinga2
    9. Internal XMPP service
    10. Jenkins
    11. Mail
    12. Meeting reminder
    13. Mumble
    14. rsync
    15. Schleuder
    16. Tor bridge
    17. VPN
    18. Web server
    19. Weblate
    20. WhisperBack relay
  7. Other pages

Goals

The Tails system administrators set up and maintain the infrastructure that supports the development and operations of Tails. We aim at making the life of Tails contributors easier, and to improve the quality of the Tails releases.

Principles

Infrastructure as code

We want to treat system administration like a (free) software development project:

  • We want to enable people to participate without needing an account on the Tails servers.
  • We want to review the changes that are applied to our systems.
  • We want to be able to easily reproduce our systems via automatic deployment.
  • We want to share knowledge with other people.

This is why we try to publish as much as possible of our systems configuration, and to manage our whole infrastructure with configuration management tools. That is, without needing to log into hosts.

Free Software

We use Free Software, as defined by the Debian Free Software Guidelines.
The firmware our systems might need are the only exception to this rule.

Relationships with upstream

The principles used by the broader Tails project also apply for system administration.

Duties

In general

As said above, "set up and maintain the infrastructure". This implies for example:

  • dealing with hardware purchase, upgrades and failures;
  • upgrading our systems to a new version of Debian.

During sysadmin shifts

  • create Git repositories when requested
  • update access control lists to resources we manage, as requested by the corresponding teams
  • keep systems up-to-date, reboot them as needed
  • keep Jenkins plugins up-to-date, by upgrading any plugin that satisfies at least one of these conditions:
    • brings security fixes
    • fixes bugs we're affected by
    • brings new feature we are interested in, without breaking the ones we rely on
    • is needed to upgrade another plugin that we want to upgrade
    • is required by a system upgrade (e.g. of the Jenkins packages)
  • report bugs identified in Jenkins plugins after they have been upgraded (both on the upstream bug tracker and on our own one)
  • act as the de facto interface between Tails and the people hosting our services (boum.org, immerda.ch) for non-trivial requests
  • when a sysadmin shift includes the beginning of a yearly quarter, ensure that sysadmin shifts are filled and agreed on for the next two quarters
  • quarterly: self-evaluate our work and report to the -summit@ mailing list
  • When the deadline for taking over a given maintenance task (see below) has passed, the sysadmin on duty must make it clear s·he's handling the problem before starting to work on it, in order to avoid work duplication.

Outside of sysadmin shifts

  • Read email at least twice a week to check if the sysadmin currently on duty needs help.

  • Once 48 hours have passed after a problem was identified, the sysadmins not currently on duty can/should take over maintenance tasks if the on duty sysadmin is MIA; for critical problems this delay shall be reduced.

Tools

The main tools used to manage the Tails infrastructure are:

  • Debian GNU/Linux; in the vast majority of cases, we run the current stable release
  • Puppet, a configuration management system
  • Git to host and deploy configuration, including our Puppet code

Sysadmins can login to all hosts and have write access to the Puppet masters' Git repositories.

Communication

In order to get in touch with Tails sysadmins, you can:

The following lists of issues are also of interest to sysadmins:

Services

Below, importance level is evaluated based on:

  • users' needs: e.g. if the APT repository is down, then the Additional Software feature is broken;
  • developers' needs: e.g. if the ISO build fails, then developers cannot work;
  • the release process' needs: we want to be able to do an emergency release at any time when critical security issues are published.

APT repositories

Custom APT repository

  • purpose: host Tails-specific Debian packages
  • documentation
  • access: anyone can read, Tails core developers can write
  • tools: reprepro
  • configuration:
  • importance: critical (needed by users, and to build & release a Tails ISO)

Time-based snapshots of APT repositories

  • purpose: host full snapshots of the upstream APT repositories we need, which provides the freezable APT repositories feature needed by the Tails development and QA processes
  • documentation
  • access: anyone can read, release managers have write access
  • tools: reprepro
  • configuration:
  • importance: critical (needed to build a Tails ISO)

Tagged snapshots of APT repositories

  • purpose: host partial snapshots of the upstream APT repositories we need, for historical purposes and compliance with some licenses
  • documentation
  • access: anyone can read, release managers can create and publish new snapshots
  • tools: reprepro
  • configuration:
  • importance: critical (needed by users and to release Tails)

Bitcoind

  • purpose: handle the Tails Bitcoin wallet
  • access: Tails core developers only
  • tools: bitcoind
  • configuration: bitcoind class
  • Vcs-Git: bitcoin and libunivalue
  • importance: medium
  • To save disk space: as the bitcoin@bitcoin.lizard user, run bitcoin-cli getblockcount to get the ID of the last block, then run bitcoin-cli pruneblockchain XYZ, with XYZ being a Unix timestamp that's at least 5 months in the past.

BitTorrent

  • purpose: seed the new ISO image when preparing a release
  • documentation
  • access: anyone can read, Tails core developers can write
  • tools: transmission-daemon
  • configuration: done by hand (#6926)
  • importance: low

DNS

  • purpose: authoritative nameserver for the tails.boum.org and amnesia.boum.org zones
  • access:
    • anyone can query this nameserver
    • members of the mirrors team control some of the content of the dl.amnesia.boum.org sub-zone
    • Tails sysadmins can edit the zones with pdnsutil edit-zone
  • tools: pdns with its MySQL backend
  • configuration:
  • importance: critical (most of our other services are not available if this one is not working)

GitLab

Gitolite

  • purpose:
    • host Git repositories used by the puppetmaster and other services
    • host mirrors of various Git repositories needed on lizard, and whose canonical copy lives on GitLab
  • access: Tails core developers only
  • tools: gitolite3
  • configuration: tails::gitolite class
  • importance: high (needed to release Tails)

git-annex

Icinga2

  • purpose: Monitor Tails online services and systems.
  • access: only Tails core developers can read-only the Icingaweb2 interface, sysadmins are RW and receive notifications by email.
  • setup: We have one Icinga2 instance installed on a dedicated system used as the master of all our Icinga2 zones. We use a VM on the other bare-metal host as the Icinga2 satellite of our master. Icinga2 agents are installed on every other VM and the host itself. They report back to the satellite, which transmits to the master. We spread the Icinga2 configuration with Puppet. This way, we achieve a certain isolation where the master or the satellite have no right to configure agents or run arbitrary commands on them.
  • tools: Icinga2, icingaweb2
  • configuration:
  • documentation:
  • importance: critical (needed to ensure that other, critical services are working)

Internal XMPP service

  • purpose: an internal XMPP service that can be used by Tails developers and some contributors.
  • access: at the moment everyone that is on the tails-summit mailinglist has and/or can request an account.
  • tools: prosody
  • configuration:
  • importance: low

Jenkins

Mail

Meeting reminder

Mumble

rsync

  • purpose: provide content to the public rsync server, from which all HTTP mirrors in turn pull
  • access: read-only for those who need it, read-write for Tails core developers
  • tools: rsync
  • configuration:
    • tails::rsync
    • users and credentials are managed with the tails_secrets_rsync Puppet module
  • importance: critical (needed to release Tails)

Schleuder

  • purpose: host some of our Schleuder mailing lists
  • access: anyone can send email to these lists
  • tools: schleuder
  • configuration:
  • importance: high (at least because WhisperBack bug reports go through this service)

Tor bridge

VPN

  • purpose: flow through VPN traffic the connections between our different remote systems. Mainly used by the monitoring service.
  • access: private network.
  • tools: tinc
  • configuration:
  • importance: transitively critical (as a dependency of our monitoring system)

Web server

  • purpose: serve web content for any other service that need it
  • access: depending on the service
  • tools: nginx
  • configuration:
  • importance: transitively critical (as a dependency of Jenkins)

Weblate

WhisperBack relay

  • purpose: forward bug reports sent with WhisperBack to tails-bugs@boum.org
  • access: public; WhisperBack (and hence, any bug reporter) uses it
  • tools: Postfix
  • configuration:
  • importance: high

Other pages