Goals
The Tails system administrators set up and maintain the infrastructure that supports the development and operations of Tails. We aim at making the life of Tails contributors easier, and to improve the quality of the Tails releases.
Principles
Infrastructure as code
We want to treat system administration like a (free) software development project:
- We want to enable people to participate without needing an account on the Tails servers.
- We want to review the changes that are applied to our systems.
- We want to be able to easily reproduce our systems via automatic deployment.
- We want to share knowledge with other people.
This is why we try to publish as much as possible of our systems configuration, and to manage our whole infrastructure with configuration management tools. That is, without needing to log into hosts.
Free Software
We use Free Software, as defined by the Debian Free Software
Guidelines.
The firmware our systems might need are the only exception to
this rule.
Relationships with upstream
The principles used by the broader Tails project also apply for system administration.
Duties
In general
As said above, "set up and maintain the infrastructure". This implies for example:
- dealing with hardware purchase, upgrades and failures;
- upgrading our systems to a new version of Debian.
During sysadmin shifts
- create Git repositories when requested
- update access control lists to resources we manage, as requested by the corresponding teams
- keep systems up-to-date, reboot them as needed
- keep Jenkins plugins up-to-date, by upgrading any plugin that satisfies
at least one of these conditions:
- brings security fixes
- fixes bugs we're affected by
- brings new feature we are interested in, without breaking the ones we rely on
- is needed to upgrade another plugin that we want to upgrade
- is required by a system upgrade (e.g. of the Jenkins packages)
- report bugs identified in Jenkins plugins after they have been upgraded (both on the upstream bug tracker and on our own one)
- act as the de facto interface between Tails and the people hosting our services (boum.org, immerda.ch) for non-trivial requests
- when a sysadmin shift includes the beginning of a yearly quarter, ensure that sysadmin shifts are filled and agreed on for the next two quarters
- quarterly: self-evaluate our work and report to the -summit@ mailing list
- When the deadline for taking over a given maintenance task (see below) has passed, the sysadmin on duty must make it clear s·he's handling the problem before starting to work on it, in order to avoid work duplication.
Outside of sysadmin shifts
Read email at least twice a week to check if the sysadmin currently on duty needs help.
Once 48 hours have passed after a problem was identified, the sysadmins not currently on duty can/should take over maintenance tasks if the on duty sysadmin is MIA; for critical problems this delay shall be reduced.
Tools
The main tools used to manage the Tails infrastructure are:
- Debian GNU/Linux; in the vast majority of cases, we run the current stable release
- Puppet,
a configuration management system
- our Puppet code
- Git to host and deploy configuration, including our Puppet code
Sysadmins can login to all hosts and have write access to the Puppet masters' Git repositories.
Communication
In order to get in touch with Tails sysadmins, you can:
- Create an issue in the tails/sysadmin project
- Ping all sysadmins anywhere in our GitLab by mentioning the
@sysadmin-team
group - See if one of us is on shift in one of our chat rooms
- Send an e-mail to the sysadmin's mailing list
The following lists of issues are also of interest to sysadmins:
- issues that should be taken care of as part of sysadmin shifts or are on the sysadmin team's roadmap
- tasks that belong to the Infrastructure category
Services
Below, importance level is evaluated based on:
- users' needs: e.g. if the APT repository is down, then the Additional Software feature is broken;
- developers' needs: e.g. if the ISO build fails, then developers cannot work;
- the release process' needs: we want to be able to do an emergency release at any time when critical security issues are published.
APT repositories
Custom APT repository
- purpose: host Tails-specific Debian packages
- documentation
- access: anyone can read, Tails core developers can write
- tools: reprepro
- configuration:
tails::reprepro::custom
class- signing keys are managed with the
tails_secrets_apt
Puppet module
- importance: critical (needed by users, and to build & release a Tails ISO)
Time-based snapshots of APT repositories
- purpose: host full snapshots of the upstream APT repositories we need, which provides the freezable APT repositories feature needed by the Tails development and QA processes
- documentation
- access: anyone can read, release managers have write access
- tools: reprepro
- configuration:
tails::reprepro::snapshots::time_based
class- signing keys are managed with the
tails_secrets_apt
Puppet module
- importance: critical (needed to build a Tails ISO)
Tagged snapshots of APT repositories
- purpose: host partial snapshots of the upstream APT repositories we need, for historical purposes and compliance with some licenses
- documentation
- access: anyone can read, release managers can create and publish new snapshots
- tools: reprepro
- configuration:
tails::reprepro::snapshots::tagged
class- signing keys are managed with the
tails_secrets_apt
Puppet module
- importance: critical (needed by users and to release Tails)
Bitcoind
- purpose: handle the Tails Bitcoin wallet
- access: Tails core developers only
- tools: bitcoind
- configuration:
bitcoind
class - Vcs-Git: bitcoin and libunivalue
- importance: medium
- To save disk space: as the
bitcoin@bitcoin.lizard
user, runbitcoin-cli getblockcount
to get the ID of the last block, then runbitcoin-cli pruneblockchain XYZ
, withXYZ
being a Unix timestamp that's at least 5 months in the past.
BitTorrent
- purpose: seed the new ISO image when preparing a release
- documentation
- access: anyone can read, Tails core developers can write
- tools: transmission-daemon
- configuration: done by hand (#6926)
- importance: low
DNS
- purpose: authoritative nameserver for the
tails.boum.org
andamnesia.boum.org
zones - access:
- anyone can query this nameserver
- members of the mirrors team control some of the content of the
dl.amnesia.boum.org
sub-zone - Tails sysadmins can edit the zones with
pdnsutil edit-zone
- tools: pdns with its MySQL backend
- configuration:
- importance: critical (most of our other services are not available if this one is not working)
GitLab
- purpose:
- host Tails issues
- host most Tails Git repositories
- access: public + some data with more restricted access
- operations documentation: GitLab
- end-user documentation: GitLab
- configuration:
- immerda hosts our GitLab instance using this Puppet code.
- We don't have shell access.
- Tails system administrators have administrator credentials inside GitLab.
- Groups, projects, and access control:
- high-level documentation
- configuration: tails/gitlab-config
- importance: critical (needed to release Tails)
- Tails system administrators administrate this GitLab instance.
- See our documentation about GitLab for Tails sysadmins.
Gitolite
- purpose:
- host Git repositories used by the puppetmaster and other services
- host mirrors of various Git repositories needed on lizard, and whose canonical copy lives on GitLab
- access: Tails core developers only
- tools: gitolite3
- configuration:
tails::gitolite
class - importance: high (needed to release Tails)
git-annex
- purpose: host the full history of Tails released images and Tor Browser tarballs
- access: Tails core developers only
- tools: git-annex
- configuration:
- importance: high (needed to release Tails)
Icinga2
- purpose: Monitor Tails online services and systems.
- access: only Tails core developers can read-only the Icingaweb2 interface, sysadmins are RW and receive notifications by email.
- setup: We have one Icinga2 instance installed on a dedicated system used as the master of all our Icinga2 zones. We use a VM on the other bare-metal host as the Icinga2 satellite of our master. Icinga2 agents are installed on every other VM and the host itself. They report back to the satellite, which transmits to the master. We spread the Icinga2 configuration with Puppet. This way, we achieve a certain isolation where the master or the satellite have no right to configure agents or run arbitrary commands on them.
- tools: Icinga2, icingaweb2
- configuration:
- master:
tails::monitoring::master
class.- some configuration in the ecours.tails.boum.org node manifest.
- See Vpn section.
- web server:
tails::monitoring::icingaweb2
class, that wraps around upstreamicingaweb2
module.- some configuration in the ecours.tails.boum.org node manifest.
- satellite:
- agents:
- private keys are managed with the
tails_secrets_monitoring
Puppet module
- master:
- documentation:
- importance: critical (needed to ensure that other, critical services are working)
Internal XMPP service
- purpose: an internal XMPP service that can be used by Tails developers and some contributors.
- access: at the moment everyone that is on the tails-summit mailinglist has and/or can request an account.
- tools: prosody
- configuration:
- importance: low
Jenkins
- purpose: continuous integration, e.g. build Tails ISO images from source and run test suites
- access: only Tails core developers can see the Jenkins web interface (#6270); anyone can download the built products
- tools: Jenkins, jenkins-job-builder
- configuration:
- master:
jenkins
classtails::jenkins::master
class- a few Jenkins plugins installed with
jenkins::plugin
- YAML jobs configuration lives in a dedicated Git repository; Jenkins Job Builder uses it to configure Jenkins
- slaves:
tails::iso_builder
,tails::jenkins::slave
,tails::jenkins::slave::iso_builder
,tails::jenkins::slave::iso_tester
, andtails::tester
classes- signing keys are managed with the
tails_secrets_jenkins
Puppet module
- web server:
- master:
- design documentation: Jenkins
- importance: critical (as a key component of our development process)
- purpose: handle incoming and outgoing email for some of our Schleuder lists
- access: public MTA listening on
mail.tails.boum.org
- tools: postfix, amavisd-new, spamassassin
- configuration:
tails::postfix
,tails::amavisd_new
, andtails::spamassassin
classes - importance: high (at least because WhisperBack bug reports go through this MTA)
Meeting reminder
- purpose: send email reminders, for example about upcoming meetings
- access: not applicable
- configuration:
- to add a new reminder, or modify an existing one:
- implementation:
tails::meeting
,tails::meeting::reminder
, andmeeting.py
script
- importance: to be defined
Mumble
- purpose: internal communication for some internal teams
- access: members of some internal teams
- tools: mumble-server
- configuration:
- https://github.com/voxpupuli/puppet-mumble
mumble::*
parameters in Hiera
- importance: low
rsync
- purpose: provide content to the public rsync server, from which all HTTP mirrors in turn pull
- access: read-only for those who need it, read-write for Tails core developers
- tools: rsync
- configuration:
tails::rsync
- users and credentials are managed with the
tails_secrets_rsync
Puppet module
- importance: critical (needed to release Tails)
Schleuder
- purpose: host some of our Schleuder mailing lists
- access: anyone can send email to these lists
- tools: schleuder
- configuration:
tails::schleuder
classtails::schleuder::lists
Hiera setting
- importance: high (at least because WhisperBack bug reports go through this service)
Tor bridge
- purpose: provide a Tor bridge that Tails contributors can easily use for testing
- access: anyone who gets it from BridgeDB
- tools: tor, obfs4proxy
- configuration:
- importance: low
VPN
- purpose: flow through VPN traffic the connections between our different remote systems. Mainly used by the monitoring service.
- access: private network.
- tools: tinc
- configuration:
- importance: transitively critical (as a dependency of our monitoring system)
Web server
- purpose: serve web content for any other service that need it
- access: depending on the service
- tools: nginx
- configuration:
- importance: transitively critical (as a dependency of Jenkins)
Weblate
- URL: https://translate.tails.boum.org/
- purpose: web interface for translators
- design documentation
- usage documentation
- admins: to be defined (#17050)
- tools: Weblate
- configuration:
- importance: to be defined
WhisperBack relay
- purpose: forward bug reports sent with WhisperBack to tails-bugs@boum.org
- access: public; WhisperBack (and hence, any bug reporter) uses it
- tools: Postfix
- configuration:
tails::whisperback::relay
class- private keys are managed with the
tails_secrets_whisperback
Puppet module
- importance: high