Ticket: #8667

This blueprint helps to keep track of the discussions and decisions about the specification of automated tests ran in Tails' Jenkins on the automatically build ISOs (#5288).

Facts

Running the full test suite on 1 isotester hosted on Lizard takes around 8 hours. We intend to run 4 isotesters, so at the moment we would be able to run 12 full test suites per day. (#6094 will save ~25% on the full test suite runtime, but we'll keep on adding more tests, so let's stick to the 8 hours estimate for now.)

We have 2 isobuilders on Lizard, that build a total of a bit less than 400 ISOs/month (that's an average of 13 ISOs/day). At the moment, we build from 6 ISOs to 30-40 ISOs a day, depending on the activity.

We usually build the stable, devel, experimental, feature/jessie (+ testing sometimes) and a bunch of other branches.

This numbers are expected to grow when the automated builds will be put in production. It's difficult to guess what would the maximum number of builds per day be, but the minimum should be expected to raise to in between 10/20 ISOs/day.

Things to keep in mind

There will probably be new hardware at the end of the year to deploy more isotesters. If a machine is dedicated to that usage, we can throw in faster CPUs and run the test suite on bare metal, which would speed up the test process. That's #9264.

So in this discussion, we have to think to a deployment that might have two iterations with different computational powers (and thus different amounts of tests/day possible), and the defined implementation should be modular enough to handle both of them without too many changes.

Questions

When to test the builds

It doesn't sound reasonable to run the full test suite on every ISO automatically built by Jenkins, at least in the first iteration.

We need to lessen the number of tests per day. Probably even in the second iteration, with the expected growth of the number of automated builds.

At first, we'll try to run the full test suite on all of them. If our infra can't cope with that, we have ideas written below to help in that. The one that everyone seem to agree on is to run the full test suite for base branches and features branches that are in ReadyForQA status in Redmine.

We might want to play with job priorities, the same way we'll do with the automated builds (see #9760). Luckily, configuring that for the builds will probably also work by itself for their automated tests, as this jobs will be chained together.

Current proposal

For branch stable:
  Must test ISOs built daily and on every Git push;
    so that we're always ready to put out an emergency release;
For the next scheduled release branch (either stable, testing, or devel):
  Must test ISOs built daily and on every Git push;
    so that we always know the state of the next release;
For other branches:
  For branches with a `Ready for QA` ticket:
    Must test ISO built daily and on every Git push;
  Must test ISOs built on every Git push,
    with some rate-limiting if necessary;

How to run the tests

This section heavily depends on the discussion about the previous one.

To test a specific ISO, the automated test suite MUST have access to:

  • the corresponding build artifacts;
  • the commit ID of that build;
  • if applicable, the commit at which the base branch was at, when it was merged into the branch being built;
  • the ISO of the previous Tails release.

The automated test suite MUST be run in a clean environment, using a fresh --tmpdir for each run.

The automated test suite MUST be run on a freshly built ISO, corresponding to the commit it tests.

When #9515 and friends will be resolved and a first deployment of the automated test suite will clarify its need, we'll see if it should be able to accept a treshold of failures for some features before sending notifications. This could help if a scenario fails e.g. because of a network congestion.

Notifications

Email will be the main interface. Use the same settings than for automated builds:

  • For base branches, notify the RM.
  • For topic branches, notify the author of the last commit.

When notifying Redmine, the use of the ReadyForQA custom field is not so semantically correct, and may overlap with the dev usage of it. We will probably have to add a new one for Jenkins to decide to test the branch if we choose to test only the feature branch that are ready to be reviewed.

What kind of result shall be kept

The test suite produces different kind of artifacts: logfiles, screen captures for failing steps, snapshots of the test VM, and also videos of the running test session.

We can keep the video captures in the build artifacts, now that #10001 is resolved.

Decision:

  • For green test suite run: keep the test logs (Jenkins natively do that).
  • For red test suite run: keep the screenshots and video captures, the logs and the pcap files.

In #10155 we calculated that we can probably keep the video captures for a full release cycle. This will be refine is reality claims the contrary after an evaluation.

Scenarios

In the following scenario:

  1. topic branches are named branch F
  2. base branches are named branch B

We also assume that the following scenarios expectations only cover the test suite artifacts, the ISO one being covered in the automated builds specification.

Scenario 1 : reviewer

As a reviewer
When I'm asked to review branch F into branch B
Then I need to know if branch F passes the automated tests
  once locally merged into branch B and built (fresh results!)
And if all the automated tests scenarios suceeded
  The resulting test logs must be made available to me
  The Redmine ticket should be notified
Otherwise the developer who proposed the merge must be notified
  And the developer *needs* to see the test logs and screenshots
  And the ticket should be reassigned to the branch submitter
  And QA check should be set to "Dev needed"

Scenario 2 : developer

As a developer who has the commit bit
When I'm working on branch F
Then I need to know if my branch passes the automated tests suite
  once locally merged in base branch B
And I need to know if my branch fails to pass the automated test
  suite because of an external change possibly weeks after my
  last commit (by e.g Debian changes, changes in branch B, ...)
And if my branch passes the automated test suite
  The resulting test logs must be made available to me
  The Redmine ticket should be notified
Otherwise I *need* to see the build logs and screenshots
  And I must be notified
  And the ticket should be reassigned to me if needed
  And QA check should be set to "Dev needed"

Scenario 3 : RM

As the current RM
When working the full dev release cycle
Then I need to know if a base branch does not pass the automated
  test suite
And when this happens the build logs and screenshots must be made
  available to me.

Future ideas

Cutting the number of run per day

For feature branches, we could run the full test suite only on the daily builds or on ReadyForQA ones, and either only the automated tests related to the branch on every git push, and/or a subset of the whole test suite.

We can also maybe find more ways to split the automated test suite in faster subsets of features depending on the context, define priorities for built ISO and/or tests.

The automated test suite could be able to run features in parallel for a single automated build ISO. This way, if more than one isotester are idle, it can use several of them to test an ISO faster.

The automated suite then should be able when there are more than one ISO queued for testing to fairly distribute the parallelizing of their features.

The automated test suite also should not allocate all the isotesters for one ISO when others are waiting to be tested.

Scenarios

Folowing is a list of other scenarios that we have also envisaged for the second iteration of the automated builds deployment, as they are really tightened.

Scenario 10

As a Tails developer working on branch B
When I upload a package to APT suite B
Then I want to know if the automated test suite passes

(same responsiveness as when pushing to git) (acceptable workaround: being able to manually trigger a test suite.)

Scenario 11

As the current RM
When I push new tag T on branch B
Then I want the APT suite for tag T to be created
And I want the APT suite B to be copied into the APT suite T
And once this is done, I want the automated test suite to be run
  on the ISO build from the checkout of tag T

Scenario 12

As a Tails developer
When the test suite is ran on the ISO build from my last commit
I want to watch TV and see the test video in HTML5 from the Tor Browser

Scenario 14

As a Tails developer
When I push a new commit or a new Debian package to a base branch
I want the affected feature branches ISO to be tested with that change