This is about migrating our data and workflow from Redmine to GitLab, which is tracked as #15878.

Issues tracking

See also the GitLab doc on issues.

Private issues

One can make an issue confidential when creating it; confidentiality can later be toggled on/off at any time. A confidential issue is visible only by whoever created it and by project members with at least Reporter access.

Given "Reporter" access is needed for lots of relatively basic operations, such as labelling and assigning issues, presumably most active Tails contributors would get at least this access level on the tails-team/tails project, and then in turn access to this project's confidential issues. So Tails teams that need to further restrict access to their confidential issues will need to track their issues under their own tails-team/$project. See "Particularly sensitive data" in the "Access control" section.

Relationships

On Redmine we heavily use relationships between fine-grained issues: parent/subtasks, Related, Blocks, etc. There's no such thing in GitLab FOSS ("CE") edition (feature request) so we'll need to significantly change our workflow here.

Below we describe potential solutions.

Parent/subtask and Blocks relationship

A GitLab issue can have a list of tasks (checklist). GitLab displays "X of Y tasks completed" prominently.

Describing each subtask in free-form text, directly as an item on this list of tasks, should work fine to emulate a set of subtasks that are all assigned to the same person. For example, see nautilus/issues/667 on GNOME GitLab.

For more complex cases, e.g. when non-trivial subtasks are done by different people (GitLab CE does not support multiple assignees per issue), each subtask could be an item on this list of tasks, that links to another issue. The downside is that after resolving one of the subtasks, one will also need to mark the corresponding item as completed on the task list of the "parent" issue.

This should also replace most of our current usage of the Blocks relationship.

Alternatively, one can add a comment with a list of the blocker issues. Comments can be updated and links will indicate which of the blockers are closed (strike-through).

Related to

We can write this information directly either in the description of one of the issues that are related to each other or in a comment. Either way, this adds a message in the Activity stream of the referenced issue, with a link to the other issue. Granted, on issues with one single long discussion thread, as we're used to on Redmine, such messages will be lost, so better cross-reference issues in the description of each related issue, or use labels, or get used to "Start Discussion" to separate multiple sub-threads that can be marked as resolved independently from each other.

Duplicates

We can close duplicates with a comment that references the duplicated issue. It adds a message to the Activity stream of the referenced issue, which allows one to find duplicates later on if needed.

And to ensure we can list issues that have really been resolved, add a "Duplicate" label.

Status

Each open issue must have one of these labels:

  • "1. To do" (previously: "Confirmed")
  • "2. Doing" ("In progress" was too vague: it could mean anything between "someone did the first 2% of the work 5 years ago" to "this is what I'm focused on today")
  • "3. Needs Validation"

… except issues that were just created and need to be triaged (previously: "New").

This lends itself to issue boards with 4 columns: "1. To do", "2. Doing", "3. To review", and "Closed".

Closing an issue means one of:

  • The fix or feature the issue is about was merged and will be in a future release.

    To list these issues: closed issues whose milestone is a version was not released yet.

  • The fix or feature the issue is about is already available to our users (previously: "Resolved").

    To list these issues: closed issues whose milestone is a version that's been released already.

  • We've rejected it or marked it as a duplicate (previously: "Rejected" and "Duplicate")

    To list these issues: closed issues with respectively the "Rejected" or "Duplicate" label.

Most closed issues will still have the "3. To review" label. That should not cause any problem in practice. Worst case this can be fixed automatically, either via a webhook or a scheduled batch job.

Other issues metadata

  • Target version → Milestone
  • Feature branch: GitLab will automatically link a branch that mentions an issue.
  • Category, Affected Tool, Priority, Type of work → use a set of labels, each with a prefix, for each of them; and maybe simplify a bit. For example:
    • Qubes uses "P: $priority", "C: $category", etc.
    • One can set multiple labels so we could perhaps merge "Category" and "Affected Tool". For example, a ticket about Thunderbird persistence could have the two "C: email" and "C: persistence" labels.
  • Log time: was enabled as an experiment, not actively used anymore; contributors can use GitLab's Time tracking if they wish so.
  • Due date → Due date
  • Starter → dedicated label
  • Tracker: drop it (we've never really taken advantage of the fine distinction between describing a problem as a bug vs. describing its solution as a feature)
  • % Done: drop it (we don't use this field enough to provide any value)
  • Estimated time → description
  • Attachments: any issue/MR description or comment can have file attachments

Project management

Currently we use the Redmine "Deliverable for" custom field. A "Deliverable for SponsorX" label should do the job for tracking global progress on a grant: then one can use the issues list or an issues board to visualize progress (despite the doc saying otherwise while I'm writing this, one can now have multiple issue boards in the community edition of GitLab).

An issues list can be ordered by due date, milestone due date, etc., which should emulate the most important aspects of the Redmine custom queries we use, except naming the query (see "Custom queries" below).

We can track progress on a specific project (be it a grant deliverable or not) by adding another label, e.g. "Project XYZ". For example, https://redmine.tails.boum.org/code/projects/tails/issues?query_id=298 lists issues that have "Deliverable for" = "SponsorX" and whose parent task contains #14568. We would have labels "Deliverable for SponsorX" and "Project XYZ". And if an additional intermediary level of tracking is needed between "this is part of SponsorX" and "this is about Additional Software" we can create tracking issues that each have list of tasks.

Personal self-management

For planning, the same solutions as for project management apply, modulo adding a filter for the assignee.

And for more lightweight tracking, GitLab's Todos are great. Todos are fully orthogonal to "tickets assigned to me"; they are listed on a different page and can be marked as done with one single click. For example:

  • Every time one mentions me somewhere, I get a Todo item. This allows me to track the questions I've been asked, that is, to replace our past usage of "Info Needed" and current usage of @nick "mentions", without the need to reassign an issue nor to create a subtask. As a bonus, for many of us, a list of Todos made prominent by GitLab will be easier to keep track of than mere email notifications.

  • I can click "Add todo" on an issue and it will appear on my list of Todos (regardless of whether I'm the assignee or not).

Core team (self-)management

XXX: how to replace e.g. https://redmine.tails.boum.org/code/projects/tails/issues?query_id=307?

Use a dedicated set of labels?

Custom queries

We use Redmine custom queries to have easy access to named searches and visualizations that we either often need, or that we want to share within a team or the overall Tails project.

In GitLab, the closest equivalent to Redmine custom queries is the URL of an issues list or issues board filtered by assignee, milestone, and/or label.

We will need ways to share these URLs and ideally, to name them. We can do that on the GitLab project's description (home page), on parent tracking tickets, on blueprints, on a team's page on our website, and possibly in GitLab's own wiki if we decide to use it (either only for this use case or for our blueprints).

Access control

Objects

  • Canonical Git repo: the authoritative tails.git repository hosted on GitLab

  • Major branches: master, stable, testing, devel, and possibly feature/bullseye

  • Release tags: a signed Git tag that identifies the source code used to build a specific Tails release; currently all tags in the authoritative tails.git repository are release tags; the tag name is a version number, with '~' replaced by '-'.

  • Particularly sensitive data: confidential data that specific teams like Fundraising and Accounting need to handle, but that other contributors generally don't need direct access to. This definitely include issues; this might include Git repositories at some point.

    Note that as of 2019-11-20, it is undefined:

    • What subset of this data can go to a web-based issue tracker or not.
      This is already a problem with Redmine.
      Fixing this will require discussions between various stakeholders.

    • What subset of this data could live in a cleartext Git repository hosted here or there, as opposed to requiring end-to-end encryption between members of these teams. This is a hypothetical problem for now.

Subjects

  • An admin can do anything that other roles can, and:

    • can delete issues
    • can edit team membership
    • MUST comply with our "Level 3" security policy
    • can view issues that contain particularly sensitive data
  • A committer:

    • can push and force-push to any ref in the canonical Git repo, including major branches and release tags;
      incidentally, this ensures the following requirement is met:
    • their branches are picked up by Jenkins; it follows that they MUST comply with our "Infrastructure" security policy
    • can merge MRs into major branches
    • can modify issues metadata
    • MAY be allowed to view confidential issues in the main GitLab project; if it's the case, then particularly sensitive data MUST live somewhere else with stricter access control
    • can edit other users' comments
    • MAY be allowed to add new team members
    • MUST comply with our "Level 3" security policy
  • A regular, particularly trusted contributor:

    • can push and force-push to a subset of refs in the canonical Git repo; this subset MUST NOT include any major branch nor release tag;
      this is required to ensure the following requirement is met:
    • their branches are picked up by Jenkins; it follows that they MUST comply with our "Infrastructure" security policy
    • can modify issues metadata
    • MAY be allowed to view confidential issues in the main GitLab project; if it's the case, then particularly sensitive data MUST live somewhere else with stricter access control
  • A regular contributor:

    • can fork the Git repositories and push changes to their own fork
    • can modify issues metadata
    • MAY be allowed to view confidential issues in the main GitLab project; if it's the case, then particularly sensitive data MUST live somewhere else with stricter access control
  • Anybody with a GitLab account on the instance we use:

    • can submit issues
    • can submit MRs

Implementation

Relevant GitLab doc

Idea: Protected branch flow

The Protected branch flow may be the most adequate for regular contributors, since it's the least disruptive in terms of workflow and habits and requires no work to adjust our Jenkins CI setup:

  • We mark our major branches and release tags as "Protected".
  • Committers get "Maintainer" access.
  • The Git workflow remains unchanged for regular developers who are not granted full commit access: they get "Developer" access, can push a topic branch to the canonical Git repository and our CI will pick it up. The only difference is that they are not restricted to pushing to their own $nickname/* namespace, which makes things simpler and has other advantages, e.g. they can use the wip/ prefix (so our Jenkins CI ignores the branch) and collaborate with others on shared branches.
  • Other contributors get access strictly lower than "Developer". They push topic branches to their own fork of the repository and create merge requests.
  • Our current Jenkins CI jobs generation process remains unchanged. (Technically, we could adjust it so it generates CI jobs for any merge request (based on refs/merge-requests/*/head), but this would give arbitrary code execution on our CI to anyone. Our infrastructure is not currently equipped to cope with this.)

Wiki

It's out of scope for the first iteration but at some point, we might want to migrate our blueprints to GitLab's wiki: #9174).

Importing data from Redmine

Specification

Note: some of what follows still needs to be negotiated with the Tails community.

Scope

  • issues in the Tails Redmine project; data from other projects does not matter to us
  • target versions (milestones)

Out of scope

  • Redmine wiki: we don't use it

MUST

  • For every existing Redmine issue, the meaning of its #NNNN ID MUST be preserved unambiguously. This implies that we can't reuse existing Redmine issue IDs in our main GitLab project, where most issues will live.)

  • #NNNN and https://redmine.tails.boum.org/code/issues/NNNN links MUST point or — possibly transitively — redirect to the relevant GitLab issue.

    Implementation idea: start with a static web redirector, and then:

    • This is sufficient to solve the problem for issues imported into our main GitLab project, if they have the same ID as they used to.

    • For issues imported into another GitLab project, we either need extra redirection rules based on a mapping from Redmine issue IDs to GitLab (project, issue ID) tuples, or placeholder issues on GitLab that handle a second redirection step (GitLab might handle the redirection itself when an issue is moved from one project to another)

  • Preserve issue private/public status. Given GitLab's permissions model, this implies that some issues MUST be migrated to another GitLab project than the default one where most of our issues will live; for details, see the "Private issues" section above.

    Example: #10346

  • Preserve parent/subtask relationship in a way that GitLab will enforce, i.e. prevent closing the GitLab issue corresponding to the Redmine parent issue, as long as one of its subtasks is not completed.

    Examples: #15878, #12213

    Implementation idea: a GitLab issue imported from a Redmine issue that has subtasks could have a list of tasks (checklist), with each subtask being an item on that list; ideally, the status of each such Redmine subtask (closed or open) should be reflected as the status of the corresponding GitLab checklist item (completed or not).

  • Preserve the "Blocks" and "Related to" information.

    It's OK if the "Blocks" semantics from Redmine (if X blocks Y, one cannot close Y until X is resolved) is not enforced by GitLab.

    XXX: some of these relationships should be dropped on GitLab. see the "Core team (self-)management" section.

    Examples: #15878, #12213

    Implementation idea: add the "X blocks Y" information in the description of issues X and Y and/or as comments on X and Y.

  • Preserve issue status:

    • "New" → open
    • "Confirmed" → open, "1. To do" label
    • "In Progress" → open, "2. Doing" label
    • "Needs Validation" → open, "3. Needs Validation" label
    • "Duplicates" → closed, "Duplicate" label
    • "Rejected" → closed, "Rejected" label
    • "Resolved" → closed
  • Preserve "Target version" → Milestone

  • Preserve the "Feature branch" information.

    Adding this information as-is in a comment would be good enough.

    Example: #16173

  • Preserve Category, Affected Tool, Priority, Type of work… somehow.

    Implementation idea: with labels, see the "Other issues metadata" section above.

  • Preserve any true "Starter" boolean value.

    Implementation idea: dedicated label.

  • Preserve "Deliverable for" values for open issues.

    Implementation idea: "Deliverable for SponsorXYZ" label.

  • Preserve attachments

    Example: #7763

  • Convert Textile issue description and comments to Markdown (pandoc can do most of the work see https://pandoc.org/try/?from=textile&to=markdown):

    • commit:c6f4d0fd
    • bold
    • italic
    • strike-through
    • <pre> blocks
    • @code@
    • bullet lists, including nested ones

    Examples: #17241

    Textile: https://www.redmine.org/projects/redmine/wiki/RedmineTextFormatting GitLab Flavored Markdown (GFM): https://docs.gitlab.com/ee/user/markdown.html

  • Convert public notes to GitLab comments.

    Example: #17121

  • Private comments (notes): migrate their contents somewhere, and make it accessible from the corresponding issue, in the right place in the discussion.

    Example: #16875#note-3

    It looks like GitLab does not support private comments.

    XXX (intrigeri): check if it's OK to publicly leak the fact that someone (unspecified) posted a private note on an issue, along with the corresponding timestamp. To list all private notes, grouped by user:

      ./redlab discuss --verbose -C config.py --private-notes
    
    • 2019-11-20: asked the 10 affected users if they consent with leaking this information publicly.

    If folks don't consent to this infoleak, we'll add a link from the GitLab issue description to the privately archived Redmine, and to follow an old discussion, one may want to systematically follow this link to ensure they did not miss a private note.

  • Preserve issue assignee

    It is acceptable to require assignees to manually create their GitLab user account themselves before the final migration.

  • Existing issues assigned to a group must remain trackable, somehow.

    This feature is not intensively used. We can leave it up to the teams who use it to decide how they want to track these tickets on GitLab. A temporary label to not lose track of these tickets would be good enough as far as the migration is concerned.

    We have two such groups:

    • Help Desk: recently created, not used yet
    • Sysadmins: 15 tickets as of 2019-11-18
  • Preserve the value of the Blueprint custom field

    Adding this information to the issue description is good enough.

  • Preserve metadata changes

    It's good enough to:

    • Link to a static mirror of Redmine that has all the historical info. We need 2 mirrors: a public one and a private one (accessible only by people who had access to the private issues on Redmine).
    • Add this information as free form text as comments.
  • Preserve attribution (authorship) of actions, e.g. which user created an issue, wrote a note, or modified the metadata of an issue

    It's good enough to add this as free-form text in the corresponding issue or note, like e.g. https://gitlab.gnome.org/GNOME/gnome-shell-extensions/issues/22, but:

    • This information should come first in notes. Otherwise (as segfault reported) to follow a discussion, one first has to scroll down to the end of every note to get this key piece of context, and then scroll back up to read the note itself.
    • It would be really nicer if we can really preserve attribution.

SHOULD

  • Convert more Textile issue description and comments to Markdown (examples):

    • links to an issue note: #124-6, or #124#note-6
    • links with specific text: "Redmine web site":http://www.redmine.org
    • tables, if supported by GitLab
    • h1 and h2 headings
    • source:some/file

    Examples: #17241

  • Preserve watchers (as "participants") for open tickets.

    Example: #16875

    It is acceptable to give Redmine users a data file + script to run in order for them to re-subscribe to the issues they were watching.

Information that can be lost during the migration

See "Other issues metadata" section above for some discussion about why.

  • Log time
  • Start date
  • Due date
  • Tracker
  • % Done
  • Estimated time
  • Code syntax highlighting
  • List of commits that referenced an issue

To be triaged

Relevant features — to be triaged as MUST/SHOULD/MAY:

  • Does not require global GitLab administration token.
  • Does not require SSH access to the GitLab machine.
  • Existing user accounts
  • Description history

Tools

A number of tools are available online.

These data migration tools come in various shape and some don't support GitLab API v4, but generally there's a fork on GitHub that fixes the most critical problems. Start there, explore the network of forks, and follow the white GitHub rabbit(-hole):

Interfaces that will need adjustments

XXX: check what part of this is in the scope of this project

Git repository → various mirrors

Our Git repository keeps a number of mirrors up-to-date:

Git repositories → website

A refresh of our website is triggered by:

  • pushing to the master branch of our Git repository
  • pushing to a few "underlay" repositories

Current implementation:

Git repository → Jenkins

Pushing to our Git repository pings Jenkins so it can run jobs as needed: https://git.tails.boum.org/puppet-tails/tree/templates/gitolite/hooks/tails-post-receive.erb

Jenkins → Git

Jenkins jobs are generated on jenkins.lizard, from the list of branches in our main Git repository. For details and pointers to the corresponding code, see the corresponding blueprint.

Here are the kinds of jobs relevant in this discussion:

  • check_PO_master runs the check_po script on all PO files

    • this script comes from a Git submodule referenced by tails.git's master branch (tails::tester::check_po)
    • in a Jenkins isobuilder
    • as a sudoer user
  • build_website_master runs ./build-website

    • from tails.git's master branch
    • in a Jenkins isobuilder
    • as a sudoer user
  • build_Tails_ISO_* and reproducibly_build_Tails_ISO_* run rake build

    • from the corresponding tails.git branch
    • in a Jenkins isobuilder
    • as a sudoer user
  • test_Tails_ISO_* run ./run_test_suite

    • from the corresponding tails.git branch
    • in a Jenkins isotester
    • as root via sudo

Wrt. who can push to which branch in tails.git, see the "Access control" section.

Jenkins → Redmine

Tails images build reproducibility is tested if the corresponding ticket has "Needs Validation" status: https://git.tails.boum.org/puppet-tails/tree/files/jenkins/slaves/isobuilders/decide_if_reproduce

(Currently disabled:) Email notifications are sent on job status change if the ticket has "Needs Validation" status: https://git.tails.boum.org/puppet-tails/tree/files/jenkins/slaves/isobuilders/output_ISO_builds_and_tests_notifications

Ticket triaging → Redmine

Our Redmine email reminder sends email to users who have at least one ticket assigned to them, that satisfies at least one of these criteria:

  • "Stalled work-in-progress": status "In Progress", that were not updated since more than 6 months

  • "Reviews waiting for a long time": not been updated since 45 days or more

Current implementation:

Translation platform → Git

A hook ensures that Weblate only pushes changes to our main Git repository that meet certain conditions:

https://git.tails.boum.org/puppet-tails/tree/files/gitolite/hooks/tails-weblate-update.hook

Issues and merge requests triaging

GitLab hosting options

See https://salsa.debian.org/tails-team/gitlab-migration/wikis/hosting/comparison (tables in pure Markdown are too tedious for me -- intrigeri).

Resources