Our current approach to managing Tails todo does not scale. We have some ideas to improve it slightly, but it's unlikely the result will be good enough, so we will instead migrate to Redmine.

Roadmap

All this was done!

See our migration scripts.

  • write a conversion script to import our existing tickets (see Convert and import section below).
  • update developers documentation
  • migrate!
    • import existing tickets
    • setup rewrite rules from the ikiwiki tickets URL to the new ones
    • remove tickets that are not blueprints from ikiwiki
  • If needed, have additional plugins packaged and installed to fine tune our processes.

Wiki / Tasks

Let's keep our wiki for blueprints and research for long-term big next features.

Let's try not to use the wiki for tasks (else it defeats the purpose of tracking tasks), let's not try to use the tasks manager for research and other things that would be better suited for the wiki (else it harms mostly-offline developers).

Specifications

Must have

  • Should have categories (and display the related tasks in a meaningful way). => Redmine, bugs-everywhere
  • Should let us assign tasks to individuals => bugs-everywhere
  • Can handle tasks and sub-tasks => Redmine, bugs-everywhere
  • Create tickets over email. => Redmine, bugs-everywhere
  • Reply to tickets over email. => Redmine, bugs-everywhere
  • Being able to search through tickets offline (caching). Ability to import a DB dump into a local instance of the webapp would be an acceptable way to do it, even if clearly suboptimal. => bugs-everywhere
  • Email notifications: subscribing to a ticket, or to new tickets, etc. => Redmine (for existing tickets), bugs-everywhere
  • Advanced search queries, or filters (among metadata) (roadmap, milestone, overview, individual dashboard) => Redmine
  • Hosted somewhere else or fully available in Debian (or all deps in Debian? to be discussed if need arises.) => Redmine (Riseup Labs & in a little bit outdated Debian), bugs-everywhere (needs some care)
  • milestones (target version) => Redmine, bugs-everywhere
  • Random users may reply to tasks. => bugs-everywhere

Important

  • Blocking property => Redmine, bugs-everywhere
  • Change a ticket's metadata via email. => Redmine, bugs-everywhere
  • Due time, deadlines => Redmine (with email reminders), bugs-everywhere

Bonus

  • Having tags => bugs-everywhere
  • Possible to assign tasks to a team / to multiple people
  • Sub-tasks having precedence support or wait state or something that makes the "let's create all future sub-tickets for a task at the same time" workflow practically doable (i.e. I don't want to be shown all future steps that are blocked by the first one)
  • Possible to change a ticket's state from Git commit messages (todo->pending), bonus points if merging that branch into stable ->done, or something like that.
  • Reminder for deadlines.

Unsorted

  • Encryption support.

Redmine

  • homepage
  • easy to setup thanks to the Debian package
  • supports issue creation or comments via email,
  • supports updating ticket attributes over email
  • the email interface, according to someone I trust who set it up and uses it, is quite fragile and can be a bit mysterious to setup
  • apparently, there no more powerful way to interact with it asynchronously (feature request for offline mode)
  • Textile syntax. Pandoc knows how to convert our existing Markdown to Textile, but still, having Markdown support would be awesome.
  • Git integration won't easily work with our many-branches workflow, but we can probably do without for a while.

Plugins of interest

  • Markdown syntax. A blocker is that each such plugin switches the syntax instance-wide, no way to enable it for a single project.
  • Carousel: can be used for periodic actions that occur during project development process. It automatically generates issue assigned to the next user in the carousel queue every specified time period. Marked as compatible with 1.1.x and 1.4.x.

    Looks interesting, but most (if not all) tasks we do switches for depend on our availability etc., so I'm not sure an automated solution would do. --intrigeri

  • Digest: send a summary of a project's activity over a period of time by email. Marked as compatible with 1.4.x and 2.0.x. The sample email output does not make me think this is any better than some way (feed reader or rss2email) to subscribe to the project's activity.
  • Importer: import issues in bulk from .csv files. Compatible with 1.1.x. We do need this to import existing tickets. A fork supports Redmine 2.x.
  • Issue checklist: add checklist functionality to issues. Marked as compatible with 2.0.x -- what about 1.4.x or older? Having something lighter than sub-tickets could be good, but certainly not critical. We'll see.
  • Git branch hook: add issue related revision by branch name. Can close tickets on merge. One apparently may configure the branch that, when merged into, triggers this behavior, thanks to the merge_branch setting. Apparently impossible to use it for the two-steps pending + fixed workflow we've been using, but we may want to change this when switching tools anyway. According to Git log, should at least support Redmine 1.4.x and 2.0.x. We can start using Redmine and see later how much we need something like this.
  • Silencer: suppress email notifications (at will) when updating issues. Marked as compatible with 1.1.x, not with 1.4.x.

    I'm not sure why we would want this. --intrigeri

  • Whining: email alerts when an issue had not been updated since X days. Marked as compatible with 1.1.x, not with 1.4.x. Would be very useful, looks easy to package and install.
  • Custom Workflows: define own rules of issue processing, e.g. change issue properties or create sub-task if some conditions are met, enforce policies... Marked as compatible with 1.2.x to 2.1.x. Hopefully we won't need it, but it's still good to know it exists.

Offline usage

  • Sending an email can create an issue with:
    • subject: mail subject
    • description: mail body
    • tracker: keyword (Tracker:) in mail body
    • priority: keyword (Priority:) in mail body
    • status: keyword (Status:) in mail body
    • category: keyword (Category:) in mail body
    • assignee: keyword (Assigned To:) in mail body
    • kind of next thing to be done: keyword in mail body
    • target version: keyword (Fixed version:) in mail body
    • QA check (or any other custom field): keyword (name of the custom field) in mail body
  • Sending an email with '[#24175]' in the subject will add information to the ticket. Same keywords as before can be used to change metadata.
  • Email address in From, To, or Cc are added to watchers (if they match a Redmine user) when creating a new ticket over email.
  • Clicking on Watch enables one to receive emails when the ticket changes.
  • Every ticket has an Atom feed that contains all changes made to a ticket.
  • There is an Atom feed with all open tickets, together with status and description.
  • Missing: set parent task, set related issues, delete ticket.

Convert and import

Note: this is an initial rough draft, that probably misses tons of things to do, but should be enough to run some initial tests to confirm the general idea is workable.

Set up Redmine

  1. done Add a External Id custom field. Check "Used as a filter".
  2. done Add a QA Check custom field. Make it searchable.
  3. done Add a Type of work custom field whose possible values are the same as our current todo/* tags: Code Discuss Documentation Pass Test Promote Qa Research Sysadmin Test Translate Upstream Wait Website Mark it as "used as a filter" and searchable.
  4. done Add a Blueprint text custom field, with regexp ^https://tails[.]boum[.]org/blueprint/.
  5. done Add a Fix committed issue status.
  6. done Make the Fix committed status available in Administration -> Workflow.
  7. done Add a Confirmed issue status.
  8. done Make the Confirmed status available in Administration -> Workflow.
  9. done Mark the Resolved issue status as "Issue closed".
  10. done Add an Elevated issue priority, rank it between Normal and High.
  11. done Add a Feature branch custom field to ease review.

Adapt impacted stuff

  1. done Prepare a branch that updates the website to advertise the new task manager instead of the old one.
  2. done Generate the Apache rewrite rules from the (External Id, Redmine Id) mapping. (PRODUCTION=1 make rewrite-rules should do the job.)

Clean up and gather data

  1. done Close wishlist tickets not modified since more than a year.
  2. done Split tickets that have several todo/* tags, and save the parent/child relationship using [[!parent ]]. In the end, each ticket should only have one todo/* tag:

     PRODUCTION=1 make list-more-than-one-todo-tag
    
  3. Write a custom ikiwiki plugin to:

    • done save original ikiwiki ticket name as External Id column
    • done save parent/child relationship
    • done wrap the whole ticket information into a CSV line
    • done filter out toc
    • done save todo/* tags into the "Type of work" column
    • done save release/1.0 and todo/2.0 tags into the "Fixed version" column, ditto for broken windows
    • done turns todo/qa into QA Check == Ready for QA.
    • done convert ikiwiki shortcuts to external links
    • done turn internal links to non-todo pages into external links
    • done convert Markdown to Textile
    • done removes links to wishlist
    • done converts TOC directive into Redmine's syntax
    • done drops obsolete taglinks
    • done import priority/* tags
    • done import category/* tags
    • done feed the blueprint custom field with the URL to the relevant blueprint, if any

Freeze and import data

  1. Run ikiwiki with this special plugin, then merge all these CSV lines to produce a first CSV file that can be imported with the Importer plugin.

     PRODUCTION=1 make export1.csv
    
  2. Import the first CSV file:

    • use an account that has Manager privs on the current project
    • ignore Parent Issue fields
  3. Import this CSV file again to apply the relationships:
    • use an account that has Manager privs on the current project
    • use External Id as Unique Column
    • check "Update existing issues"
  4. Build a (External Id, Redmine Id) mapping

    PRODUCTION=1 make ids.map
    
  5. With a modified version of the ikiwiki plugin, plus this mapping information, produce a second CSV files with tickets description modified so that links to other tickets are updated to use Redmine ticket linking syntax.

     PRODUCTION=1 make export2.csv
    
  6. Import the second CSV file, using External Id as Unique Column again, and enabling the Update Existing Issues option. Links between tickets should now be good.

Polish imported data and update the rest of the world

  1. done Mangle the content of the Git repository:
    • Move blueprints tickets to wiki/src/blueprint/: PRODUCTION=1 make move-blueprints
    • Move attachments out of the way for future processing: PRODUCTION=1 make move-attachments
    • Delete the rest.
  2. done Manually take care of tickets sub-pages that were imported as full-blown tickets. These sub-pages are used in too many different ways to allow us to process them automatically:
  3. done Manually add attachments (the Importer plugin does not support this).
  4. done Add some tickets relationships back.
  5. For each ticket:
    • Add a Category value.
    • done Add a Type of work value.
    • Decide if it should have been kept as a blueprint. If so, salvage the content of the old ikiwiki ticket into a shiny new blueprint.
    • If Type of work is wait, mark the ticket as blocked by the relevant other ticket, if any.
  6. done Move to the wiki stuff that should never have been migrated.
  7. done Remove the Qa value from possible types of work.
  8. done Mark the Type of work custom field as required.
  9. wontfix Sort bugs and features to the relevant trackers.
  10. done Add sub-tickets blocks/before/after relationships, starting with tickets that are tagged todo/wait for another one.
  11. done Remove the External Id field.
  12. wontfix Clean tags/* up.

Setup

  1. Install a Squeeze system with backports enabled.
  2. Install packages:

     apt-get install redmine/squeeze-backports redmine-sqlite/squeeze-backports
    
  3. Follow the "QUICK LAUNCH USING WEBRICK" instructions in /usr/share/doc/redmine/README.Debian.gz

  4. Install the importer plugin:

     cd /usr/share/redmine/vendor/plugins && git clone https://github.com/leovitch/redmine_importer.git
     cd /usr/share/redmine && rake db:migrate_plugins RAILS_ENV=production
    
  5. Install a backport of ruby-fastercsv, that's a dependency of the importer plugin. One has to decrease the build-dep on gem2deb to 0.2.7~.

  6. Restart WEBrick.