We already have an automated test suite. This page is about tools that could allow us to improve it.

Tools

behave

  • homepage
  • Cucumber-like, in Python
  • used by GNOME
  • examples from the eog source tree, that use behave and dogtail:
  • In Debian (in unstable since 2017/07/20)
  • Python (with Jython) is now Sikuli's preferred scripting language; it's also the language that has the best maintained bindings to interact with libvirt, accessibility technologies, and more
  • does behave work fine under Jython?

dogtail

  • homepage
  • actively maintained upstream as of 2017-05-31
  • GUI test tool and automation framework written in ​Python
  • uses Accessibility (a11y) technologies to communicate with desktop applications
  • used by GNOME in combination with behave: see the section about that one
  • in Debian Wheezy, Jessie, and (as of 2016-02-14) Stretch; but the package is orphaned
  • how much do we still need Sikuli if we have dogtail?
  • ships a sniff GUI that allows one to explore GUI applications widgets and content using a11y technologies: very useful if we use dogtail, or any similar tool
  • most of the work is done by RedHat

Misc helpers

  • The pyatspi and Accessibility Python modules can be helpful regardless of the actual tool we use. E.g. pyatspi can allow us to listen for GUI events. Also see gir1.2-atspi-2.0 and gir1.2-wnck-3.0.
  • Accerciser, an interactive accessibility explorer for the GNOME desktop

autopilot

LDTP

LDTP is an open source testing tool that uses computer assistive technology (accessibility) to automate GUIs. It is (was?) used by GNOME, Mozilla and others:

  • Linux Desktop Testing Project
  • homepage
  • latest upstream release (as of 2016-02-14) is 3.5, released mid-2013
  • Debian Wheezy, Jessie and (as of 2016-02-14) Stretch have 2.3.1, released 2012-02-26
  • tutorial
  • The main bindings are Python, but there also are a Ruby client and Perl bindings in the Git repo
  • The LDTP dev mailing list is very quiet, and it's unclear whether GNOME still uses it, or instead switched to dogtail.

misc

  • Martin Pitt announces umockdev (source code), a set of tools to record and mock hardware for debugging and testing

Open questions

Using accessibility technologies?

#10721

In some cases, it could simplify some testing steps, such as anything about navigating menus, that we're currently mostly avoiding since it's hard to do in a robust way with Sikuli.

A downside is that we're not exactly testing how most users interact with the software. Some upsides are that it would ensure that our stuff does support accessibility technologies, and that we would have to maintain much less pictures.

How to do it

We want to build a high-level Ruby library that helps us generate bits of Python (that use e.g. dogtail) that we'll then send to the system under testing (using the remote shell we have already), and run there.

For example, one feature this library should provide would be to select a particular conversation tab in Pidgin, based on the conversation tab's name; that is, using text and not a picture; this can be especially useful when we learn what name we are looking for dynamically, at run time, and thus can't add the corresponding picture to Git.

Some stuff won't be doable in a nice, high-level way, so likely we will need, for some corner cases, lower-level access to the generation of the Python code that will be sent and run.