With this in mind, I thought it would be useful to bring attention to some new and under-used features of autopilot, along with providing a brief explanation of some companion tools you might find useful. Thus I present to you, an autopilot primer. Let's talk through some new features shall we?
Python3 Support
Autopilot started as a python2 tool but has since migrated to python3 and you should too! For now the entire source tree remains python2 compatible, but you really should migrate your tests to python3. You'll notice the autopilot3 binary in newer releases which should be used to run autopilot with python3.
Scenario Support
Scenarios are a wonderful way to keep your tests simple and easy to read while allowing you to test with many different inputs. In short, you might need to test several edge cases as part of your acceptance testing. This is most easily accomplished by keeping the test itself generic and utilizing a scenario to vary your inputs. You can check out more information on scenarios specific to autopilot in the autopilot documentation.
Screenshots / Video
Autopilot allows you to get a video recording of a test failure. To make sure autopilot records failures, install recordmydesktop and pass the -r argument to your autopilot3 run command. However, at the moment this requires X so for now it doesn't work with the MIR backend (which things like the ubuntu phone utilize). Fortunately a screenshot at the point of failure when combined with the log is generally sufficient to solving your issue. Getting those screenshots brings us to subunit support.
Subunit Support
By default autopilot generates the test output and logs straight to your console in a text format. However autopilot also supports outputting to xml and subunit. Subunit support is what I would like to highlight for a few reasons. When you set the output format as subunit, you get a few niceties. One of which is an easier to grok format for tools, and the other is screenshots of the application when failures occur. To get a subunit stream, pass the -f subunit argument to your autopilot3 run command. You will want to also pass -O with a filename to save the output to a file as the subunit stream contains binary data.
Test Result Viewer
So, with this subunit test results file, how can you enjoy all of it's goodness? Enter trv, a simple python ui that will let organize the test run in an easy to view manner, including screenshots. The tool is the creation of Thomi Richards who describes it as a quick hack (:p), and has a youtube video demonstrating it's use. It's perfect for viewing the subunit stream and visualizing your test results. For now, it's not packaged but can be easily obtained via launchpad. Grab it with bzr branch lp:trv.
autopilot3 vis
The vis tool allows you to visually interact with the introspection tree after launching an application using autopilot launch. What this means is you can visualize the application in the same way autopilot does at runtime, with live tree updates. It lets you see what autopilot sees, allowing you to interactively build your testcase.
I'll refer you the official tutorial for more information, as well as a youtube video by yours truly. It's from a livestream, but covers what you need to know. autopilot3 vis also contains a search box, and a highlight tool which didn't exist in the orignal version, so it's even nicer now than before. Give it a whirl!
autopilot3-sandbox-run
I talked about this utility when I covered the test runners for autopilot. Still I would be remiss if I didn't mention it again. Everything I said in the test runners for autopilot post still applies, so go have a quick read about how to use the tool if you need more information. This tool enables you to easily run autopilot tests on the desktop in a nested xserver. What that means to you as a test author is that you can run tests without giving up your desktop session. No more waiting for autopilot to hand back control of your mouse after a test. If you are writing tests, you should be using this tool along with autopilot vis mentioned above during your test writing process.
Per test timeout
Although we all only write "good tests", sometimes you may find your test misbehaves. When this happens the test may even not exit cleanly or get stuck in a loop. The result is autopilot and the system under test to wait forever for the test to finish. To prevent a rouge test from killing a test suite run, autopilot is introducing support for per-test timeouts. This has landed in vivid; you'll need version 1.5.0+15.04.20141031-0ubuntu1 or later. To use the feature, add the --test-timeout argument to autopilot run and give is a timeout in seconds.
In conclusion
Autopilot has gotten many new features along the way, and these are but a few of the most recent and important ones. I hope this helps you take another look at what autopilot might be able to help you test. Happy Testing!
No comments:
Post a Comment