Final Beta testing for Utopic

Can you believe final beta is here for utopic already? Where has the summer gone? The milestone and images are already prepared for the final beta testing. This is the first round of image testing for ubuntu this cycle. A final image will also be tested next month, but now is the time to try out the image on your system. Be sure to report any bugs you may find. This will help ensure there is time to fix them before the release images.

To help make sure the final utopic image is in good shape, we need your help and test results! Please, head over to the milestone on the isotracker, select your favorite flavor and perform the needed tests against the images.

If you've never submitted test results for the iso tracker, check out the handy links on top of the isotracker page detailing how to perform an image test, as well as a little about how the qatracker itself works. If you still aren't sure or get stuck, feel free to contact the qa community or myself for help. Happy Testing!

Autopilot Test Runners

In my last next post, I discussed will discuss notable autopilot features and talk about how autopilot has matured since it became an independent project.

In the meantime I would be remiss if I didn't also talk about the different test runners commonly used with autopilot tests. In addition to the autopilot binary which can be executed to run the tests, different tools have cropped up to make running tests easier.

autopilot-sandbox-run
This tool ships with autopilot itself and was developed as a way to run autopilot test suites on your desktop in a sane manner. Run the autopilot3-sandbox-run command with --help to see all the options available. By default, the tests will run in an Xvfb server, all completely behind the scenes with the results being reported to you upon completion. This is a great way to run tests with no interference on your desktop. If you are a visual person like me, you may instead wish to pass -X to enable the test runs to occur in a Xephyr window allowing you to see what's happening, but still retaining control of your mouse and keyboard.

I need this tool!
sudo apt-get install python3-autopilot

I want to run tests on my desktop without losing control of my mouse!
autopilot3-sandbox-run my_testsuite_name

I want to run tests on my desktop without losing control of my mouse, but I still want to see what's happening!
autopilot3-sandbox-run -X my_testsuite_name

Autopkgtest
Autopkgtest was developed as a means to automatically test Debian packages, "as-installed". Recently support was added to also test click packages and to run on phablet devices. Autopkgtest will take care of dependencies, setting up autopilot, and unlocking the device. You can literally plug in a device and wait for the results. You should really checkout the README pages, including those on running tests. That said, here's a quick primer on running tests using autopkgtest.

I need this tool!
sudo apt-get install autopkgtest
If you are on trusty, grab and install the utopic deb from here.

I want to run tests for a click package installed on my device!
Awesome. This one is simple. Connect the device and then run:
adt-run --click my.click.name --- ssh -s adb

For example,
adt-run --click com.ubuntu.music --- ssh -s adb

will run the tests for the installed version of the music app on your device. You don't need to do anything else. For the curious, this works by reading the manifest file all click packages have. Read more here.

I want to run the tests I wrote/modified against an installed click package!
For this you need to also pass your local folder containing the tests. You will also want to make sure you installed the new version of the click package if needed.

adt-run my-folder/ --click my.click.name --- ssh -s adb

Autopkgtest can also run in a lxc container, QEMU, a chroot, and other fun targets. In the examples above, I passed --- ssh -s adb as the target, instructing autopkgtest to use ssh and adb and thus run the tests on a connected phablet device. If you want to run autopilot tests on a phablet device, I recommend using autopkgtest as it handles everything for you.

phablet-test-run
This tool is part of the greater phablet-tools package. It was originally developed as an easy way to execute tests on your phablet device. Note however that copying the tests and any dependencies to the phablet device is left to you. The phablet-tools package provides some other useful utilities to help you with this (checkout phablet-click-test-setup for example).

I need this tool!
sudo apt-get install phablet-tools

I want to run the tests I wrote/modified against an installed click package!
First copy the tests to the device. You can use the ubuntu sdk or click-buddy for this, or even do it manually via adb. Then run phablet-test-run. It takes the same arguments as autopilot itself.

phablet-test-run -v my_testsuite

Note the tools looks for the testsuite and any dependencies of the testsuite inside the /home/phablet/autopilot folder. It's up to you to make sure everything that is needed to run your tests are located there or else it will fail.

other ways
There are of course other possible test runners that wrap around autopilot to make executing tests easier. Perhaps you've written a script yourself. Just remember at the end of the day the autopilot binary will be running the tests. It simply needs to be able to find the testsuite and all of it's dependencies in order to run. For this reason, don't be afraid to execute autopilot3 and run the tests yourself. Happy test runs!