A vision for our testing future

As I eluded to in my previous post, it's time for us as a community to fully embrace the place of automated testing in our workflows. Over the past year we've learned how to write testcases and to then apply that knowledge towards writing autopilot tests. At the same time ubuntu has been building testing infrastructure and launching a CI team to help run and maintain it.

With these changes and our acquired knowledge we can construct a sustainable vision for testing. So just what is that vision?

Let's make manual testing fun again. As we've ramped up our workloads, we find ourselves repetitively testing the same packages again and again (and running the same tests!). We'd call this regression testing and it's a perfect candidate for automated testing. Let's automate these tests and keep our focus on where we as humans excel.

In addition, we as a community participate in "calls for testing". These calls have become more and more exploratory in nature. In general, we have moved further away from a defined testcase ("perform steps 1, 2, and 3"). Instead, we encourage testers to adopt a try and break it attitude. This is where we as humans can excel, and you know what, have some fun at trying! Remember QA is the only place we encourage you to break things!

So let's get manual testing back to the exploratory testing we love.

Thank you little testing robot!

Let's expand our automated tests. We can increase testing coverage forever with a one time investment of writing a testcase. Developers, write tests for your code and feature sets as you develop them. As a quality community, let's do our best to make it easy for developers to do so.

In addition, a well written bug report can become an excellent testcase to be added to the application testuite. Let's look at bug reports for potential testcases. We can write the test and then forget it. The little robots will tell us if it becomes a problem again in the future and even prevent the bad code from getting shipped where it can break our machines (again). Not bad little robots!

So let's focus on ensuring tests exist for critical regressions when they get fixed.

Let's tackle some big projects. Who said we can't achieve amazing things in the quality community? This vision would be nothing more than an idea without some actions behind it!

Right now we as a community already have a chance to put these ideas in action. Exploratory testing is happening right now on the phablet devices. Get involved! This is the manual, have fun and try and break it, exploratory testing we all love. Even without a phablet device, regressions can be turned into autopilot tests. This meets our goal to expand our automated tests and to look at bugs as potential sources for those tests.

All test and no play makes for a sad image

Moving forward there are still a some big projects to tackle. One of the largest is the amount of manual testing effort required in cadence and milestone testing. We can bring automated testing to the mix to both reduce our on-going effort and raise the quality bar in our releases.

So let's be the leaders for change in ubuntu.

Are you in?

Spreading the (testing) weight

With all the buzz and excitement around quality over the last few cycles, we've been busy. We've built tools, started processes, wrote testcases and committed ourselves to testing. When the time came for us to write new applications for the ubuntu touch platform, we took our quality aspirations with us and tested all through development. The "core apps" for the phone is a perfect example of this; community developed software including tests. Kudos to everyone who is taking part in these efforts.

However, as more and more tests came online, new problems developed. How can we run these tests? When do we want to run them? How can we develop more? What tools do we use? The quality team met all of these challenges, but the weight started to grow heavier.

Rob Macklem Victoria BC derivative: Plasticspork [CC-BY-SA-3.0]

Something needed to be done. Before the bar breaks or our arms give out, let's split the weight. And thus a new team was formed.

Ivanko Barbell Company

Enter the CI team. Behind the scenes running tests, getting our infrastructure in place, and chasing down regressions, the CI team has their hands full. In a nutshell they will ensure all the tests are run as needed. If a test fails they will chase down and ensure the right folks are notified so fixes can be made. Ohh, and yes, that means potential bugs will never even hit your system dear user.

The QA team's mission then gets to be refocused. They will continue to write tests, develop tools if needed, and continue to push for greater quality in ubuntu and upstream. However they no longer have to balance these tasks with ensuring builds stay green, or writing a dashboard to review results. The result is a better focus and the opportunity to do more.

So how does this affect us as a community? The simple answer is that we too can take advantage of this new testing infrastructure and CI team. Let's focus on what coverage we need and what the best way to achieve it might be. We'll also be able to align our goals and ambitions even more with the QA team. I'll share more of my thoughts on this in the next post.

Autopilot Sandboxing

Autopilot has continued to change with the times, and the pending release of 1.4 brings even more goodies; including some performance fixes. But today I wanted to cover a newly landed feature from the minds of Martin and Jean-Baptiste (thanks guys!).

If you've developed autopilot tests in the past you will have noticed how cumbersome it can be to run the tests. If you run a test on your desktop, you lose control of your mouse and keyboard for the duration, and you might even accidentally cause a test to fail. This can be especially noticeable when you are iterating over getting your test to run "just right" while wanting to keep your introspection tree in vis open, or reviewing someone else's code while wanting to verify the tests run. Enter sandbox mode.

A new command called autopilot-sandbox-run lets you easily run a testsuite inside your choice of two sandboxes; Xvfb by default, or if you want to visually see the output, Xephyr. Have a quick look at the command options below as of this writing:

Usage: autopilot-sandbox-run [OPTIONS...] TEST [TEST...]
Runs autopilot tests in a 'fake' Xserver with Xvfb or Xephyr. autopilot runs
in Xvfb by default.
   
    TEST: autopilot tests to run

Options:
    -h, --help           This help
    -d, --debug          Enable debug mode
    -a, --autopilot ARG  Pass arguments ARG to 'autopilot run'
    -X, --xephyr         Run in nested mode with Xephyr
    -s, --screen WxHxD   Sets screen width, height, and depth to W, H, and D respectively (default: 1024x768x24)

The next time you want to get your hands dirty with some autopilot tests, try out the sandbox. I'm sure you'll find a very nice use for it in your workflow; after all wouldn't it be handy to run multiple testsuites at once?

Testing Ubuntu Touch: The final month before release

As of today, we are exactly one month away from the release of Saucy Salamander. As part of that release, ubuntu is committed to delivering an image of ubuntu-touch, ready to install on supported devices.

And while folks have been dogfooding the images since May, many changes have continued to land as the images mature. As such, the qa team is committing to test each of the stable images released, and do exploratory testing against new features and specific packagesets.

If you have a device, I would encourage you to join this effort! Everything you need to know can be found upon this wiki page. You'll need a nexus device and a little time to spend with the latest image. If you find a bug, report it! The wiki has links to help. Testing doesn't get anymore fun than this; flash your phone and try to break it! Go wild!

And if you don't own a device? You can still help! As bugs are found and fixed, the second part of the process is to create automated tests for them so they don't occur again. Any bug you see on the list is a potential candidate, but we'll be marking those we especially think would be useful to write an autopilot tests for with a " touch-needs-autopilot" tag.

Join us in testing, confirming bugs, or testwriting autopilot tests. We want the ubuntu touch images to be the best they can be in 1 month's time. Happy Testing!

Jamming Quality Style

It's that time of year again! Time to get your jam on (I like mine on a bagel).

  

While you are making plans for Ubuntu Global Jam, don't forget you can contribute to quality as well. There's a separate subpage of the global jam wiki dedicated to it.

We love new test contributions, and there's a collection of wiki tutorials and videos to help you contribute them. You don't have to be technical to write tests -- we need manual testcases also which are written in plain English :-)

More interested in submitting your results for tests? We've also got you covered. We have tests for the default applications of ubuntu as well as the images of ubuntu. Download an image and run it on your machine. Try running through some default testcases for ubuntu or your favorite flavor. An image and a pc or laptop is all you need to get started. Happy Jamming!