Quality and the community in the Trusty Cycle

Since I last posted on the topic of our communities future, we've helped push out a new release of ubuntu. Saucy and fresh, the little salamander is now out in the wild. With the release out it's time to move forward with these changes.

(c) Carla Sella

The beginning of a new cycle is always a time to breathe for us in quality and these past few days of relative calm has indeed been a welcome respite from the craziness of testing leading up to a release.

We can rebuild it. We have the technology. Better than it was before. Better, stronger, faster.

So, let's talk about some of the changes coming to quality for this new cycle.

Defined Roles
I want to help our new members become a productive part of ubuntu quality as soon as possible. To this end I've created a list of roles for the quality team. By defining roles within the team it is easy to see how you can contribute as a 'tester', 'test writer', or 'developer'.

Contribute anytime
While release milestones and calls for testing will still be important, contributing to ubuntu quality can be a daily task. There are activities you can perform any day and anytime in the cycle. The roles pages list activities for each role that can be done right now. If you are a tester, check out the activities you can do right now to help!

More exploratory testing
We're iterating faster and faster. Builds of new code are landing each day, and in some cases several times a day. We can't afford to only test every other week with cadence testing. Instead, we've ramped up efforts to automatically test on each of these new builds. But we still need manual testing!

As a tester you can provide testing results for images, packages and hardware at any time! In addition, exploratory testing is highly encouraged. This is were we as manual testers shine! I want to encourage ongoing exploratory testing all throughout the cycle. Run and use the development version of ubuntu on your machine all the time!

Tackling some big projects
One of the things I wanted to push us as a team towards was tackling some projects that have a wider impact than just us. To that end, you can see several big projects on the activities pages for each role.

For testers, we are undertaking making reporting issues better for users. For the test writers, one of the largest projects is spearheading the effort to make manual image testing less burdensome and more automated by automating ubiquity testing. And for developers, the autopilot emulators are big projects as well and need help.

More involvement with bugs
As a quality community with interact with many bugs. Sometimes we are finding the bug, other times we might confirm them or verify a fix works. However, we haven't always gone the extra step towards doing work with SRU verifications and bug triage. The bugsquad and others have traditionally performed these tasks. If you'll notice these activities are now encouraged for those in the tester role. Let's dive more into the bug world.

A potential expansion of the team
With the mention of the bugsquad and the encouraged involvement with bugs for our team, I would like to propose a union between the quality and bugsquad teams. I would encourage current bugsquad members to take up tester roles, and consider some of the additional opportunities that are available. For those who have been testing with the quality team in past cycles, let's get more invovled with bugs and traditional bugsquad activities as mentioned above.

Making a quality LTS
Trusty Tahr is going to be the next LTS release. Those of you who remember and use Precise will agree that it is going to be a tough act to follow. The bar is set high, but I am confident we can reach it and do better.

We can't do this alone. We need testers, test writers, and developers! If you are interested in helping us achieve our goal, please join us! Now is an excellent time to learn and grow with the rest of us on the team. Thanks for helping making ubuntu better!

Testers, assemble! Final RC testing is here!

(c) http://www.flickr.com/photos/lalalaurie/

Here are 3 easy steps to testing prowess. Do your part to be prepared! Let's help make saucy a great release!

1) Review both the critical bugs and those found during isotesting throughout the cycle. This is handy in case you find something during testing and want to know if someone has already reported it or not.

2) Make sure you know how to use the tracker and how to test an iso. When the milestone is created and the images appear on October 10th, you want to be ready to go!

3) Test and report your results to the tracker ;-) Note, at this point in the cycle we prefer real hardware results. Upgrade your machine! Try a fresh install. If you've been sitting on raring, now is the time to migrate to saucy and let us know of any bugs you find.

Happy Testing!

Dancing, testing, and releasing

We're still here and we're still testing!

And you can still join us!

Grab your nexus device, check out the wiki page, and flash away. File bugs, and keep ontop of the updates coming out. Use the device as you would any other phone. Ohh and check out the app development contest winners by installing the apps from the click store on the phone (check out the more suggestions section).

The release is only 2 weeks away. That's 14 days or a mere 336 hours. And we'll be spending 112 hours sleeping (perhaps less, get some sleep!). That leaves just 224 hours for testing. Join us!

#UbuntuTestDanceMaster out.

A vision for our testing future

As I eluded to in my previous post, it's time for us as a community to fully embrace the place of automated testing in our workflows. Over the past year we've learned how to write testcases and to then apply that knowledge towards writing autopilot tests. At the same time ubuntu has been building testing infrastructure and launching a CI team to help run and maintain it.

With these changes and our acquired knowledge we can construct a sustainable vision for testing. So just what is that vision?

Let's make manual testing fun again. As we've ramped up our workloads, we find ourselves repetitively testing the same packages again and again (and running the same tests!). We'd call this regression testing and it's a perfect candidate for automated testing. Let's automate these tests and keep our focus on where we as humans excel.

In addition, we as a community participate in "calls for testing". These calls have become more and more exploratory in nature. In general, we have moved further away from a defined testcase ("perform steps 1, 2, and 3"). Instead, we encourage testers to adopt a try and break it attitude. This is where we as humans can excel, and you know what, have some fun at trying! Remember QA is the only place we encourage you to break things!

So let's get manual testing back to the exploratory testing we love.

Thank you little testing robot!

Let's expand our automated tests. We can increase testing coverage forever with a one time investment of writing a testcase. Developers, write tests for your code and feature sets as you develop them. As a quality community, let's do our best to make it easy for developers to do so.

In addition, a well written bug report can become an excellent testcase to be added to the application testuite. Let's look at bug reports for potential testcases. We can write the test and then forget it. The little robots will tell us if it becomes a problem again in the future and even prevent the bad code from getting shipped where it can break our machines (again). Not bad little robots!

So let's focus on ensuring tests exist for critical regressions when they get fixed.

Let's tackle some big projects. Who said we can't achieve amazing things in the quality community? This vision would be nothing more than an idea without some actions behind it!

Right now we as a community already have a chance to put these ideas in action. Exploratory testing is happening right now on the phablet devices. Get involved! This is the manual, have fun and try and break it, exploratory testing we all love. Even without a phablet device, regressions can be turned into autopilot tests. This meets our goal to expand our automated tests and to look at bugs as potential sources for those tests.

All test and no play makes for a sad image

Moving forward there are still a some big projects to tackle. One of the largest is the amount of manual testing effort required in cadence and milestone testing. We can bring automated testing to the mix to both reduce our on-going effort and raise the quality bar in our releases.

So let's be the leaders for change in ubuntu.

Are you in?

Spreading the (testing) weight

With all the buzz and excitement around quality over the last few cycles, we've been busy. We've built tools, started processes, wrote testcases and committed ourselves to testing. When the time came for us to write new applications for the ubuntu touch platform, we took our quality aspirations with us and tested all through development. The "core apps" for the phone is a perfect example of this; community developed software including tests. Kudos to everyone who is taking part in these efforts.

However, as more and more tests came online, new problems developed. How can we run these tests? When do we want to run them? How can we develop more? What tools do we use? The quality team met all of these challenges, but the weight started to grow heavier.

Rob Macklem Victoria BC derivative: Plasticspork [CC-BY-SA-3.0]

Something needed to be done. Before the bar breaks or our arms give out, let's split the weight. And thus a new team was formed.

Ivanko Barbell Company

Enter the CI team. Behind the scenes running tests, getting our infrastructure in place, and chasing down regressions, the CI team has their hands full. In a nutshell they will ensure all the tests are run as needed. If a test fails they will chase down and ensure the right folks are notified so fixes can be made. Ohh, and yes, that means potential bugs will never even hit your system dear user.

The QA team's mission then gets to be refocused. They will continue to write tests, develop tools if needed, and continue to push for greater quality in ubuntu and upstream. However they no longer have to balance these tasks with ensuring builds stay green, or writing a dashboard to review results. The result is a better focus and the opportunity to do more.

So how does this affect us as a community? The simple answer is that we too can take advantage of this new testing infrastructure and CI team. Let's focus on what coverage we need and what the best way to achieve it might be. We'll also be able to align our goals and ambitions even more with the QA team. I'll share more of my thoughts on this in the next post.

Autopilot Sandboxing

Autopilot has continued to change with the times, and the pending release of 1.4 brings even more goodies; including some performance fixes. But today I wanted to cover a newly landed feature from the minds of Martin and Jean-Baptiste (thanks guys!).

If you've developed autopilot tests in the past you will have noticed how cumbersome it can be to run the tests. If you run a test on your desktop, you lose control of your mouse and keyboard for the duration, and you might even accidentally cause a test to fail. This can be especially noticeable when you are iterating over getting your test to run "just right" while wanting to keep your introspection tree in vis open, or reviewing someone else's code while wanting to verify the tests run. Enter sandbox mode.

A new command called autopilot-sandbox-run lets you easily run a testsuite inside your choice of two sandboxes; Xvfb by default, or if you want to visually see the output, Xephyr. Have a quick look at the command options below as of this writing:

Usage: autopilot-sandbox-run [OPTIONS...] TEST [TEST...]
Runs autopilot tests in a 'fake' Xserver with Xvfb or Xephyr. autopilot runs
in Xvfb by default.
   
    TEST: autopilot tests to run

Options:
    -h, --help           This help
    -d, --debug          Enable debug mode
    -a, --autopilot ARG  Pass arguments ARG to 'autopilot run'
    -X, --xephyr         Run in nested mode with Xephyr
    -s, --screen WxHxD   Sets screen width, height, and depth to W, H, and D respectively (default: 1024x768x24)

The next time you want to get your hands dirty with some autopilot tests, try out the sandbox. I'm sure you'll find a very nice use for it in your workflow; after all wouldn't it be handy to run multiple testsuites at once?

Testing Ubuntu Touch: The final month before release

As of today, we are exactly one month away from the release of Saucy Salamander. As part of that release, ubuntu is committed to delivering an image of ubuntu-touch, ready to install on supported devices.

And while folks have been dogfooding the images since May, many changes have continued to land as the images mature. As such, the qa team is committing to test each of the stable images released, and do exploratory testing against new features and specific packagesets.

If you have a device, I would encourage you to join this effort! Everything you need to know can be found upon this wiki page. You'll need a nexus device and a little time to spend with the latest image. If you find a bug, report it! The wiki has links to help. Testing doesn't get anymore fun than this; flash your phone and try to break it! Go wild!

And if you don't own a device? You can still help! As bugs are found and fixed, the second part of the process is to create automated tests for them so they don't occur again. Any bug you see on the list is a potential candidate, but we'll be marking those we especially think would be useful to write an autopilot tests for with a " touch-needs-autopilot" tag.

Join us in testing, confirming bugs, or testwriting autopilot tests. We want the ubuntu touch images to be the best they can be in 1 month's time. Happy Testing!