Keeping ubuntu healthy: Manual Image Testing

Continuing our discussion of the testing within ubuntu, today's post will talk about how you can help ubuntu stay healthy by manually testing the images produced. No amount of robots or automated testing in the world can replace you (well, at least not yet, heh), and more specifically your workflow and usage patterns.

As discussed, everyday new images are produced for ubuntu for all supported architecture types. I would encourage you to follow along and watch the progression of the OS through these images and your testing. Every data point matters and testing on a regular basis is helpful. So how to get started?

Settle in with a nice cup of tea while testing!

The Desktop
For the desktop images everything you need is on the image tracker. There is a wonderful video and text tutorial for helping you get started. You report your results on the tracker itself in a simple web form, so you'll need a launchpad account if you don't have one.

The secondary way to help keep the desktop images in good shape is to install and run the development version of ubuntu on your machine. Each day you can check for updates and update your machine to stay in sync. Use your pc as you normally would, and when you find a bug, report it! Bugs found before the release are much easier to fix than after release.

Phablet
Now for the phablet images you will need a device capable of running the image. Check out the list. Grab your device and go through the installation process as described on the wiki. Make sure to select the '-proposed' channel when you install so you will see updates to get the latest images being worked on every time they are built. From there you can update everyday. Use the device and when you find a bug, report it! Here's a wiki page to help guide your testing and help you understand how and where to report bugs.

Don't forget there's a whole team of people within ubuntu dedicated to testing just like you. And they would love to have you join them!

A simple look at testing within ubuntu

Since just before the last LTS, quality has been a buzzword within the ubuntu community. We've come a long way since precise and I wanted to provide some help and prospective on what ubuntu's process for quality looks like this cycle. In simple terms. Or as reddit would say, "explain to me like I'm 5".

I'll try and define terms as we go. First let me define CI, which is perhaps the buzzword of this cycle, lest I lose all of you! CI stands for continuous integration, and it means we are testing ubuntu. All the time. Non-stop. Every change we make, we test. The goal behind this idea is to find and fix problems, before well, they become problems on your device!

CI Dashboard
The CI dashboard then is a way to visually see the results of this testing. It acts as a representation of the health of ubuntu as a distribution. At least once a day runs are executed, apps and images are tested and benchmarked, and the results are populated on ci.ubuntu.com. This is perhaps the most visible part of the CI (our continuous testing efforts) that is happening within ubuntu. But let's step back a minute and look at how the overall CI process works within ubuntu.

CI Process
App developers hack on a bit of code, fixing bugs or adding new features to the codebase. Once the code is ready, a merge proposal₁ is created by the developer and feedback is sought. If the code passes the peer review and the application's tests, it will then become part of the image after a journey through the CI train.

For the community core apps, the code is merged after peer review, and then undergoes a similar journey to the store where it will become part of the image as well. Provided of course it meets further review criteria by myself and Alan (we'll just call him the gatekeeper).

Though menacing, Alan assures me he doesn't bite

Lest we forget, upstream₂ uploads₃ are done as well. We can hope some form of testing was done on them before we received them. Nevertheless, tests are run on these as well, and if they pass successfully, the new packages will enter the archive₄ and become part of the image.

Generating Images
Now it's time to generate some images. For the desktop a snapshot of what's in the ubuntu archive is taken each day, built, and then subjected to a series of installation tests. If the tests pass, it is released for general testing called Image (or ISO) testing. An image is tested and declared stable as part of a milestone (or testing event) and can become the next version of ubuntu!

Adopted images are healthy images!

On the ubuntu phone side of things, all the new uploads are gathered and evaluated for risk. If something is a large change, it might be prudent to not land it with other large changes so we can tell what broke should the image not work properly. Once everything is ready, a new image is created and is released for testing.  The OTA updates (over-the-air; system updates) on your ubuntu phone come from this process!

How you can help?
Still with me I hope? As you can see there's many things happening each day in regards to quality and lots of places where you can create a positive change for the health of the distro! In my next few posts, I'll cover each of the places you can plug in to help ubuntu be healthy everyday!

1. A merge proposal is a means of changing an applications code via peer review.
2. By upstream, I mean the communities and people who make things we use inside of ubuntu, but are not directly a part of it. Something like the browser (firefox) and kernel are good examples.
3. This can happen via a general sync at the beginning of the cycle from debian. This sync copies part of the debian archive into the ubuntu archive, which in effect causes applications to be updated. Applications are also updated whenever a core ubuntu developer or a MOTU uploads a new version to the archive. 
4. In case you are wondering, "the archive" is the big repository where all of your updates and new applications come from!

Click-buddy and you: how to test your ubuntu sdk application

Recently the ubuntu core app developers and myself have been on an adventure towards adopting cmake for the all the core applications. While some of the applications are pure qml it's still been useful to embark on adopting a singular build system for all of the projects. Now that (most) of the pain of transitioning is gone, I'm going to talk about one of the useful features of setting up cmake for your project; click-buddy!

Click-buddy is an evolving tool that helps you build and deploy click packages to your phablet device. In addition, it has the ability to setup the device to run your autopilot test suite. So, rather than writing anything further, let's cover an example. You are going to need phablet-tools installed for this to work. I'm going to branch the clock app, build the click package, install it on my device, and finally run the tests.

bzr branch lp:ubuntu-clock-app
cd ubuntu-clock-app
click-buddy --dir . --provision
phablet-test-run -v ubuntu_clock_app

Click-buddy is also gaining the ability to build your project, even it involves a plugin and you are interested in building for your phablet device (armhf). Once landed you will be able to run something like this for non-qml applications.

sudo click chroot -a armhf create
click-buddy --arch armhf --provision

This will setup a chroot automagically for you and compile and build your application. Give it a try!

Note, as of this writing, emulator support for the ubuntu-ui-toolkit emulator is not yet built in. If your tests fail with a module import, run this line from your connected pc. It will copy over the ubuntu-ui-toolkit emulator (provided you have it installed on your pc :-) ) and your tests should now properly run.

adb push /usr/lib/python2.7/dist-packages/ubuntuuitoolkit /home/phablet/autopilot/ubuntuuitoolkit

Trusty Cycle Updates

First things first, we have our burndown up for the trusty cycle! If your name appears on the list next to a work item, congratulations! Let's work through the items and make it happen ;-) If your name doesn't appear, don't worry! Feel free to help complete the tasks anyway, or jump in an get started on something you love. The mailing list is always open and we love to hear new ideas. See a blueprint that looks interesting and want to help? Get in touch!

What's happening now?
Now, for an update as to what is going on in quality. This week starts Alpha 1 for the flavors that are participating. In addition, the first work items are starting to see completion.

Ali has been working on getting the wiki pages and workflow ready for our community calls for testing. You can check out the page on the wiki here, and schedule testing events. The page explains the basic workflow -- I'd encourage you to schedule events for your favorite packages and let others know about the event on the mailing list. Be sure and report the results once you've completed.

Alberto has been working on getting the papercuts project running for the trusty cycle and ensuring the buglist is maintained and accessible.

Dan has been working on getting our automated testing in place and we're meeting with the CI team this week to get this implemented as part of the release process. Excellent work Dan!

Dan has also been working on the autopilot gtk emulator, which seeks to do the same thing as we've done with the autopilot ubuntu sdk emulator. Both of these frameworks help make writing tests easier.

Speaking of tests, Carla and others have been busy adding and fixing tests to make sure our dashboard stays green. And look at it! We're over 99% passing tests for the ubuntu touch platform. I want to personally thank everyone who's helped write and maintain these tests. Kudos to you all!

On the desktop side, Dan is helping new folks land tests, and the ubuntu desktop default apps continue to run and provide feedback on the desktop.

We also saw several new members join the ubuntu community via ubuntu quality recently. Congrats to Jean-Baptiste, Dan Chapman and Daniel Kessel (my apologies if I missed you!). 

Finally we have Thibaut and other folks looking at refactoring and improving ubuntu startup disk creator. Checkout the full blueprint for more information.

So, what's next?
Well for most of us, it's the Holiday season. Looking beyond the Holidays, , there's several big items in my list to work on in January:

1. Continue to keep the dashboard green all the time
2. Do wiki, mailing list, launchpad integration with bugsquad
3. Add wiki documentation and videos for exploratory testing

What am I doing?
Well, you dear reader, I trust is getting involved in some of the work items and other activities. It starts with installing the development version of ubuntu and then diving in from there. Perhaps you want to write some tests, triage or fix some bugs, or schedule a testing event. If you are interested in making a contribution to ubuntu, we can always use good testers, test writers, developers and triagers!

If you are celebrating holidays this time of year, enjoy your time!  Thanks everyone and Happy Testing!

Quality and the community in the Trusty Cycle

Since I last posted on the topic of our communities future, we've helped push out a new release of ubuntu. Saucy and fresh, the little salamander is now out in the wild. With the release out it's time to move forward with these changes.

(c) Carla Sella

The beginning of a new cycle is always a time to breathe for us in quality and these past few days of relative calm has indeed been a welcome respite from the craziness of testing leading up to a release.

We can rebuild it. We have the technology. Better than it was before. Better, stronger, faster.

So, let's talk about some of the changes coming to quality for this new cycle.

Defined Roles
I want to help our new members become a productive part of ubuntu quality as soon as possible. To this end I've created a list of roles for the quality team. By defining roles within the team it is easy to see how you can contribute as a 'tester', 'test writer', or 'developer'.

Contribute anytime
While release milestones and calls for testing will still be important, contributing to ubuntu quality can be a daily task. There are activities you can perform any day and anytime in the cycle. The roles pages list activities for each role that can be done right now. If you are a tester, check out the activities you can do right now to help!

More exploratory testing
We're iterating faster and faster. Builds of new code are landing each day, and in some cases several times a day. We can't afford to only test every other week with cadence testing. Instead, we've ramped up efforts to automatically test on each of these new builds. But we still need manual testing!

As a tester you can provide testing results for images, packages and hardware at any time! In addition, exploratory testing is highly encouraged. This is were we as manual testers shine! I want to encourage ongoing exploratory testing all throughout the cycle. Run and use the development version of ubuntu on your machine all the time!

Tackling some big projects
One of the things I wanted to push us as a team towards was tackling some projects that have a wider impact than just us. To that end, you can see several big projects on the activities pages for each role.

For testers, we are undertaking making reporting issues better for users. For the test writers, one of the largest projects is spearheading the effort to make manual image testing less burdensome and more automated by automating ubiquity testing. And for developers, the autopilot emulators are big projects as well and need help.

More involvement with bugs
As a quality community with interact with many bugs. Sometimes we are finding the bug, other times we might confirm them or verify a fix works. However, we haven't always gone the extra step towards doing work with SRU verifications and bug triage. The bugsquad and others have traditionally performed these tasks. If you'll notice these activities are now encouraged for those in the tester role. Let's dive more into the bug world.

A potential expansion of the team
With the mention of the bugsquad and the encouraged involvement with bugs for our team, I would like to propose a union between the quality and bugsquad teams. I would encourage current bugsquad members to take up tester roles, and consider some of the additional opportunities that are available. For those who have been testing with the quality team in past cycles, let's get more invovled with bugs and traditional bugsquad activities as mentioned above.

Making a quality LTS
Trusty Tahr is going to be the next LTS release. Those of you who remember and use Precise will agree that it is going to be a tough act to follow. The bar is set high, but I am confident we can reach it and do better.

We can't do this alone. We need testers, test writers, and developers! If you are interested in helping us achieve our goal, please join us! Now is an excellent time to learn and grow with the rest of us on the team. Thanks for helping making ubuntu better!

Testers, assemble! Final RC testing is here!

(c) http://www.flickr.com/photos/lalalaurie/

Here are 3 easy steps to testing prowess. Do your part to be prepared! Let's help make saucy a great release!

1) Review both the critical bugs and those found during isotesting throughout the cycle. This is handy in case you find something during testing and want to know if someone has already reported it or not.

2) Make sure you know how to use the tracker and how to test an iso. When the milestone is created and the images appear on October 10th, you want to be ready to go!

3) Test and report your results to the tracker ;-) Note, at this point in the cycle we prefer real hardware results. Upgrade your machine! Try a fresh install. If you've been sitting on raring, now is the time to migrate to saucy and let us know of any bugs you find.

Happy Testing!

Dancing, testing, and releasing

We're still here and we're still testing!

And you can still join us!

Grab your nexus device, check out the wiki page, and flash away. File bugs, and keep ontop of the updates coming out. Use the device as you would any other phone. Ohh and check out the app development contest winners by installing the apps from the click store on the phone (check out the more suggestions section).

The release is only 2 weeks away. That's 14 days or a mere 336 hours. And we'll be spending 112 hours sleeping (perhaps less, get some sleep!). That leaves just 224 hours for testing. Join us!

#UbuntuTestDanceMaster out.