A glance at Autopilot

So, as has been already mentioned, automated testing is going to come into focus this cycle. To that end, I'd like to talk about some of the tools and methods for automated testing that exist and are being utilized inside ubuntu.

I'm sure everyone has used unity at some point, and you will be happy to know that there is an automated testsuite for unity. Perhaps you've even heard the name autopilot. The unity team has built autopilot as a testing tool for unity. However, autopilot has broader applications beyond unity to help us do automated testing on a grander scale. So, to introduce you to the tool, let's check out a quick demo of autopilot in action shall we? Run the following command to install the packages needed (you'll need quantal or raring in order for this to work):

sudo apt-get install python-autopilot unity-autopilot

Excellent, let's check this out. A word of caution here, running autopilot tests on your default desktop will cause your computer to send mouse and keyboard commands all by itself ;-) So, before we go any further, let's hop over into a 'Guest Session'. You should be able to use the system indicator in the top right to select 'Guest Session'. Once you are there, you'll be in a new desktop session, so head back over to this page. Without further ado, open a terminal and type:

autopilot run unity.tests.test_showdesktop.ShowDesktopTests.test_showdesktop_hides_apps

This is a simple test to check and see if the "Show Desktop" button works. The test will spawn a couple of applications, click the show desktop button and verify clicking on it will hide your applications. It'll clean up after itself as well, so no worries. Neat eh?

You'll notice there's quite a few unity testcases, and you've installed them all on your machine now.

autopilot list unity

As of this writing, I get 461 tests returned. Feel free to try and run them. Pick one from the list and see what happens. For example,

autopilot run unity.tests.test_dash.DashRevealTests.test_alt_f4_close_dash

Just make sure you run them in a guest session -- I don't want anyone's default desktop to get hammered by the tests!

If you are feeling adventurous, you can actually run all the unity testcases like this (this will take a LONG TIME!).

autopilot run unity

As a sidenote, you are likely to find some of the testcases fail on your machine. The testsuite is run constantly by the unity developers, and the live results of commit by commit success or failure is actually available on jenkins. Check it out.

So in closing, this cycle we as a community have some goals surrounding easing the burden for ourselves in testing, freeing our resources and minds towards the deeper and more thorough testing that automation cannot handle. To help encourage this move of our basic testcases towards automation, the next series of blog posts will be a walkthrough on how to write Autopilot testcases. I hope to learn, explore and discover along with all of you. Autopilot tests themselves are written in python, but don't let that scare you off! If you are able to understand how to test, writing a testcase that autopilot can run is simply a matter of learning syntax -- non-programmers are welcome here!

UDS-R: Rise of the the (quality) machines

Greetings from Copenhagen! I thought I would give a mid-UDS checkup for the quality team community. You may have already heard some of the exciting stuff that is already been discussed at UDS. Automated testing is being pursued with full vigor, the release schedule has been changed, and cadence testing is in. In addition, ubuntu is being focused into getting into fighting shape by targeting the Nexus 7 as a reference platform for mobile.

I was honored enough to have a quick plenary where attendees here got to see and hear about the various automated testing efforts going on. Does that mean the machines have replaced us? Hardly! The goal with bringing automated testing online is to help us be more proactive with how and why we test. We've done an amazing job of reacting to changes and bugs, but now as a community I would like us to focus on being proactive with our testing. The changes below are all going to help set us firmly in this direction. By proactively testing things, we eliminate bugs, and repetitive or duplicated work for ourselves. This frees us to explore more focused, more interesting, and more in-depth testing. So without further ado, here's a quick rundown of the changes discussed here in Copenhagen -- hang on to your testing hats!

Release
The Release schedule has dropped all alphas, and the first beta, resulting in a beta and then final release milestone only. In addition, the freezes have been moved back a few weeks. The end result is the archive will not be frozen till late in the cycle, allowing development and testing to continue unencumbered. This of course is for ubuntu only. Which brings us to flavors!


Flavors
Flavors will now have complete control over there releases. They can chose to test, freeze, and re-spin according to there own schedule and timing. Some will adopt ubuntu's schedule, others may retain the old milestones or even do something completely different.


ISOs
Iso's will now be automatically 'smoke' tested before general release. No more completely broken installers on the published images! In addition, the iso's will be published daily as usual, but will not have typical milestones as mentioned above. Preference will be given to the daily iso -- the current one -- throughout the cycle. Testing will occur in a cadence instead of a milestone.

Cadence
Rather than milestones, a bi-weekly cadence of testing will occur with the goal of assuring good quality throughout the release cycle. The cadence weeks will be scheduled and feature testing different pieces of ubuntu in a more focused manner. This includes things like unity, the installer, and new features landing in ubuntu, but will also be the target of feedback from the state of ubuntu quality.

State of ubuntu Quality
A bold effort to generate a high level view of what needs testing and what is working well on a per image basis inside of ubuntu. This is an experimental idea whose implementation will garner feedback early in the cycle and will collect data and influence decisions for testing focus during the cycle. *fingers crossed*

AutoPilot
This tool will integrate xpresser to allow for a complete functional UI testing tool. One of the first focuses for testcases will be automating the installer from a UI perspective to free our manual testing resources from basic installer testing! From the community perspective, we can join in both the writing, and executing of automated, as well as the development of the tool itself.

Hardware Testing Database
This continuing experiment will become more of a reality. The primary focus of the work this cycle will be to bring the tool, HEXR, online and to do basic integration with the qatracker for linking your hardware profiles. In addition, focused hardware testing using the profiles will be explored.

I hope this gives you a nice preview of what's coming. I would encourage you to have a look a the blueprints and pads for the sessions, and ask questions or volunteer to help in places you are interested. I am excited about the opportunities to continue bringing testing to the next level inside of ubuntu. I  owe many thanks to the wonderful community that continues to grow around testing. Here's to a wonderful cycle.

Readying for UDS

I trust everyone is readying themselves -- don't blink! Ubuntu UDS-R is already upon us. Those of you who have been watching closely may have heard about some of the planned sessions for QA, but if not feel free to take a look. Don't worry, I'll wait.

But wait, there's more! In addition, there is going to be an evening event where testing is the focus. It's happening Tuesday evening. The goal is to learn about some of the testing efforts going on inside ubuntu, including automated testing; and more importantly, to write some testcases! Folks will be on hand to help talk you through and discuss writing both automated and manual test cases.

Looking through the tsessions, I hope you have the sense that testing is continuing to play a large role in ubuntu. And further, that you can be even more invovled! UI testing, automated testing, testcase writing -- all of these are focus points this cycle and have sessions. Get involved -- and if your at UDS, please do come to a session or two, add your voice, and grab some work items :-) Let's make it happen for next cycle.

Community Charity-a-thon: The Aftermath

I wanted to express my heartfelt thanks to everyone who contributed. To those who gave on behalf of the debian community, thank you as well! I stated that for every five donations I would do a manpage for a package that is missing one :-) I received just under 5 donations marked debian, but not to worry, I'll still create a manpage for one in need. Although I did other work during the marathon, I purposefully held onto creating the manpage until I was a bit more rested -- I have enough trouble speaking English sometimes without adding in sleep deprivation. The man page readers will thank me, and I'm sure those who get my page to review will as well.

To the rest of you, thank you very much. We raised $943.34 for WaterAid. That's amazing! I'm truly touched by your generosity. Here's the complete list of donors, hats off to all of you -- I know several of you donated anonymously, thank you!

Anonymous :-)
Cormac Wilcox
Gema
Anders Jonsson
Arthur Talpaert
Sam Hewitt
Alvaro
Ólavur Gaardlykke
Joey-Elijah Sneddon
steve burdine
Thomas Martin (tenach)
Daniel Marrable
sebsebseb Mageia
Jonas Grønås Drange
Gregor Herrmann
Mark Shuttleworth
phillw
Thijs K
Alvaro
Max Brustkern
Jane Silber
Gema Gomez-Solano
Martin Pitt
Michelle Hall

Now I know no one wants to re-watch that crazy 24 hours of video, but I wanted to bring you a few highlights as well. I spent time doing some of my normal work, but I also promised to do something outside the norm. I was able to scratch an itch, and although my on-air demo failed (an uh-duh moment), I was able to record this video immediately after demonstrating where we in QA are focusing next cycle. In addition, there were several talks from QA personnel, and I recommend watching this clip if your interested in hearing Rick's take on where ubuntu is going, and indeed how quality will play a role. You can skip to here if you only want to hear his take on quality. Now is a great time to be involved in QA -- I'm excited to see things unfold for 14.04, and I hope you are to.

For the readers who actually made it this far, I've got the best for last. There were some gags in those 24 hours; for instance, check out my chicken dance! (*cough* this was supposed to be a group thing *cough*). Ohh, and there's always this lovely screencap. To be fair, this was about 20 hours or so in.

Preparing for the Community Charity-a-thon

In preparation for the 24 hour marathon, I thought I would share with everyone my thoughts on my I chose my charity, and what I plan to do for 24 hours :-) To start off the post, let me get this right out front <insert flashing lights and whirlygigs> DONATE HERE </insert>

First the charity, WaterAid. I chose WaterAid upon realizing how important water is to me. I love water -- I love to stare out across a vast sea, or to sail along it using the wind, and hearing only the splashing of the waves, and the smell of life beneath it. And of course, I consume water each day in order to sustain life. I am happy to support an organization whose goal it is to provide sustainable water sources to everyone. Water is important to life, and is a basic need for us all as humans. We need clean water, and even moreso, we need sustainable access to it. Water is precious, and it's important for us to not pollute the water we have as well. WaterAid understands this need and works with locals to help create clean renewable water sources. Consider donating to help those who don't have access to the same resource we take for granted -- available anytime out of our faucet.

I'm also placing a call out to those who are interested in QA on both the Debian and ubuntu communities to participate. I plan to spend my time during the 24 hours doing something to further the work of how you interact with ubuntu and QA. So, to that end, I'd like to ask those of you who are interested in ubuntu to donate and install ubuntu during the marathon. I'll be here to provide technical installation support for you during the install. Let's see how many exotic configurations we can see successful installations on. Got a RAID setup or some exotic hardware? Multiple disks, partitions, and Os's? Get your systems backed up now, and let's try and install. NOTE, I'd encourage using the latest daily iso for installation, but you are welcome to also use beta2.

In addition for those of you in the Debian community, I am issuing a challenge for donations. For every 5 donations from the Debian community, I will write a missing man page from the list. I'll be focusing on things I use, but feel free to offer a suggestion during the marathon!

I would also issue a challenge to the greater ubuntu community. Do you have a problem that you are unable to solve within ubuntu? While I can't promise a fix for your issue, I will offer you my personal attention to help solve your problem. I'll help you file a bug, confirm it, or help you debug the problem to the best of my ability. I'll even offer my karma on askubuntu to your question ;-) If I get overwhelmed with donations, I'll pull the highest donators first -- but we do have 24 hours to fill! Note, I plan to do this work on Thursday, on-air, but you can donate in advance. Just leave me a note, or simply send me an email after your donation with your request if you donate in advance.

Finally, if none of the above suits you, I am happy to have a personal 1 on 1 match of highest level house of cards building, or another quick playing game of your choice on air. And don't worry, I'm ruthless competition, no pushover here!

Getting your bug fixed; the art of bug shepherding

We've all had this experience. Upgrade our hardware to the new version of ubuntu (or perhaps installing ubuntu for the first time on some new hardware), and everything works perfectly. We quickly dive in and enjoy the new experience of our favorite OS. Well, it's almost perfect. There's just this one bug that is bothering you.

Perhaps instead you are a tester. Someone who lives and breathes for breakage and the latest and greatest raw code packaged from the hands of it's developers. There's a new upgrade out and everything is perfect; well, except for that one bug.

So what do you do in these situations? Sadly some people may chose to do nothing but hope and wait. "Next cycle, or next release I'm hoping my bug will be fixed". Odds are, no one knows about your bug, and it can't be fixed if it's unknown. And if you are using proprietary software, "wait and see" is about the limit of your options. As a part of ubuntu however, we as a community can do so much more!

With that let me present my patented wizzbang approach to a successful resolution of your bug within ubuntu!

  
First, let me clarify what a successful resolution is. If you have been around ubuntu, you may have seen a bug that expired as 'incomplete'. This is clearly not a successful resolution. In my eyes, a successful resolution to a bug sees the status changed to a 'won't fix, fix committed, triaged, etc'.

Ok, so here's the steps:

1. If you don’t know how to file a good bug, ask first! It's important to do your best to describe the problem you are experiencing, and if possible how to repeat the bug. Check out the post on askubuntu which has a nice summary of resources available to help you.
2. File a good bug, using your newly formed knowledge from above :-)
3. Get someone else to confirm it. This is important! If possible, have a friend confirm the bug on their system. Once they've confirmed it, have them mark the bug as affecting them as well on launchpad.
4. Answer questions promptly when asked by others. Make sure you are getting emails from launchpad, and when someone asks a question on your bug, respond promptly.
5. Get your bug triaged. If your bug is confirmed and filed correctly, the bug triagers should help triage the bug. If a long time has passed without this occurring, check to make sure you bug is in good order to be triaged. If so, asking politely for a triager to look at your bug on the #ubuntu-bugs channel is a good way to keep your bug moving.
6. Help debug, test, and confirm and fixes that are made for your issue. If the developer spends time working on your bug, do what you can to help confirm it fixes your issue.
7. Remember no one will care about your bug as much as you do! When you file a bug, commit to carrying it as far along in the process as you can.

That's it! There's no guarantee every bug you file now will receive your desired outcome, but you should see proper resolution, instead of your bugs expiring. By being a good participant you are ensuring you can feel good about the resolution of the bugs you file. Remember we are all human, and sometimes things get missed. Stick with your bug, and shepherd it through.

So is the process perfect? Not at all. We as a community still need to think more about improving our experience in dealing with problems. Not every "problem" encountered is a bug, and a process to better handle these problems is still worthy of thought. I invite those of you interested in this to look for a UDS session on the topic.

Special thanks to TheLordofTime and hggdh for their discussions surrounding bugs, and of course for our marvelous bugsquad without whom this would not be possible!

Global Jam with QA love

Global Jam is a unique time for folks to get together across the world and celebrate ubuntu. As part of the celebrations, there is an opportunity to download the latest beta (released yesterday!) and check out the next version of ubuntu. You can run it in a livecd or perhaps in a virtual machine. Ether way, there's opportunities for you to test the common applications and report your results.

The testcases are available here: Ubuntu Global Jam Testcases. For some of you, this page may seem a bit funny, but fortunately there is a handy walk-through for you to understand how to use the tracker and specifically, how to report results against these tests. Check out the guides below. If you follow the link, you'll even find some video of yours truly giving you a visual demonstration.

https://wiki.ubuntu.com/Testing/QATracker
https://wiki.ubuntu.com/Testing/CallforTesting/Walkthrough

If you get stuck, remember your friends in #ubuntu-testing on freenode are also happy to help. Have fun jamming, and if you do test, Happy Testing! Most of all, celebrate ubuntu!