Quality mid-cycle checkup

About 2.5 months ago I wrote about the plans for the ubuntu QA community for the quantal cycle. We were building off of lots of buzz from the precise release and we planned to undertake lots of new work, while being very careful to avoid burnout. Our focus was to take QA to the next level and help us communicate and grow as a team to take on the opportunities we have.

So, how are we doing? Let's go over each of the points noted in the original post and talk about the progress and plans.

ISOTesting
Our alpha1 testing went very well, but the alpha2 and alpha3 have seen less participation. In addition we were asked and responded to a plan to test our iso's every 2 weeks as part of a more cadenced testing. Overall, isotesting continues to be a weak spot for us as a community. ISO Testing is perhaps the most important piece of testing for us as a greater ubuntu community. The image we produce is literally the first experience many folks have with ubuntu. If it fails to install, well, there went our chance for a positive first impression :-( I would be happy to hear ideas or comments on isotesting in particular.

Application Testing
This work has been mostly completed. The package tracker now allows us to perform work that was done via checkbox or manual testing last cycle. We can now manage results, tests and reporting all in one tool -- and it's all publicly available. For more information about the qatracker, see this wiki page.

SRU Verification
This work is still on paper, awaiting for the 12.04.1 release before further discussions and work will begin.

General Testing (eg, Day to Day running of the development version)
I am still experimenting with understanding how to enable better reporting and more focused testing on this. The current plan is to track specific packages that are critical to the desktop, and allow those run the development version the ability to report how the application is working for each specific upload during the development release. This is done with the qatracker. I'll blog more about this and the results in a later post. Contact me as always if your interested in helping.

Calls for Testing
This has been a wonderful success. There have been several calls for testing and the response has been wonderful. A big thank you to all of you who have helped test this. We've had over 50 people invovled in testing, and 41 bugs reported. Myself and the development teams thank you! But we're not done yet, unity testing among other things are still coming!

QATracker development
There is still room for more developers on the qatracker project. It's written in drupal, and I would happy to help you get started. As we grow, there will continue to be a need for people who want to build awesome tools to help us as a community test. If you have ideas for a tool (or know of a tool) that would help us test, please feel free to share with me.

Hardware Database
Work has been completed to spec out the design, and is scheduled now to land this cycle not in a future cycle. Fingers crossed we'll sneak this in before we release quantal :-) I'm very excitied to share this new tool with you; as soon as it's complete we'll be able to incorporate it into our workflow on the qatracker.

Testcases
Done, and for the most part our testcases have been migrated over. In addition, there is now a special team of folks who help to manage and maintain our testcases. If you have a passion for this work, contact me and I can help get you involved with the team.

Overall, I am happy to see signs of growth and newcomers to the community. If your on the fence about getting more involved with ubuntu, I would encourage you to check out QA. We collaborate with almost every area of ubuntu in some way, and no two days are the same :-) Send an email to the ubuntu-qa mailing list and introduce yourself.

So what's your opinion? Feel free to respond here with your thoughts and/or fill out the quality survey to give feedback.

Quality Perceptions Survey

What's your perception of quality this cycle? Are things working well for you? It's been several months now since precise landed, and ubuntu development for the next version has been ongoing. The ubuntu QA team has had a busy summer putting into place the new tools we spoke about at UDS. The qatracker has been revamped to allow us to consolidate our testcases and test reporting across all of our activities. In addition, we've been helping in the release of 3 alpha milestones, and 3 testing campaigns. To all those who have helped in this testing, a very big thank you!

I have my own thoughts about the impact to the ubuntu project this testing has had, and I will continue to share my thoughts to point out the progress we make in this regard. But now, I want your input. I have created a survey to understand the community perspective on how we as a ubuntu project are doing on quality. If you have a few moments, please fill out the survey and let your thoughts and perspective be known. The survey will be anonymous, but I will share an aggregation and summary of the results.

My hope is to help gain an understanding of how we can focus our efforts on what's important to ubuntu as a project in terms of quality, as well as how we can help you (yes you!) become a more active part of QA if your interested.

Here's a link to survey. I'll leave it open until next Friday August 10th. Thanks in advance for your participation.

A video walkthrough of ISO Testing

Notice I'm not mentioning a good video walk-through, but it is in HD! It can be viewed on youtube here. I had a lot of trouble doing my first screencast, but I think I have a promising setup now for the future. I promise HD, lag-free, and better editing in the future. For reference, I highly recommend flowblade to edit and kazam to record. Kudos to both developers for nice pieces of software!

The text version is available here and I encourage you to follow along with the video. You don't need anything more than a reasonably modern pc that can run a VM and you can help iso test. Take 20 minutes to watch and follow along and take the fear out of iso testing. Then join us next week as we continue our testing cadence and verify our daily isos are in good shape.

And don't worry, no one will know it's running the installation on a separate workspace while you watch youtube videos.. It'll be our secret.

What can webapps do? Help test and see!

As I'm sure you've already heard, webapps are becoming first rate citizens on ubuntu. Some of you might even have installed the ppa and are enjoying exploring what webapps does for you on the desktop. If you've been one of those early adopters and are wondering exactly what all webapps can do, or are wondering how you can help squash bugs, this is your chance!

The webapps team has made available some manual testcases that cover the functionality found in webapps currently. These testcases have been made available via a 'call for testing' for webapps. Head over to the package tracker and select the tests corresponding to your version of ubuntu. For precise, Webapps Precise Testing, and for quantal Webapps Quantal Testing.

Never participated before in a call for testing? No worries! Check out this handy walk-through and you'll be submitting results in no time. Thanks for helping test!

Remembering the good ole days

Quality has been a buzzword for a couple releases now. Certainly, it's safe to say we saw many more people talking about quality last cycle than any previous cycle I can remember. Ubuntu of course has done LTS's in the past, but something about this past release was different. People yearned again for the perceived quality of the past LTS release and wanted to see ubuntu succeed.

Ahh yes, the good ole days of 8.04 or 10.04 LTS releases when everything was right in the world and ubuntu just worked, etc. Heh, it's easy for us to remember the past and the good things we enjoyed. But to the extent those releases served me well, I have the community to thank. Ubuntu is community, and it's success or failure is determined by all of us.

Open source projects are not typically known for there quality. In the same way ubuntu has innovated and paved the ground for a consumer focused desktop linux, I believe we as a community are uniquely positioned to show how open source can be better quality than competing ideological offerings. In the same way a minuteman can win a battle against a better equipped mercenary, so too can a dedicated community provide better quality software than a commercial offering. This only makes sense. I would rather work with a group of passionate people than with folks who don't care about there work.

One of the best parts about getting to work in QA is seeing the end result of the entire communities work. Those who perform QA represent the last pair of eyes for the work our community does. The developers, translator, doc writers, bug triagers, forums and IRC admins can all do marvellous work in support of ubuntu. But if it doesn't install/work on your pc, then sadly you won't be able to enjoy any of that work.

Recently, those running the development version of ubuntu noticed a bug that caused there webcam to no longer work. The community responded by helping figure out exactly what commit caused the regression. Many folks were involved here! People with the issue tested different kernel versions to narrow down when the regression occurred, while the kernel team made these kernels available and provided insight. Ultimately with everyone's help the problem code was identified and confirmed by testing a custom built kernel. Armed with this knowledge an upstream bug was created.

Now as I sit here today, my webcam works again! And not just for me, or other ubuntu users, but for everyone who uses the linux 3.5 kernel. After upstreaming the bug,  the original developer made a fix available which we tested. Ultimately, this fix made it into the 3.5 kernel. This prevented the 3.5 kernel from shipping with broken logitech webcams for everyone. This is the power of our community!

The story is but one of many examples of problems the ubuntu community has solved. If you enjoyed or remember the good ole days in ubuntu, I would encourage you to get involved. If your running ubuntu on your desktop now and want to continue doing so, consider donating your skills to help. A healthy QA community is vital for ubuntu to continue to grow and get better. We, as a community, can be the standard for quality in open source.  So how can we take ubuntu, and specifically quality in ubuntu, to the next level? It starts with you.

Testing in Cadence

Last month, an interesting thread emerged on ubuntu-devel. A proposal to change the way we as ubuntu look at testing and quality. In many ways it was more of a codification of ideas and thoughts from the precise cycle than a proposal. 

One of the outcomes of this was a change to how to test isos. Rather than focus on arbitrary moments in time, we've been asked to stick to a two week cadence for testing. What that means is a regular checkup of our images every two weeks. Quite a task, but not impossible! Given the fact the change happened mid-cycle, there has been some confusion over what exactly this means. I decided to put together a post detailing exactly what's on the table for us as a community and more importantly how you can help!


If you have a look at this wiki page, I've listed the images we test and produce for ubuntu. So far during the quantal cycle we have achieved 100% coverage for most of these iso's for our mandatory testcases. That's thanks to the wonderful efforts of folks like you testing isos! But in some cases, like our first non-milestone cadence last week, the coverage was provided by a single person -- meaning we have only 1 confirmation of success or failure. I'd like us as a community to take this to the next level. I'm asking for you to commit to an iso over the course of the cycle. Would you be willing to commit to running through the mandatory testcases every 2 weeks for the iso and making sure it's in good shape? If so signup on that wiki page, underneath the iso in question. Don't be afraid if this sort of testing sounds scary. I and the rest of the community are happy to help you through your first testcase. As part of iso testing, I'm still growing my knowledge of linux and ubuntu and interacting with wonderful and talented people while doing it. This is a natural expansion of the 'adopt an iso campaign' with a new cadence. Powerpc, and mac users, this is your chance to make a difference as your hardware is less common and therefore harder to ensure proper testing for.


During the precise cycle over 100 people submitted a result to the isotracker for an iso they downloaded and tested as part of daily testing. That's excellent work, and I thank all of you very much! Many more of you downloaded and installed iso's throughout the cycle, but perhaps didn't report your work. I would encourage you to get involved and help share your results with others. If we have 100 people signup in support of iso testing, the workload required of each individual will be quite small. Yet the benefits for us as a community will be huge. More hardware and more testing results in more bugs caught sooner. We all want a good upgrade experience in October. This is your chance to be a part of making sure it happens.


As a small addendum, I'd like to point out the results of the work this testing achieves. If you have a look on this page, you'll notice a very long list of bugs; many of which are rated as high or critical in launchpad. All of these bugs were found during iso testing -- a testament to those who have tested before us. We all thank you.

Call for Testing: 12.10 kernel on 12.04 -- Part Deux

As announced earlier the kernel team is looking for a some folks to help bring the 12.10 kernel to 12.04. Once 12.10 has been released, the team wants to enable newer hardware support via the kernel for the LTS version of the desktop. So, since the original announcement, we've had 10 people help test the various builds of the kernel from the ppa. Thanks so much to all who tested! Now we'd like to take this testing to the next level.

We've put together a list of commonly used hardware that we want to ensure proper support for the kernel. I'm asking for volunteers to run the kernel from the ppa on precise and report results. The catch here is that we want to have at least 1 person who has each piece of hardware listed be represented. Make sense? For example, we want folks running nvidia cards to have at least a couple people reporting results using nouveau and the proprietary driver. Same for AMD and Intel. On the wireless side, getting someone who has a chipset of each of the manufacturers listed is our goal. This is a first step in our on-going efforts to help make testing and quality a more assured and quantitative effort. We're going 'quantal' if you will. For those who want to have even more detailed and specific hardware testing, hang tight. If you'll remember this past UDS we spoke about creating a community hardware database. Work to enable this is on-going, and I hope to be able to share more about it in the coming months. In the meantime, let's build up a list of folks and systems ready to populate such a database, shall we? ;-)

So if your interested in helping, go ahead and edit that wiki page. Add yourself under one or more pieces of hardware. There's a handy script that should help you identify what's in your system if your not quite sure. Then head over to the QATracker. 


Once there click on 'Quantal kernel for precise LTS', you will wind up on a page showcasing the tests and instructions for this call for testing. If you click the on 'Link to the installation information' you will get information on installing and uninstalling the package. Similarly the 'Link to bug reporting instructions' provides details on reporting a bug you find in the test case. Finally, if you click 'Kernel Smoke Tests' you'll arrive on the page to report your testcase results. Note you will need to sign in using your ubuntu sso account to report results.

If you encounter issues, you can always reboot into your current working kernel and be back to normal. As always, if you have any issues in using the tracker, feel free to get in touch with me.