Testing in Cadence

Last month, an interesting thread emerged on ubuntu-devel. A proposal to change the way we as ubuntu look at testing and quality. In many ways it was more of a codification of ideas and thoughts from the precise cycle than a proposal. 

One of the outcomes of this was a change to how to test isos. Rather than focus on arbitrary moments in time, we've been asked to stick to a two week cadence for testing. What that means is a regular checkup of our images every two weeks. Quite a task, but not impossible! Given the fact the change happened mid-cycle, there has been some confusion over what exactly this means. I decided to put together a post detailing exactly what's on the table for us as a community and more importantly how you can help!


If you have a look at this wiki page, I've listed the images we test and produce for ubuntu. So far during the quantal cycle we have achieved 100% coverage for most of these iso's for our mandatory testcases. That's thanks to the wonderful efforts of folks like you testing isos! But in some cases, like our first non-milestone cadence last week, the coverage was provided by a single person -- meaning we have only 1 confirmation of success or failure. I'd like us as a community to take this to the next level. I'm asking for you to commit to an iso over the course of the cycle. Would you be willing to commit to running through the mandatory testcases every 2 weeks for the iso and making sure it's in good shape? If so signup on that wiki page, underneath the iso in question. Don't be afraid if this sort of testing sounds scary. I and the rest of the community are happy to help you through your first testcase. As part of iso testing, I'm still growing my knowledge of linux and ubuntu and interacting with wonderful and talented people while doing it. This is a natural expansion of the 'adopt an iso campaign' with a new cadence. Powerpc, and mac users, this is your chance to make a difference as your hardware is less common and therefore harder to ensure proper testing for.


During the precise cycle over 100 people submitted a result to the isotracker for an iso they downloaded and tested as part of daily testing. That's excellent work, and I thank all of you very much! Many more of you downloaded and installed iso's throughout the cycle, but perhaps didn't report your work. I would encourage you to get involved and help share your results with others. If we have 100 people signup in support of iso testing, the workload required of each individual will be quite small. Yet the benefits for us as a community will be huge. More hardware and more testing results in more bugs caught sooner. We all want a good upgrade experience in October. This is your chance to be a part of making sure it happens.


As a small addendum, I'd like to point out the results of the work this testing achieves. If you have a look on this page, you'll notice a very long list of bugs; many of which are rated as high or critical in launchpad. All of these bugs were found during iso testing -- a testament to those who have tested before us. We all thank you.