Testing in Cadence

Last month, an interesting thread emerged on ubuntu-devel. A proposal to change the way we as ubuntu look at testing and quality. In many ways it was more of a codification of ideas and thoughts from the precise cycle than a proposal. 

One of the outcomes of this was a change to how to test isos. Rather than focus on arbitrary moments in time, we've been asked to stick to a two week cadence for testing. What that means is a regular checkup of our images every two weeks. Quite a task, but not impossible! Given the fact the change happened mid-cycle, there has been some confusion over what exactly this means. I decided to put together a post detailing exactly what's on the table for us as a community and more importantly how you can help!


If you have a look at this wiki page, I've listed the images we test and produce for ubuntu. So far during the quantal cycle we have achieved 100% coverage for most of these iso's for our mandatory testcases. That's thanks to the wonderful efforts of folks like you testing isos! But in some cases, like our first non-milestone cadence last week, the coverage was provided by a single person -- meaning we have only 1 confirmation of success or failure. I'd like us as a community to take this to the next level. I'm asking for you to commit to an iso over the course of the cycle. Would you be willing to commit to running through the mandatory testcases every 2 weeks for the iso and making sure it's in good shape? If so signup on that wiki page, underneath the iso in question. Don't be afraid if this sort of testing sounds scary. I and the rest of the community are happy to help you through your first testcase. As part of iso testing, I'm still growing my knowledge of linux and ubuntu and interacting with wonderful and talented people while doing it. This is a natural expansion of the 'adopt an iso campaign' with a new cadence. Powerpc, and mac users, this is your chance to make a difference as your hardware is less common and therefore harder to ensure proper testing for.


During the precise cycle over 100 people submitted a result to the isotracker for an iso they downloaded and tested as part of daily testing. That's excellent work, and I thank all of you very much! Many more of you downloaded and installed iso's throughout the cycle, but perhaps didn't report your work. I would encourage you to get involved and help share your results with others. If we have 100 people signup in support of iso testing, the workload required of each individual will be quite small. Yet the benefits for us as a community will be huge. More hardware and more testing results in more bugs caught sooner. We all want a good upgrade experience in October. This is your chance to be a part of making sure it happens.


As a small addendum, I'd like to point out the results of the work this testing achieves. If you have a look on this page, you'll notice a very long list of bugs; many of which are rated as high or critical in launchpad. All of these bugs were found during iso testing -- a testament to those who have tested before us. We all thank you.

Call for Testing: 12.10 kernel on 12.04 -- Part Deux

As announced earlier the kernel team is looking for a some folks to help bring the 12.10 kernel to 12.04. Once 12.10 has been released, the team wants to enable newer hardware support via the kernel for the LTS version of the desktop. So, since the original announcement, we've had 10 people help test the various builds of the kernel from the ppa. Thanks so much to all who tested! Now we'd like to take this testing to the next level.

We've put together a list of commonly used hardware that we want to ensure proper support for the kernel. I'm asking for volunteers to run the kernel from the ppa on precise and report results. The catch here is that we want to have at least 1 person who has each piece of hardware listed be represented. Make sense? For example, we want folks running nvidia cards to have at least a couple people reporting results using nouveau and the proprietary driver. Same for AMD and Intel. On the wireless side, getting someone who has a chipset of each of the manufacturers listed is our goal. This is a first step in our on-going efforts to help make testing and quality a more assured and quantitative effort. We're going 'quantal' if you will. For those who want to have even more detailed and specific hardware testing, hang tight. If you'll remember this past UDS we spoke about creating a community hardware database. Work to enable this is on-going, and I hope to be able to share more about it in the coming months. In the meantime, let's build up a list of folks and systems ready to populate such a database, shall we? ;-)

So if your interested in helping, go ahead and edit that wiki page. Add yourself under one or more pieces of hardware. There's a handy script that should help you identify what's in your system if your not quite sure. Then head over to the QATracker. 


Once there click on 'Quantal kernel for precise LTS', you will wind up on a page showcasing the tests and instructions for this call for testing. If you click the on 'Link to the installation information' you will get information on installing and uninstalling the package. Similarly the 'Link to bug reporting instructions' provides details on reporting a bug you find in the test case. Finally, if you click 'Kernel Smoke Tests' you'll arrive on the page to report your testcase results. Note you will need to sign in using your ubuntu sso account to report results.

If you encounter issues, you can always reboot into your current working kernel and be back to normal. As always, if you have any issues in using the tracker, feel free to get in touch with me.

Bleeding Orange and Purple

So today I headed over to ubuntu planet to get my daily fix, when I happen to spy my floating head next to a post. What's this? The last post from my blog was published! That simply won't do for an introduction, so this post will do so!

The reason my blog appeared in the planet blogroll is the same reason I now write this post. I would like to introduce myself as a newly minted ubuntu member! I couldn't be happier to bleed orange and purple. I have been 'around' ubuntu for a long time, but only recently began pursuing a more active role in the project. This culminated in me joining Canonical as the QA Community Coordinator₁ and now becoming a member :-) I would encourage anyone who might want to know more about me to simply visit the about me and contact me pages above. It has links to goodies like my ubuntu wiki page, launchpad information and contact details.

To those of you within the community who I've been working with already, thank you! It's been a blast getting to know everyone. And for those whom I haven't yet had the pleasure of meeting or working with, feel free to say hello. Join a call for testing, or other testing event and get some hands on time with myself and the other folks in QA; we always enjoy meeting new faces and new people.

You can expect to see more QA and testing posts coming from me on this space. I hope to here more from all of you as well. Happy Testing!

1. Yes, Jono Bacon is my boss, and no it's not quite as cool as you think it would be. jk <3 u Jono!

Call for Testing: 12.10 kernel on 12.04

EDIT: This has been migrated to the production instance. Check out the new post here to get invovled.

The first calls for testing for this cycle are happening! I am excited to not only announce the opportunity to help test this new kernel, but also to unveil some of the new features to the qatracker to allow us to better serve our needs for calls for testing.

Last cycle, calls for testing was a manual thing -- I asked, and the community responded, following along using instructions from a blog or mailing list post. Now we're going to put some more structure around this as I spoke about earlier.

The kernel team is committing to keeping precise up-to-date by providing kernels from the future releases in precise. The first one of these will be the 12.10 kernel which will land in precise as part of the normal 12.04.X update. In order for that to happen, the team is making available kernels to test on 12.04. If you need a newer kernel for hardware enablement, this is the kernel you are encouraged to run and report on. For those running the mainline kernel, this differs in that the ubuntu patchset and official support for this kernel running on 12.04 will happen when it is pushed via update.

So we're really testing a couple things here -- the 12.10 kernel on 12.04, but also the new qatracker. Feedback is encouraged on the qatracker also! Ok, so how does this work?

First, you'll need to use the staging site for the qatracker, were a package tracker has been setup. If you click on the 'Quantal kernel for precise LTS' link, you will wind up on a page showcasing the tests and instructions for this call for testing. If you click the 'Link to the installation information' link you will get information on installing and uninstalling the package and filing bugs against the package. Especially note the instructions for filing a bug properly; additional information is requested to help make your bug report more helpful to the development team. If you click the 'Kernel Smoke Tests' link you'll arrive on the page with the instructions for the testcase. If you login to the tracker using your ubuntu SSO credentials, you will be able to report results as well. This should look very familiar to those of you who have used the isotracker in the past. Neat eh? If you have any issues in using the tracker, feel free to get in touch with me.

I'm asking those folks willing to help test please head over to the qatracker and submit results. Note that the qatracker emails are turned off, but otherwise everything should function as expected for you. To leave feedback on the new site, file a bug and mention your using the new staging qatracker. Contributions to the qatracker are welcome and encouraged, contact me if your interested in helping out.

Adopt an ISO: Quantal Style

It's almost hard to believe, but the new cycle is starting to ramp up. In just over one week's time, we'll be putting out an alpha 1 iso for quantal! If you will remember last cycle, I began an adopt an iso campaign to help insure precise got the iso testing coverage it needed to be an awesome release. This cycle, that campaign will continue with an open call for folks to adopt an iso and help test it all cycle long. Instead of managing and updating all of our excellent testers myself via email, I am asking instead for you to subscribe to the iso your interested in adopting and helping to make sure it's in a ready state for each milestone release. Subscribing to the iso will alert you via email when there is a new build for you ready to test, so you don't need to watch the page or await an email from me. If you miss the email from me, you may of course contact me for personal interactions at anytime you wish :-)

Interested? Awesome! This wiki page should detail everything you need to get started. Specifically, you should ensure you subscribe to the testcases for the iso you wish. For example, if I am interested in Ubuntu Desktop i386, I would head over to this page. See that button at the bottom called subscribe? Hit it and you should be subscribed to new builds for that iso. Please note the subscription feature is a work in progress, and there is not (yet!) a management page for subscriptions. Additionally, there is no visual indication on the page that your subscription is active (contributions welcome, contact me if you know drupal and wish to help!). Please read the wiki page on ISO Testing for more information on confirming your subscription in the interim period.

Ok, great, so now your subscribed. There's just one piece left in making sure things go well for your iso. This cycle we are trying something new to help make the alpha, beta, final release process smoother for all of our iso's (and you!). We would like to have our adopters run the daily iso before we spin the first candidate for release. What this means for you is that the week before each milestone release date, go ahead and try testing out the daily version of the iso. Think of it as warm-up for the big day. After all, you don't want your iso to be the one causing the re-spin do you?

This schedule will come in handy. It shows the timeline for how and when we'll be undertaking this testing. I know it looks big and scary, but focus on just the first column called 'Community Testing'. See the week of May 31st with the 'Q-D' listed by it in the 'Community Testing' column? If you glance at the 'Legend' at the top you will notice that 'Q' stands for quantal, and 'D' stands for daily. You will also notice that the Alpha 1 release for quantal is schedule one week after. So our goal is to spend this week getting our iso's in shape for the first spins for Alpha 1. The good news is time spent now is respins saved later. Happier isos make happier users!

Before I close I do want to remind everyone that each cycle is a marathon. We spoke at UDS about burnout, and that is something to keep in mind. Pace yourself and share the work. We'll have a wonderful cycle together. As always, contact me if you need any help. Your response was overwhelming for ubuntu precise, let's keep going strong for quantal. Thanks for helping make ubuntu better for everyone!

Quality in Quantal: A community perspective

I originally posted this wonderful wall of text on the ubuntu-qa mailing list. If you want to get invovled in QA on ubuntu this cycle, you should subscribe to that list. Additionally, sign-up for the ubuntu testing team. Monitoring this blog and @UbuntuTesting will also keep you informed.


ISOTesting
My goal is to help ensure things are smooth before milestones, and
before isotesting events. Before we spin an iso, we want to feel good
about what's going on that iso. And we as a community can help make that
happen. Overall, I want each individual to have a lighter workload than
last cycle, despite having a similar amount of overall work we need to
achieve. To do this I'd like to help enable more people to be testing,
and to expand the 'adopt an iso' program so that folks can focus on
testing things they like and are able to test without becoming
overwhelmed or burnt out. Additionally, respins will be a continuous
focus and communication of what has changed and what needs tested will
be a priority. As a community we want to avoid doing re-work/extra work
and dedicate ourselves to performing quality testing, not merely having
a large quantity of testing.

Application Testing
Last cycle we utilized checkbox to deliver manual application tests.
During UDS, we spoke of expanding the isotracker to do our testcase
management, and thus consolidate our application testing by using the
same tool used for the isotracker to create an application tracker. This
work is on-going, but should be finished at some point during the cycle
so we can adopt it and use it. In the interim period will be continue
utilizing checkbox or doing manual testing via blogs or mailing lists, etc.

SRU Verification
SRU verification is currently a manual process with a high learning
curve and little visibility for many people. During the cycle, we hope
to help change that but also utilizing a new tracker to do SRU testing.
This testing will involve running the stable version (currently precise)
of ubuntu, but testing fixes to individual packages. This makes it a
good fit for those who aren't living on the bleeding edge but wish to
help. When this process is ironed out (sometime during the cycle) I will
contact everyone again with information on howto get involved.

General Testing (eg, Day-to-Day running of the development version)
Some good feedback was given on how to help make this better. There are
a few things we would like to do to help improve this process. First,
day to day changes should be able to be followed easier with some
proposed changes to update-manager to better display changelogs for
updated packages. I'll be detailing some information about how
'whoopsie' works and what it means to you. In addition, keeping the
development release stable at all times will continue to be a priority
for the development teams.

Calls for testing (specific feature or new features of critical package
or focused testing on a specific package)
Last cycle this typically involved me posting and laying out a basic
testplan on my blog with instructions on how to help test. This cycle,
again we hope to consolidate this onto a tracker where the tests and
results can be recorded. I will still be utilizing my blog, the
@ubuntutesting twitter account, this mailing list and our IRC meeting to
publicize events like this for people to get involved and contribute.
It's always fun to see new features before they come to everyone else,
and the feedback loop with the developers was welcome on both sides.

QATracker Development
With these changes to the qatracker, there is room for some folks who
know python and django to get involved and help improve the qatracker
codebase to make testing and reporting easier  Contact me, or simply
have a look at the code on launchpad and start hacking.
lp:~ubuntu-qa-website-devel/ubuntu-qa-website/drupal7-rewrite
lp:~ubuntu-qa-website-devel/ubuntu-qa-website/drupal7-rewrite-testcase-management
lp:~ubuntu-qa-website-devel/ubuntu-qa-website/python-qatracker
lp:~ubuntu-qa-website-devel/ubuntu-qa-website/python-qatracker-testcase-management

Hardware Database
The idea for having a hardware database for testing is not a new one,
but work has begun anew. This is work that will go beyond this cycle,
but ideas are being explored at using ubuntu friendly and other tools to
make this a reality.

Testcases
As a testcase management system will soon be in place (hurray!), we'll
be migrating all of the testcases over to this system. That means will
have much better visibility and ease of maintenance for all of our
testcases. Cleanup and expansion of the number of testcases is
definitely a goal for the cycle, and expect to hear more about getting
involved in this area.

Whew, that's a wall of text, but I hope it helps outline what the plans
are for the cycle. Feedback appreciated and encouraged  Happy Testing!

Executing on an idea; a UDS story

UDS is now behind us, and the excitement of the work that lies before us for the next cycle is fresh on our minds and hearts. Last cycle I solicited and received some amazing ideas for improving how we as a community do QA inside of ubuntu. As UDS neared I encouraged many of those with ideas to participate in UDS by attending, signing up for work items, and advocating their ideas.
This is a key portion of being a part of the community -- you must be willing to act. If you are unwilling to act upon your own idea, why would anyone else? If you don't believe in it, no one else will. Own the problem you wish to solve and you will find others who share your passion along the way to help you achieve your goals. This is the heart of open source.
But how? How can I act? What if the problem is outside of my skillset? Because of the greater community and the nature of open source, you don't have to solve all of the problem by yourself. As you undertake work to execute your idea, you will find it attracts those who are of like-mind and similar persuasion to you. The best part is that they will have different skillsets to bring to the problem and can help you accomplish more than you could alone.
In a previous job, I was given the freedom to spend a percentage of my time on anything I chose; provided I could convince two of my workmates to help out. The idea behind the requirement was a litmus test for my idea. If the idea has merit, I should be able to convince my colleagues to work on it with me. Ubuntu is one of several open source projects to operate on this idea of 'meritocracy'. The basic premise is to have the best people making the most informed decisions possible about problems specific to there expertise. This is achieved by granting authority to make decisions to anyone who demonstrates there ability to do so by contributing to the project.
So, returning to UDS I would like to tell you a small story of just one example of executing on an idea. Let me introduce Paolo Sammicheli to you. Paolo is from the Italian Loco team, and has been active in driving growth in the localized iso community. He began his work by starting an "Italian Testing Team" several UDS's ago, and has been advocating greater testing and community participation for several cycles now. This past UDS, Paolo wanted to help kickstart a localized iso community beyond just his Italian loco iso. Before UDS, he had already produced a set of wiki pages documenting how to use the isotracker admin features with a bent towards running your own localized iso tracker. Additionally, the Italian loco team planned and tested during the 12.04 cycle to create a localized ubuntu 12.04 image for release. Finally, Paolo came to UDS and created a blueprint so he could share his idea with others. Have a look at it yourself:

https://blueprints.launchpad.net/ubuntu/+spec/community-q-localized-iso-community-growth

Paolo was able to generate good ideas, and see other people attempt to replicate his work within their own locos. Plans were made to have two other loco teams produce localized isos this cycle, and ultimately use there findings as a model for future loco teams. Although the work is on-going this cycle, Paolo, I think, has been successful at bringing his idea to life.
How can you replicate Paolo's example? A couple key points I see in what happened.

• Lay the groundwork
• Start proving the idea out as best you can. Perhap's it's a demo or prototype -- maybe even just a specification or a storyboard. You need to convince yourself (and others!) your idea makes sense and can be done
• Tell others
• Let others know about your work. Blog about it, come to UDS, present it at a Ubuntu user days event, post it to the forums, talk to people on IRC about it, etc
• Do it
• This is key. You need to start executing your idea as best you can. People are not going to make your idea a reality without you! (and why would you want them to? It's your idea! Own it :-) )
• Share your work
• Invite others to work with you on your idea. It's helpful to have specific and easy ways to get involved, but don't limit people. You want to work openly in a way that anyone can participate at any level.

Go forth and own your ideas! I empower all of you to do so. Who knows, maybe your OS also won't "just be a hobby, won't be big and professional like gnu".