Wednesday, March 13, 2013

Kick-starting quality for Ubuntu Touch

I wanted to write a post about the excitement of the new platform and the wonderful new challenges we face ahead of us. Now, given that this platform is being written right now from the ground up, those with a nose for quality instantly perk up. We love well tested applications, and developing with tests in mind from the start is much easier than attempting to retrofit. Seeing the first fruits of the developer effort is very exciting -- good work everyone!

So with that in mind, I started looking at some of the excellent work the core apps teams are doing with there applications. They've been working with the design community to turn the nice mockups into reality. I took the liberty of checking out and running some of the first versions of these applications. The calculator is one that stood out to me as already closing in on its specification. So armed with some of the design conversation for the calculator, I started a branch to create a set of manual tests for ubuntu touch applications, starting with the calculator. If you are interested in quality, now is the time to be involved! The applications can all be installed and run on your phone or even a ubuntu desktop.

So what can you do?

If you're a tester;
If you're a developer and have questions on writing tests for your application, feel free to contact me. I would love to see not only nice unit test driven code, but also some end user tests via autopilot, and I want to make sure you as a developer have the resources to do so. In addition, we as a quality community are happy to help test with you and write some manual tests to do so for your application.

I'm helping!


Thursday, March 7, 2013

Staring down the scarecrow; should ubuntu roll?

After being away and enjoying some lovely downtime, I've returned to the online world to be met with the rush of a virtual UDS, a rolling release announcement, and a new windowing stack announcement. With the discussion advancing and the UDS sessions completed, it's time to weigh in and speak my thoughts as well.

I'd like to stare down the scarecrow -- that is, let us examine the straw man argument of a rolling release.



On a rolling release
I am definitely in favor of streamlining what we ship and support. The inter-LTS releases in general only make sense to run until the next one arrives. From a quality perspective, I really like what we've done with precise. I think it's an excellent solid base and the point releases we've done keep it relevant and offer a really nice way to get the latest stuff and keep a stable and long-term supported system.

As for a rolling release with LTS's sprinkled in, I have run a rolling release distro in the past (alongside ubuntu). I definitely enjoyed having access to the latest stuff, and having everyone on the same archive all the time (community-wise) kept us tighter and more able to relate and help each other. Overall, I think the pros outweigh the cons on moving towards it, but I have several caveats with the current approach.
  • Monthly snapshots
    • As Colin Watson put it, if we're presumbly releasing and testing a monthly snapshot, we failed in a rolling release sense -- we don't have daily quality.
    • In general, I don't see a target audience for a monthly snapshot. Why can't we create an installer image (do full testing milestone on it), then call it gold for a long period (until new installer changes we want to bring in, which require us making a new image). In other words, I would like to see us only generating new images for an actual release (in this case only LTS's), or for a new installer. (Note that I mean supported images (in any sense), not just an image for testing)
  • Quasi-rolling mentatility
    • It seems like we want to support the idea of users running a snapshot of our archive on a certain date, and then only update at certain days and times
      • This is insane fragmentation and defeats the purpose of being a rolling release. People should strive to run up-to-date systems, and always be current. For us, we need to ensure the archive is always upgradeable so they can do so
  • LTS point-releases
    • Continue and enhance point-releases for LTS to keep regular flows of new, well supported and stable software
      • This was discussed and well noted in discussions and at UDS. As I mentioned, I really like how precise is going, and we can continue and bolster these efforts even more in a rolling world.

On flavors
I will prefix everything I say here with the fact I have never put together and supported a flavor, but I most certainly have enjoyed utilizing them, and working with members of the community who focus there efforts on them.

I was able to catch the end of the UDS session on flavors and had an excellent discussion with some folks from xubuntu and kubuntu. Thanks to those folks who helped provide some flavor feedback on the proposal.

I would like to challenge the flavors to engage in healthy discussions about how there release process works, how to serve the needs of there users, and how to make the best use of there time and resources. I'm sure this type of introspection happens in each flavor on a regular basis, but I'd like to call special attention to how releases work.

Last cycle, ubuntu adopted a cadence for driving quality into the daily images. This work has been on-going since precise really, and rick's idea's for a rolling model continue this line of thinking. If you'll ask the community folks who helped be a part of these cadences, they can tell you it was a challenging change, but we're really hitting our stride now. The constant iterations on how we test and what we do I think had been extremely positive towards helping quality make a bigger impact.

With that in mind, our release processes shouldn't be exempted from this. QA (and development :-) ) efforts are seen as linked to the current release process, resulting in chaos if you are radical enough to unlink them. So what options (in my opinion!) of course exist for a flavor in this new world?
  • LTS only
    • Some flavors have already gone to an LTS only model, and I think it's been extremely helpful for those who've done so, in terms of what they can focus on without worrying about supporting lots of releases.
  • Rolling only
    • You can chose to go full force into a rolling release, and eschew LTS's altogether
  • 2 year "normal" releases
    • You could choose to simply push a new image out every 2 years (like an LTS), but without long term support. Instead, consider supporting until the next (2-years or so) release.
  • Keep things as-is
    • As kubuntu and others have shown this cycle by not adopting a cadence for testing, you can keep the traditional model in place. The buildbots are still there, the testing tools still exist, and the knowledge and experience in releasing on a 6 month cadence is there. Remember, ubuntu itself has synced from upstream debian every 6 months; a flavor could choose to do the same with ubuntu.
Now of all these options, at this point I would personally recommend adopting the LTS only model. Work with and sync your development to your upstream project and land your work in the rolling release. Release an LTS as normal and deliver timely point release updates to the LTS. There is nothing stopping you from even delivering these point releases every 6 months (or a different timetable!), emulating the current process but with a stable ubuntu LTS base and a simplified upgrade process.

On some alternative ideas

6-8? month-stable releases between LTS

Not a bad idea for retaining the flavor of the current system. Indeed, if you really like the idea of monthly snapshots and updating, this is probably the better way to do it. However I don't think it solves any issues for an OEM or for flavors. Namely, the release support timeline is too short for an OEM to base an image on, while for flavors, it would force an even faster cadence and churn upon users. I also don't see a target audience for it. Who would run this, but not run a rolling? Folks who want stability couldn't adopt such a small supported time-frame, and I feel like our efforts to test and release would be wasted as we throw it away as soon as the next stable is out.

Don't change anything!
This idea is just a knee-jerk reaction to change. Unless you feel like our last release was the pinnacle of perfection (I don't), we should be evaluating how we do things, iterate and try and do them better. 

On quality in a rolling release
I want to talk specifically about quality as that's what is dearest to me. How do you ensure quality in a rolling release world?
 
First, I would like to challenge you on what you mean by quality. Is older software better quality than newer software? Age != quality, even though we often traverse down that slippery slope. I wrote about this before, but simply put how we define quality is subjective. For the sake of comparison here, let's talk about quality as having a desktop that just works. That is, your hardware works and the applications and software running on it enables you to accomplish your tasks without issue.

So, with that in mind, how does that work in a rolling release? If you've run the development versions of ubuntu in the past, there have been times where a bad package may have rendered your system unbootable. For any user trying to run this as there daily system, it's obvious that level of 'quality' doesn't work. But at the same time, I've found bugs running the development version of ubuntu that cause actual crashes (see this for example), yet have little impact (if any) on my system working properly to enable me to accomplish my tasks. So how can we define quality metrics (and we should!) for each release? Here's a quick summary of my expectations in extremely simple terms:
  • LTS
    • No issue that hinders or prevents utilizing ubuntu
  • Rolling
    • No issue that would cause a crash for expected usecases and workflows
The key difference to me is usability. If I am forced to use a workaround for a crash in a minor application or task in a rolling release is probably ok. Note that I say probably, because well, we haven't defined these metrics yet as a community. Being forced to do so in an LTS is not an acceptable level of quality. And of course, causing a system to not boot is never acceptable.

On the reaction and the future
I'm excited to see these discussions taking place, and I would encourage people to think critically and take part in these discussions. There are definitely some wonderful ideas and conversation taking place.

Just remember we're a team and all part of ubuntu. Healthy debate is a very important part of continuing to better ourselves as a community and project. 


Tuesday, February 19, 2013

Plan a ubuntu global jam testing event!

Ubuntu Global Jam is just a few short weeks away. I trust you've seen the posts announcing and asking you to plan your events. Maybe you are confused about what type of session to plan or how the event could go. I will echo my friend Daniel Holbach in saying just do it! Grab a buddy (even an online one!) and plan to jam together. If your confused about what to jam with, check out the testing page.

It's got everything you need to run a session, and the documentation has all been done for you. Folks can choose what they are interested in testing (packages, images, or hardware), or even do some hacking on testcases. No need to be a programmer, manual tests can be written by anyone! Participants don't need anything besides there laptop and an image of ubuntu on a cd or usb stick (assuming of course they aren't already running ubuntu raring :-) ).

If your curious about wanting to host a testing event, checkout the testing page on the global jam site. Feel free to get in touch with me as well if you wish to share your stories or ask questions. Let's jam, quality style!

Thursday, February 14, 2013

A thank you to some quality rockstars

The quality team has completed a series of classroom sessions held over the last two months. None of these would have been possible without the help from these wonderful instructors:

phillw, gema, noskcaj, smartboyhw, primes2h, letozaf, sergiomeneses

Thank you!

Thank you as well to pleia2, JoseeAntonioR and the other classroom team members who helped us schedule and run the sessions.


You all rock!

Some quality resources

A couple posts ago, I mentioned the ubuntu quality team was looking for people to join the team and help out in the testing efforts we undertake. Thanks to those of you who've already answered the call and our now joining our testing ranks. We love sharing the joys of testing with others!

We're serious about wanting to make sure you are able to contribute and join the community as easily as possible. So for the last couple months, as a team we've been writing tutorials, giving classroom sessions, and hosting testing events. We really do want you as part of the team. Check out some of the resources available to you and consider becoming a part of the team!

Classroom Sessions
Video Tutorials
Written Walkthroughs

Friday, February 8, 2013

Cadence Weeks, Quality, and You!

Starting tomorrow February 9th, 2013 (heh, some of you reading this might already be in tomorrow), the quality community team will start testing for cadence week #6. During this week, we as a team try and help test specific packages looking for regressions, doing new feature or hardware testing, and making sure our images are in good shape. If your still confused, there's a nice wiki page that lays out what "cadence" means in a bit more detail.

So what does this mean for you, dear reader? Well, we as a team would like you to be involved in helping us test! Everyone has unique ways of interacting with software, and naturally no two computer setups are exactly the same between us. Now, I know what your thinking -- how can I help? I'm no tester, and I don't run development versions of ubuntu!

That's ok! You can still help test without needing to compromise your system. If you don't want to install the development version on your machine, you can use a virtual machine installation instead. If you are unable to run virtual machines, or are confused at the idea, you can still help test by simply running a live session and executing tests there. It's not too hard for you! Check out this walk-through for participating using only an image of the development version of ubuntu and your computer.

To help demonstrate how you can participate, I'll be hosting two live events this next week where I'll be on-hand running through the cadence week tests along with others from the quality team. There will even be a livestream, so if your a visual person (like me!), you can see for yourself how you might be able to contribute. Here's the dates you need to know:

Monday Feb 11th, 1800-1900 UTC in #ubuntu-quality. I'll also be streaming live my participation in executing the tests .

Thursday Feb 14th, 1400-1500 UTC in #ubuntu-quality. No stream, but we'll be hanging out answering questions, and working on submitting test results.


Please consider attending a session or watching the video of the stream afterwards. If you can download an image and boot your computer, you can help test. You want to be a part of ubuntu; let us help you contribute!

Tuesday, February 5, 2013

PSA: Ubuntu Quality wants you!


NOTICE: To whom it may concern, the ubuntu quality team is seeking those with a desire to help ubuntu to contribute to the quality and testing efforts. With a little time and a willingness to learn, you too can unlock the tester within you!

Interested? Please inquire below!

If that text didn't get you, I hope the picture did. Seriously though, if you are here reading this page, I want to offer you an opportunity to help out. We as a team have expanded our activities and projects this cycle and we want to extend an offer for you to come along and learn with us. We're exploring automated testing with autopilot and autopkg, manual testing of images, and the virtues of testing in a regular cadence.

But we can't do it alone, nor do we wish to! We'd love to hear from you. Please have a look at our getting involved page (but do excuse the theme dust!) and get in touch. I offered a challenge to this community in the past, and I was blown away by the emails that flooded my inbox. Send me an email, tell me your interests, and ask me how you can help. Let me help get you started. Flood my inbox again1. Let's make ubuntu better, together!

1. If anyone is counting, I believe the record is ~100 emails in one 24 hour period :-p