The grand "Cadence" experiment

It all started innocently enough. A simple idea, turned into a simple post on the mailing list. This idea eventually led the ubuntu QA community to perform an experiment for this cycle, which has been dubbed "cadence testing".

Now, before I manage to confuse everyone with this "cadence testing" term, let's define cadence testing. Scratch that, let's just give a simple definition of what was intended by original idea. If you want the whole story read the thread. heh. I'll be waiting (hint it's LONG!).

Cadence testing was intended to introduce regularity into testing. If the development release could be "stable" everyday (which was the grand experiment during the precise cycle), could we not also test to ensure that things were good all throughout the release? If the everyday images and archive were now to the quality of previous release's milestones, could we just eliminate the milestone idea and go with a calendar schedule for testing? Thus, a proposal was made test every 2 weeks, whether or not a milestone had been planned, and report the results.

Fast forward 2 months to today. So what happened? Well, I'm happy to report that the QA community despite the confusion more or less met the goal of testing the desktop images every 2 weeks (milestone or not). But what did this achieve? And where are the results?

Let's step back a moment and talk about what we learned by doing this. My comments are specific to the non-milestone cadence testing weeks. First, the development process inside ubuntu is still built around milestones. The daily images during cadence testing weeks were sometimes stable, and sometimes flat out broken by a new change landing from a development team. Second, the tools we used are built around milestone testing as well. The qatracker as well as any qa dashboard or report available doesn't have a good way to track and image health across the cadence week. This meant it was both difficult to test and difficult to see the results of the testing. Finally, the development teams were not expecting results against the daily images, and couldn't follow-up well on any bugs we reported, nor where we able to coordinate well with the release team, as the bugs reported were not available in a summarized or meaningful way.

Now, I'll save the discussion on my ideas of a healthy QA workflow for a later post, but I think we can agree that testing without good result reporting, and without developer follow-up has a limited impact. So does this mean "cadence testing" was a bad idea? No, it was simply poorly executed. The trouble comes in the assumptions listed above.

The archive has not been "stable" everyday, and development teams have have continued development, pausing only as required by the current milestones. In addition, changes, even major ones (like the ubiquity changes landing a few weeks ago, or the nvidia change just this past weekend), are not well communicated. Since they land without little or no warning, we as a QA community are left to react to them, instead of planning and executing them. In this environment, cadence testing makes little sense.

So was the experiment a failure then? In my mind, not at all! In fact, I think the future of ubuntu and QA is to push for complete adoption of this idea, and this experiment confirms the obstacles we will face in getting there. I'll be posting more about what this vision for QA looks like, but I'll leave you with a few thoughts until then.

In my mind, QA should enhance and improve developers, testers, and users lives and workflows. Our work is critical to the success of ubuntu. I would like to see a future where users receive regular, timely scheduled updates that the folks in the QA community have vetted by working with the development and release teams to deliver focused quality updates. The ideal workflow is more fluid, more agile and yes, it has a cadence.