of moles and tainted peanuts
There was a full page ad for Jif peanut butter in my morning paper that caught my attention. (For those non-US readers, our nation is experiencing a salmonella bacteria outbreak which has been traced back to contaminated peanuts.) The ad touted Jif’s rigorous testing processes and reassured readers that testing for salmonella was a long time habit for the Jif company and we should feel confident in consuming their products.
Now clearly peanut butter is not software. I very much doubt that the processes for making peanut butter have changed much over the past few decades. I also imagine that one batch of peanut butter is about the same as the last batch. I concede that we have a harder problem.
But, the term ‘long time habit’ really caught me. Because I haven’t seen too many long term habits established in the testing industry. We plan tests, write test cases, find bugs, report bugs, use tools, run diagnostics and then we get a new build and start the process all over again. But how much do we learn in the process? How much do we retain from one build to the next? Are we getting better each time we test? Are we getting better purposefully or are we just getting more experienced? In many ways, the only real repository of historical wisdom (those long term habits of Jif) is embodied in our tools.
My friend Alan Page likens testing to playing whack-a-mole. You know the one: chuck a quarter in and plastic moles pop up through a random sequence of holes and you whack them on the head with a mallet. Whack one, another appears and even previously-whacked moles can pop up again requiring additional mallet treatment. It’s a never ending process, just add quarters.
Sound familiar? Testing is whack-a-mole with developers applying the quarters liberally. Now, defect prevention notwithstanding, we can take a lesson from Jif. They understand that certain risks are endemic to their business and they’ve designed standard procedures for mitigating those risks. They’ve learned how to detect salmonella and they have wired those tests into their process.
Have we paid enough attention to our risks that we can codify such routine test procedures and require their consistent application?
Clearly software is not peanut butter. Every piece of software is different; Office’s salmonella is likely irrelevant to Windows and vice versa. But that is no excuse to play whack-a-mole with bugs. We have to get better. Perhaps we can’t codify a salmonella testing procedure into a cookbook recipe but we can start being more proactive with the whole business of learning from our mistakes.
I propose that testers take a break from finding bugs and just take some time to generalize. When the bug pops its head from some coded hole, resist the temptation to whack it. Instead, study it. How did you find it? What was it that made you investigate that particular part of the app? How did you notice the hole and what was happening in the app that caused the bug to pop out? Is the test case that found the bug generalizable to find more bugs similar to the one you are ready to whack? Is there some advice you could pass along to other testers that would help them identify such holes?
In other words, spend part of your time testing the current product you are trying to ship. Spend the rest of the time making sure you learn to test the next product better. There is a way to do this, a metaphor we use here at Microsoft to help.
I’ll discuss the metaphor and how we apply it to form long time habits, peanut butter style, in my next post.
Comments
Anonymous
February 06, 2009
PingBack from http://www.anith.com/?p=6483Anonymous
February 08, 2009
I try to document the software our development team produces and I spend time detailing general test cases and any other thoughts that may be conducive to testing. Whether others read it or take any of it on board is another matter. As testers, we're naturally sceptical of any information we have not yet verified for ourselves! It can't hurt to publish it in some form or another though. I could write a book on date and time testing alone, it's such a minefield. Actually, I feel a blog post coming on...