Testing and Quality
I’ve been thinking about testing and quality again lately. Across the industry, testers call themselves “quality assurance”, or “quality engineers”, and testers say they “measure quality”. I’ve had serious thoughts recently whether the two terms (“quality” and “testing”) are as related as most people seem to think.
Testers don’t measure quality (and they certainly don’t assure it). The role of test is to provide information – but that information is just data – data about how the product was engineered. Things like code coverage, test pass rates, and bug rates are just status about the engineering effort. Product quality – or perhaps better put – customer quality is different. Customers don’t care if you had 90% code coverage. Customers don’t care what your test pass rate is, and don’t care how many bugs they found. To them, quality is more emotional – quality products make them feel good, and help them accomplish a task in a new or exciting way. Somewhere along the line, someone decided that engineering quality was the same as product quality. Granted, the choices testers make to measure engineering quality are based on experiences in customer reactions, but it’s a crapshoot how much the the efforts align.
Graphically, you have two different measurements. In the worst case, they miss completely. For example, think of a level 5 cmm company that delivers bad software – they did it because they missed the boat on matching their process to measuring customer quality.
In a perfect world, the two measurements match exactly. In reality, I bet that engineering quality and customer quality match somewhere in the 25-50% – maybe 60% at the most.
To get customer and engineer quality to match, you need measurements that can predict how the customers will perceive quality. This dictates that the best measurements would be from customer feedback during product development – things like beta feedback, newsgroups, and customer surveys may be more valuable than you think.
So what do you do with the engineering metrics? You keep them. If the role of test is to provide information, the right set of measurements can be the perfect tool for assessing risk. All that stuff like code coverage and pass rates that the customers don’t care about – they don’t care, but you still need to do it. If I buy a new car, I don’t care how many parts were replaced on the assembly line as long as it looks and works perfectly when I drive it away. Even though I don’t care about how the car was engineered, my perception of quality is affected by the quality of the engineering process – whether I know it or not.
Comments
- Anonymous
April 13, 2008
Alan,I have difficulties in understanding various manifestations of quality as you have articulated here ....Product Quality - Intrensic to the productEngineering Quality - Quality of all (engineering) processes that produce the productCustomer Quality - Eventually what customer cares about (will he buy it? will he recommend it?)Except customer quality (which is a time-bound relationship between the user and the product), other two forms of quality appear to indicate that "quality is part or an attribute" of "process", "product"...I just left a comment to your previous post quoting Michael Bolton that ... "quality is a relationship (1-1) but not an attribute of a thing ...What do you say?Shrini - Anonymous
April 14, 2008
Let me put it differently.I'm saying there are two kinds of quality: Engineering quality, and "Experience" quality. Engineering quality is a measure of how the product was built - Experience quality is what the customer sees.I don't think engineering quality is process quality - i.e. I dont care how the process works, but I do care about the measurements. For example, I suppose there's a process to code coverage, but what I care about is the data. - Anonymous
June 09, 2008
The comment has been removed - Anonymous
June 10, 2008
上一篇談 Tester 的職責是什麼, 這一篇談另一個觀念, Tester 不應該被叫做 QA (quality assurance), 因為基本上, Tester 沒辦法真的確保 "Quality".