Don’t Worship at the Altar of Accuracy

Earlier today I found myself faced with a common management situation.  I had been sent an e-mail which showed that a piece of data we were using was inaccurate.  The specific issues was what percentage of a certain test run was automated.  We had said we were at 100% and it turned out there were a handful of tests that were being run on our behalf by someone else and those were not automated.  My initial response was to investigate how big the non-automated block of tests was, why it wasn’t included, etc.  Then I stopped and thought about it.  Even if the number were as large as reported, that number would be 10% of the total test suite.  That is almost certainly an over-exaggeration.  When we make the numbers more accurate, it probably slips to 1%.  Whether we are 90% automated, 99% automated, or 100% automated, does that change anything?  Is that number going to change what I ask of my team?  Probably not.  In all cases the items that are manual are intended to be that way.  I won’t stop running them or try to automate them.  All that I will gain by going through the process of making the number more accurate is a more accurate number.  Is there value in that?  I assert that the answer is no.  A number’s accuracy matters only to the extent that the difference will change behavior.  Within some range, different numbers won’t change behavior and so are not worth expending effort increasing the accuracy.

This isn’t to say numbers don’t matter at all.  They do—but only when decisions will be made based on them.  Effort is not free.  Spending energy refining a value that is accurate enough means not expending that same energy on something that might bring more value to the team.  It isn’t hard to find something which brings more value because the value an increasingly accurate number brings is zero.  This is especially important to note as a manager.  A manager typically does not spend a lot of effort making data accurate.  He or she merely asks others to do so.  In this way the costs are hidden and thus the tradeoff not as apparent.  Beware the cost of obtaining accuracy for its own sake.  I know it is in our DNA as engineers.  Suppress your inner urges and don’t worship at the altar.  Get to good enough and stop.

Comments

• Anonymous
  April 27, 2009
  The comment has been removed

• Anonymous
  April 27, 2009
  I get your point, though I think people's perceptions also must be taken into account. You said you were going to do something, then didn't. In future, why not say we are going to aim to automate 100%. Perhaps a more practical goal? Anne-Marie http://www.testingtimes.ie

• Anonymous
  April 28, 2009
  The comment has been removed