The Inaccurate Conception

Phillip Armour has written a very insightful article entitled "The Inaccurate Conception" for this month's Communications of the ACM (subscription required). The construction of estimates is a tough challenge for analysts, and, as Armour points out, for software developers as well. Here are a few of his insights:

"When a weather forecast indicates a 40% chance of rain, and it rains on you, was the forecast accurate? If it doesn't rain on you, was the forecast inaccurate? Thought of in these terms, the concept of accuracy takes on a different meaning. It seems that the fact that it does or does not rain on you is not a particularly good measure of the accuracy of the rain estimate."

"The accuracy of a weather forecast is not whether it rains or not but whether it rains at the likelihood it was forecast to rain. Similarly, the accuracy of an estimate on a project is not whether the project achieves its goals but whether it correctly forecasts the probablility of achieving its goals."

OK, now that we can appreciate the no-win situation that a weather forecaster is in, what's the lesson here for analysts? Here's the conclusion of Phillip's article:

"If there is a 40% chance of rain, and I decide to walk outside in my expensive suit without an umbrella and it rains on me and ruins my suit, it is not the fault of the forecast; it is the fault of my decision."

Wait a minute! You mean you can't blame the analyst?

Um, who's going to tell management?