For those of you dreaming the 100% automation dream...please wake up!

My respected colleague Keith Stobie recently posted to his blog my response an internal query regarding “unrealistically high” goals for the percentage of tests that are automated. (Since, I have been remiss in my public rants I will reuse Keith’s post. Thanks Keith.) Of course, the unrealistically high percentage is probably near the 100% mark. There has been a lot of talk recently about 100% automation goals especially in the Agile world. So, for all those out there that think 100% test automation is a “reasonable” or even achievable goal I have a suggestion. Learn to just say NO to drugs!

There are some areas of testing where I do see the need and value of approximately 100% of the tests being automated. BVT/BAT is a prime example. When I was the Setup Test Lead on Windows 95 I was responsible for developing the BVT/BAT suite. The suite contained approximately 1000 unique functional tests and ran simultaneously on 4 different language versions within the span of 30 minutes once a week for about 1½ years upon delivery of each new build. But it was only maintenance free for the last 6 months of the product cycle. Still, that’s a pretty good return on investment considering Ed Kit, and James Hancock have estimated that an automated test must run 15-17 times to break even on the cost of developing that test. Other areas which lend themselves to high percentages of automated tests include performance testing, stress testing, regression testing, etc. Areas of testing in which I need to establish baselines and run consistent tests in order to compare repeatedly predictable results against those baselines towards some ultimate predefined goal are prime candidates for high levels of automation. Scripted test automation does a very poor job of exposing "new" defects post test design, and GUI automation requires enormous maintenance especially if it is created too early in the product cycle when the U/I is still in transition. The bottom line is we need to be smart about what and when to automate, because good reusable test automation is damn expensive.

For my unabridged response to managers or others who throw around the 100% automation mantra see my comments  in Keith’s blog. For more information about making smart decisions of what and when to automate I highly recommend the book by Daniel J.Mosley and Bruce A. Posey entitled Just Enough Software Test Automation .

Comments

• Anonymous
  August 30, 2006
  The comment has been removed
• Anonymous
  September 01, 2006
  The comment has been removed