Dangers of Test Automation Revisited

Here is a comment I received via e-mail in reference to my post on the Dangers of Test Automation.  I found it insightful.  The point is the oversights are easy and automation won't catch them.  Despite a few claims that this is just a deficiency in the test plan, assuming that a test plan is perfect is a dangerous assumption.  There are always going to be holes, just like there will always be bugs in the code we write.  Having manual exploration gives us insurance that we will cover these holes.  Once again, if all of your testing is automated, you are done finding bugs after your first run.

Here is the comment:

I was reading your post:

 

... and I was immediately reminded of a scene from Jurassic Park (the book, not the movie, though that was good too)

 

The park administrator is showing off the high-tech computer systems. He says "you can search for dinosaurs... I'll just search for all of them... and the big board will show you their current locations." Sure enough, there they all are.

 

Later it is revealed that, contrary to design, the dinosaurs are breeding. Someone asks why the extra dinosaurs didn't show up on the computer search.

 

Nedry answers (I'm paraphrasing) "The search allows you to enter an 'expected number' of dinosaurs, which helps it to run faster. It's a feature."

 

Premature optimization strikes again...

Comments

• Anonymous
  April 19, 2006
  Of course a more logical way to write this part of the book is that each dinosaur was implanted with a tracker, but born dinosaurs obviously have no tracker.
  
  Plus if the search was limited by a value, it would have found a random (or by proximity) choice of dinosaurs.  Of course tracking them without some type of ID/tracking is just silly science.
• Anonymous
  April 19, 2006
  The comment has been removed
• Anonymous
  May 17, 2006
  This is the third in my series on test harnesses. In this post, I'll talk about systems that do much...