Canaries and Scouts in Software Testing

In software testing, what you don’t know is often your biggest risk.

As smart as we think we all are, no reasonable amount of brainstorming and careful test planning can discover all possible tests that should be put in place for a given system.  Even for seemingly simple systems, “test space” can be huge.  I say “reasonable”, because at some point every test planning process, you cross the cost/benefit threshold where further planning approaches pure guesswork as to what will actually pay off.

So what do you do next?  As an alternative to exhaustive brainstorming, consider using “canaries” and “scouts” in your testing to help you quickly discover which parts of uncharted test space are worth venturing into.

The term “canaries” comes from the old practice of placing canaries in cages in underground mines.  Birds’ bones are hollow, and what they breathe permeates their entire body nearly immediately.  Unfortunately for the canaries, this made them a great early-warning system for miners.  If dangerous gas was present in a mine, the canary would breathe it in and would quickly die, and the miners would be alerted to leave the area as fast as possible.

In software testing, canaries are mechanisms that provide an early indication of impending problems.  They can take any number of forms, and they don’t have to be automated.  For example, getting a couple of customer bug reports around the same issue can warn you that you have a test hole in a scenario that’s more common than you thought.  An automated canary might look like a set of automated Pri0 integration tests that runs on every daily build before anyone else picks it up for deeper testing. If any of the tests fail, it’s a good sign that the build has major flaws and shouldn’t be evaluated further.

“Scouts” venture forth into uncharted territory looking for something interesting to report back.  It could be an “all clear” report, or it could be along the lines of “danger approaching – to the battle stations!”.  I consider automated exploratory tests using real-world data to be scouting tests. For example, tests that leverage actual customer documents or databases can help you discover test holes around specific data even if you have great code coverage for your system under test.  Beta testers are also great scouts.

In summary, canaries alert you to impending danger, and scouts forge ahead to help you decide where to look next.  Both are invaluable in the testing process. 

So what canaries and scouts are you using?

Comments

• Anonymous
  July 18, 2011
  I always consider malformed/malicious/overflowing data tests to be scouts--generally looking to provide test coverage for the "garbage in" portion of "garbage in, garbage out"!

• Anonymous
  July 18, 2011
  Agree - data fuzzing tests are great scouts!