Exploratory testing inside the box

Much of the information about exploratory testing focuses on testing from an end-user perspective. Pundits of exploratory testing claim the approach is also useful from a white box test design approach, but I have yet to see any practical discussion or examples. But, professional testers use exploratory testing approaches all the time from a white box perspective to explore the code for untested paths. Professional testers learn about areas of the code that are at risk, and reactively design effective tests to evaluate previously untested or under-tested areas of the code.

Let's use a simple example to get started. Suppose we had to drive from Lynnwood, Washington to Puyallup, Washington without a map or (GPS auto navigation system). Just as we have 'clues' to point us in the various directions while performing exploratory testing at the user interface we have the numerous highway signs to help us navigate various routes to complete our journey. And, it is up to us to decide which route to take. The shortest route is I-405 south to SR-167. But, I-405 is always at a stand-still, so another popular route is I-5 to SR-18 east then SR-167 south. Of course, after traversing those routes a couple of times the scenery (and crawling in traffic) gets a bit boring, so we might find additional less travelled routes. But, regardless of how many times we make the journey or how many different drivers we choose to complete this journey it is highly unlikely that we will traverse every possible route in any reasonable amount of time. Some routes may not be obvious such as I-5 south to Seattle, then taking the ferry from Seattle to Bremerton and continuing to SR-310 south to SR-16, then I-5 north to SR-167. And, of course some routes are impossible (or at least so convoluted they would be improbable).

Fortunately, control flow through even complex algorithms is not as labyrinthine as the state roadways in western Washington. And, just as the department of transportation uses various tools to measure traffic volumes testers can use path profiling tools to measure frequently traversed paths through the code. We can also use code coverage tools to see what paths have or have not been traversed, and which decisions are made at branching statements. Using code coverage and profiling tools to map control flow through the algorithm we are able to more thoroughly explore the code. Using our 'map' we can learn what paths have not been traversed and even whether or not certain paths through the code are even possible. After we explore the 'map' we can more effectively design additional tests to traverse un-tested paths through an algorithm. Common structural test design techniques include  to evaluate code statements, code blocks, simple decisions or branches, or multiple Boolean conditional clauses in a single predicate statement. Then, using those test designs we can execute those tests either using stubs or mock objects at the unit or component level, or through the user interface to traverse those paths to reduce overall susceptibility to risk.

I discuss the various techniques commonly used in structural testing in Chapter 6 of our book How We Test Software At Microsoft , and also address the subject here, and here. Of course, the application of structural techniques is usually referred to as code coverage analysis. But, using this simple analogy hopefully other testers can begin to understand how exploratory testing approaches are used not only from the user interface, but also below the GUI at the code level. As Boris Beizer initially stated, "all testing is essentially exploratory in nature," and code coverage analysis (analyzing code coverage results to learn about, design additional tests, then execute those tests) also makes great use of exploratory approaches inside the box.

Comments

• Anonymous
  March 19, 2009
  The comment has been removed