共用方式為

Missing States

  • 發行項

Every tester forms a model of each application they test. Sometimes this model is explicit, such as when the tester is doing model-based testing. Other times it is unconscious, such as when a tester does not know why they do what they do to find problems.

Similarly, every application functions according to a model. Sometimes this model is explicit, such as when the application is built on a state machine. Other times it is unconscious, such as when it is built by a developer who does not know why they do what they do.

Every customer forms a model of each application they use. Sometimes this model is explicit, such as when the customer explains to someone else their view of why the application works the way it does. Other times it is unconscious, such as when the customer accretes a set of magic steps which get the application to do what they want it to do.

Sometimes these models match up. More often they are wildly different.

The customer's model matching the developer's model tends to be a Good Thing. If the customer's understanding of how and why the application does what it does matches the way the developer designed the application, the customer will know how to make the application do what they want it to do, and they will rarely be surprised by it. The extent to which the customer's model diverges from the developer's model tends to indicate the extent to which the customer will be annoyed and frustrated as they attempt to use the application.

The tester's model matching the developer's model is a Good Thing insofar as it indicates each has a similar understanding of how the application is supposed to function. It can also be a Undesirable Thing insofar as the tester gets trapped by model blindness.

Model blindness occurs when a tester forgets that a model can only contain those states which are expected to happen. Some of the states will be standard behavior; others will be for error conditions and other rare occurrences. All of them have been considered, else they wouldn't be part of the model. While verifying that these expected states and the transitions between them is important, a tester who limits themselves to only these states is, as Shrini reminded me, missing a target rich environment.

This is one reason that cross-feature and integration testing tends to find copious amounts of gnarly issues - while most developers are able to maintain a model of their feature in their head, doing the same for every other feature as well becomes progressively more difficult the more other features there are. So they miss things.

One way to avoid model blindness is to look at only the states, ignoring the transitions, and search for ways to get from any one state to any other state. Another is to look for additional paths for reaching those states. Your favorite get-out-of-the-box-I-am-in technique likely works well here too.

Model blindness can occur even when your model is different from that of your developer. If you are working from a model you have excluded certain items. Unexcluding one or more of these will probably find issues.

Assuming you're working from a model.

Which you always are.

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great testing and coding skills required.

Comments

  • Anonymous
    September 19, 2007
    PingBack from http://msdnrss.thecoderblogs.com/2007/09/19/missing-states/

  • Anonymous
    September 21, 2007
    Preconceived notions about the product being tested, whether from the developers, documentation, or even the tester herself, can truly hinder the process of finding information.  If testers are not careful, they could simply be subconciously providing evidence to fit whatever model they have selected.   It is also not a bad idea, if you have the time, to provide yourself with more than one model for testing the same product.  For instance, after receiving a change list on the product in test, I may use a development model for testing, then walk away and defocus. I will deliberately change my focus by asking myself questions about how I have been viewing the product.  What information have I been finding?  What am I possibly overlooking?  The questions generally depend on the product.  But I find this exercise very helpful in checking my current model of testing and the bias it may contain.  When I return to the product in test, it is essentially with a new set of eyes.  This provides me with a new model for testing the same product, which enables me to find new information and/or problems with the software. Since the models that we use are always linked to our past experiences and possible biased opinions that lie in our minds, it is a good idea to question ourselves and our models as much as we question the product.

  • Anonymous
    September 22, 2007
    The comment has been removed