Jaa

Prescriptive vs. Descriptive 'scripted' tests

  • Artikkeli

Something that raises red flags in my brain is hard-coded strings or test data in either a manual test or an automated test. Yes, I know that sometimes there are times when a test must be very prescriptive and use specific data and follow specific procedures, but I am absolutely amazed how often I see examples of test cases that are so prescriptive in the detail of execution that it completely takes any thought out of executing that test. While it can well be argued that the execution of that test might very well be a brain-dead activity, I would also argue that the person who wrote such a test also lacks creativity and generally has no clue of how to actually design a test.

We did a simple experiment on test design and execution. The purpose was to see if we could design a 'scripted test' that provided the tester with greater freedom, cognitive engagement, and deductive reasoning.

The simulation in this experiment was a simple web page in which the user entered a stock ticker symbol (test data), pressed the "Get Quote" button, and compared the displayed result against the expected price at that time (using a real-time 3rd party stock quote monitoring system). The test was a positive test, but was written 3 different ways as illustrated below.

The first test was very prescriptive and was written as follows:

Purpose: Verify the web page displays the most recent quote for a valid stock ticker symbol registered on a major stock exchange

Steps:

    1. Enter "MSFT" in the Stock symbol text box
    2. Press the "Get Quote" button on the web page

Verify: The displayed quote matches the real-time quote.

Given this test over 95% of the subjects simply entered MSFT and looked for a result. Some did not even appear to compare the result against the real-time quote.

We modified the steps in the test as follows and used a second study group.

Steps:

  1. Enter a valid stock ticker symbol in the Stock symbol text box (e.g. "MSFT")
  2. Press the "Get Quote" button on the web page

In this second session more than 75% of the the test subjects still only entered MSFT and looked for a result. On a later day in the week, we asked the same group to run the test again. Once again over 75% entered MSFT as the test data.

So, we modified the steps in the test once more as follows and used a third group in the experiment.

Steps

  1. Enter a valid stock ticker symbol in the Stock symbol text box from a list of available stock ticker symbols at
    <Link to NYSE>
    <Link to NASDAQ>
    <Link to S&P>
    <Link to London stock exchange>
    <additional links>
  2. Press the "Get Quote" button or press the Enter key

In the third part of the experiment over 95% of the third group clicked the links and selected a stock ticker symbol at random. Some testers copied and pasted the ticker symbols from the linked web pages into the Stock symbol text box, but the majority simply entered the symbol via the keyboard. Some (less than 5%) of the participants simply entered MSFT. (Which is not really surprising since they work there!) What was more interesting was that when the same groups were given this test at a later time over 95% of the testers selected a different link and 99% selected a different stock ticker symbol.

This third part of the experiment essentially is the same test (proves the same hypothesis) but uses a descriptive 'scripted' test approach. A more descriptive test can still achieve the stated purpose and provides 2 more important benefits.

  • The purpose of the positive test (verify the web page displays the most recent quote for a valid stock ticker symbol registered on a major stock exchange) is achieved without hard-coding specific test data or specific results to check against. This means the tester has to use basic deductive reasoning in order to validate the results of the test.
  • The breadth of test data used in the test significantly increased (even if the test was executed by the same person), thus increasing the variability of each successive test and provides the tester with great freedom in selecting the data to use in each test and how to interact with the system under test.

Whether or not the "Get Quote" button is pressed or the Enter key is pressed is tangential to the purpose of the test, so in this case it is not important what action the tester takes to send the request to the web service to get the stock quote; he or she has to trigger that event.

I suspect that one reason why many 'scripted' tests are very prescriptive in nature is because they are written from the "watch me" perspective. In other words, the test is crafted from "this is what I did, so that is how I will write my test...word for word." I also suspect that in many of these cases the tester really doesn't have a clear purpose of what he or she is trying to prove or disprove, that tester is simply writing a script or developing an automated test to satisfy some thoughtless process or increase some magic number.

Watching a tester perform a set of steps and then recording that same set of steps in either a manual or an automated test does not constitute test design; it is a brain-dead activity. Test design is not simply watching a person perform a set of actions and 'scripting' that into a 'test.' And test design doesn't mean reacting to the results of one test and thinking of another test 'on-the-fly.' Designing robust tests is a separate activity from test execution. Designing a robust, descriptive scripted test that enhances the effectiveness of the testing effort requires incredible creativity in order to achieve its desired objective.

So, the next time someone tells you that scripted tests are too restrictive, impede the freedom of the tester, or limits your creativity I would suggest to you that that perspective is rather narrow-minded and limited to a vision in which 'scripted' tests are all highly prescriptive in nature and result in a set of brain-dead steps. Conversely, any professional tester realizes that test design is a very creative process involving the application of your cognitive and analytical skills to help you design a test that aids you in proving or disproving your deduced hypothesis from pluralistic perspectives within the context of the situation, and to ultimately enhance the effectiveness of your testing effort.

Comments

  • Anonymous
    December 23, 2008
    Excellent post. I only stumbled upon your blog yesterday and I've now read dozens of interesting articles. Thanks, I'll definitely visit frequently. I agree with what you wrote. I'm against test documents made for monkeys (basically tests that anyone can run). Testers should definitely think and be creative and have real skills. In my test documents, I'm particularly not specific for how to test. Sure, I give guidelines (try spaces, special characters, foreign characters, blank field, etc.) but I don't give people what to cut and paste into the text field being tested. I had this argument recently with a colleague. His arguments for very prescriptive tests are:

  • They give a pure baseline of results that you can compare to. If the test has even the slightest variation (which stock symbol was chosen), then you can't compare the results of two iterations of the same test.

  • How will anyone know what was tested? In my test reporting, I just ask people to write pass or fail for each case. Using your example, I wouldn't necessarily have them write which stock symbol they tested. I suppose I could, but when I perform the tests, I don't generally have the patience to do all that busy work (which I don't see completely necessary). My response is basically that my goal is to maximize my chances of finding bugs, not to have complete consistency. Everyone should know James Bach's quote that, "Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason that stepping in someone else's footprints minimizes the chance of being blown up by a land mine." At first I thought that descriptive came from the word describe and prescriptive was the opposite. But then I was thinking about how your prescriptive tests are very consistent like automated tests. That's when I noticed the word "script" in the root of both terms. Descriptive tests de-script, that is, they take out certain details of the script (but of course leave in the essence). Prescriptive tests pre-script which means that they write the script (in full) in advance and the script isn't to be changed. -Scott

  • Anonymous
    December 23, 2008
    The comment has been removed

  • Anonymous
    September 08, 2014
    I think you've drawn a conclusion based on a skewed test - confirmation bias. Your assertion is that the first example is not prescriptive (enter "MSFT") while the last example is (enter a ticker from a list). I would contend that the last example is not prescriptive. It offers a large number of options, with the list randomizing to some extent the selection that a tester makes. I would have liked to know if testers executed the test multiple times with multiple ticker codes in the same instance? If not, there's still work to do. I would also like to have seen you test a truly open ended test like "pick four or five ticker codes at random and check that the real time quote matches the displayed quote". That would support or weaken your hypothesis. How many of the testers in the last group pressed the enter key? I note it was added as an option? Perhaps if you deleted that step and focussed the test on the input and outcome (rather than method) the testers would use their initiative to determine how to submit the quote - further broadening the scope and utility of the test.