通过

Boundary testing - hidden loops and the Deja Vu Heuristic

  • 项目

I previously discussed various types of defects exposed via application of the boundary value analysis testing technique including a repaint problem, a casting problem, and a wrapping problem. While the minimum and maximum physical linear boundaries of a parameter are often easy to identify, it is surprisingly more difficult to identify boundary conditions within the minimum and maximum range especially if the tester does not adequately decompose the data. This week I will discuss another boundary defect that is often hidden below the user interface, but could be exposed using boundary value analysis testing at the unit level.

Loops are common structures in software, and (depending on the programming language) are susceptible to boundary defects. Boundary value analysis of a loop structure involves (at a minimum) bypassing the loop, iterating through the loop one time, iterating through the loop 2 times, iterating through the loop the maximum number of times and one minus the maximum number of times, and finally trying to exceed the maximum number of iterations through a loop structure by one time. (This is the min -1, min, min +1, max -1, max, and max +1 analysis of the boundary conditions.)

For example, the following method counts the number of characters in a string (actually, it counts the number of Unicode character code points in a string). To boundary test this method we would need to bypass the loop once by passing an argument of an empty string (minimum -1), then a string of one character "a" (minimum), and a string of 2 characters "ab" (minimum +1). Next, we would test the maximum range with a string of 2,147,483,646 characters (maximum -1), 2,147,483,647 characters (maximum) and 2,147,483,648 characters (maximum +1). The ToCharArray method will copy a maximum number of Unicode characters from a string to a character array equal to a signed 32-bit int type. So, passing this method a string of 1,073,741,824 Unicode surrogate pair characters the actual number of Unicode characters will be 2,147,483,648 which will throw the out of range exception. (This happens to be a common error. Many developers assume one character == one byte or one UTF-16 Unicode code point value.)

         private static int GetCharacterCount(string myString)
        {
            try
            {
                char[] cArray = myString.ToCharArray();
                int index = 0;
                while (index < cArray.Length)
                {
                    index++;
                }
                return index;
            }
            catch (ArgumentOutOfRangeException)
            {
                throw;
            }
        }

However, it is sometimes difficult to identify the boundaries of looping structures unless the tester is familiar with the programming language and/or data types. In the above example, if the tester is not aware of Unicode encoding patterns (especially surrogate pair encoding) and simply tests the physical extreme boundary conditions using only ASCII characters the method will appear to return the correct number of characters in a string up to and including the maximum length of 2,147,483,647 characters. But, passing a string of 2,147,483,647 characters in which even one character in that string is a surrogate pair will cause the ToCharArray method to throw the out of range exception.

Occasionally it may be difficult to even identify looping structures, especially when designing tests from only a black box test design approach. For example, in Window Xp a known defect appeared to allow a device name (LPT1, CON, etc.) as the base file name if the extension was appended to the base filename component in the Filename edit control (I'll talk more about this defect  later.) A Windows Xp patch attempted to correct this defect; however classic boundary analysis testing easily revealed the defect was only partially fixed as illustrated in the steps below.

  1. Launch Notepad
  2. Select the File -> Save menu item.
  3. In the Filename edit control on the Save dialog enter "LPT1.txt" (without the quotes).
  4. Press the Save button
  5. Press the Yes button on the error dialog that states the file already exists and asks "Do you want to replace it?' As illustrated below,

lpt1error
If the patch/update is applied to the system Notepad will return an error message indicating it cannot create the file as illustrated below.

 notepad lpt1

  1. Next, press the OK button on the error dialog
  2. Repeat steps 2 through 5 above.

Now, notice instead of an error message the Window title of Notepad has changed from Untitled - Notepad to LPT1 - Notepad. But, this is a reserved device name, so how can we save a file named LPT1.txt to the Windows file system? The answer is we cannot! Although the application title reads LPT1 - Notepad a file named LPT1.txt does not exist on the file system. This essentially constitutes a data loss defect because it appears to the user that they saved a file named LPT1.txt. (Yes, I am aware of the other bugs associated with reserved device names as well and shall write about them in the future. )

lpt1 in notepad

Now, some of you may ask how I knew to test for a looping structure with the Save dialog? Quite simply; I use a technique I refer to as the deja vu heuristic anytime I encounter an error dialog. (A heuristic is a commonsense rule (or set of rules) intended to increase the probability of solving some problem. )  Anytime I encounter an error dialog I repeat exactly the same set of steps to make sure I get the same exact error dialog. Error handling routines often employ loops and are often prone to defects especially if some variable is initialized inside the loop structure. The deja vu heuristic is designed to analyze the minimum boundary of an error handling routine that employs a loop. The minimum - 1 value is not executing the error path, the minimum boundary condition is executing the path that instantiates the error dialog, and the minimum +1 value is repeating the same steps to execute the same path (or not in case of a defect).  In fact, anytime I execute the same exact steps of an error path and get a different result the underlying architecture of the code is suspect and bound to contain one or more defects.

Looping structures are another common cause of boundary class defects, and this is clearly a case where visibility into the code and in-depth knowledge of data types is advantageous for the professional tester.

Comments