Look below the UI for more effective and robust UI automated test case designs
Last month I wrote about simplistic views of UI test automation in which some people want to pretend that recording for playback or scripting hard-coded actions and data to mimic some human’s interactions at the keyboard is an automated test. Balderdash! Automating a set of sequences or preconceived steps simply for the sake of automating or preparing an environment is perhaps what Kaner, et. el. mean when they refer to computer assisted testing; however, computer assisted testing is not the same as a well designed automated test. (And yes, computers are very good tools for completely automating some types of tests quite effectively; including the oracle.) We see a lot of computer assisted testing in UI automation projects. I suspect this occurs because people are focused on trying to automate a test the same way they or an end-user would interact with the computer rather than design the automated test to evaluate an important attribute or capability of the software in order to provide significant information to the project team and add value to the testing process.
Personally, I am not a big fan of UI automation because it is usually done poorly, and it is usually very fragile and needs constant massaging; more so than test automation that runs below the UI layer. Also, I see a lot of misuse of UI automation. For example, I recently came across a comment by one fellow that wrote, “UI Automation is not necessarily meant for testing the UI (though, we use it for that also).” What??? I do understand the need for UI automation in the testing process, and done well it can provide tremendous benefit and free up my time to actually design new tests and think more critically about what has and has not been tested. But, when I automate through the UI my test cases are primarily testing behavioral aspects of the software (end-user scenarios for example) and that UI elements call the appropriate event handlers. While UI automation can be used to test functional capabilities also, it is generally not the best approach for robust functional testing. This is especially true when the automated UI test is over-loaded with excess baggage (manipulating UI elements not directly associated with the purpose of a test). The more baggage a UI test carries, the greater the potential for maintenance nightmares.
For example, not too long ago a tester was performing international sufficiency testing of his component to ensure his feature supported multiple national conventions and custom formats supported by Windows National Language Support (NLS) APIs. He knew the steps to manipulate the national conventions and custom formats required the user to click the Start menu, select Control Panel, then click on the Regional and Language Options control panel applet, click the Customize button, select the appropriate property sheet for the national convention he wanted to customize the setting for, and finally click the OK button the the Customize dialog and the Regional Settings dialog, and verify the results. Lather, rinse, and repeat as necessary!
To make matters more complicated the sequence of steps to change these settings are slightly different between Windows Xp and Windows Vista and we certainly don’t want to write 2 separate test cases, or branch the test code depending on the operating system in this case. Complexity cultivates complication; especially with UI automation! Fortunately, this fellow also knew that essentially all underlying functionality can be accessed via Windows APIs, and that is exactly the information he was looking for. In this situation I suggested he look at the SetLocaleInfo function and within minutes he incorporated that function to efficiently resolve his problem, and his automated test was capable of testing his application on any currently supported version of the Windows operating system.
In C# automation, we can use Process Invocation Services to PInvoke this Win32 API function from Kernel32.DLL as illustrated below
1: using System;
2: using System.Runtime.InteropServices;
3:
4: namespace TestingMentor.PInvoke
5: {
6: class NativeMethod
7: {
8: [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
9: public static extern bool SetLocaleInfo(
10: uint locale,
11: uint localeType,
12: string localeData);
13:
14: [DllImport("kernel32.dll", SetLastError = true)]
15: public static extern bool SetLocaleInfo(
16: uint locale,
17: uint localeType,
18: int localeData);
19: }
20:
21: class NlsConstant
22: {
23: public enum Locale : uint
24: {
25: Invariant = 0x007F,
26: SystemDefault = 0x0800, // use system default for setlocaleinfo
27: UserDefault = 0x0400,
28: Neutral = 0x0000,
29: CustomDefault = 0X0C00, // Vista and later
30: CustomUiDefault = 0x1400, // Vista and later
31: CustomUnspecified = 0X1000 // Vista and later
32: };
33:
34: public enum LocaleType : uint
35: {
36: // VALUE LCDATA TYPES
37: CalendarType = 0x00001009, // type of calendar specifier
38: CurrencyDigits = 0x00000019, // local monetary fractional digits
39: CurrencySymbol = 0x0000001B, // position of positive currency symbol
40: FractionalDigits = 0x00000011, // number of fractional digits
41: NativeDigitSubstitution = 0x00001014, // native digit substitution
42: FirstDayOfWeek = 0x0000100C, // first day of week specifier
43: FirstWeekOfYear = 0x0000100D, // first week of year specifier
44: LeadingZeros = 0x00000012, // leading zeros for decimal
45: Measure = 0x0000000D, // 0 = metric, 1 = US
46: NegativeCurrency = 0x0000001C, // negative currency mode
47: NegativeNumber = 0x00001010, // negative number mode
48: PaperSize = 0x0000100A, // paper size
49: TimeFormat = 0x00000023, // time format specifier
50:
51: // STRING LCDATA TYPES
52: // Valid Unicode characters
53: AM = 0X00000028, // AM designator
54: PM = 0x00000029, // PM designator
55: CurrencySymbol = 0x00000014, // local monetary symbol
56: DecimalSeparator = 0x0000000E, // decimal separator
57: DigitGrouping = 0x00000010, // digit grouping
58: ListSeparator = 0x0000000C, // list item separator
59: LongDate = 0x00000020, // long date format string
60: MonetaryDecimalSeparator = 0x00000016, // monetary decimal separator
61: MonetaryGrouping = 0x00000018, // monetary grouping
62: MonetaryThousandSeparator = 0x00000017, // monetary thousand separator
63: NativeDigits = 0x00000013, // native ascii 0-9
64: NegativeSign = 0x00000051, // negative sign
65: PositiveSign = 0x00000050, // positive sign
66: ShortDate = 0x0000001D, // short date format string
67: ThousandSeparator = 0x0000000F, // thousand separator
68: TimeSeparator = 0x0000001E, // time separator
69: TimeFormat = 0x00001003, // time format string
70: YearMonthFormat = 0x00001006 // year month format string
71: };
72:
73:
74: public enum LocaleData : int
75: {
76: // LOCALE_ICALENDARTYPE VALUES
77: Gregorian = 1, // Gregorian (localized)
78: GregorianUS = 2, // Gregorian(Always English)
79: GregorianMEFrench = 9, // Middle East French
80: GregorianArabic = 10,
81: GregorianXlitEnglish = 11, // transliterated English
82: GregorianXlitFrench = 12, // transliterated French
83: Japan = 3,
84: Taiwan = 4,
85: Korea = 5,
86: Hijri = 6,
87: Thai = 7,
88: Hebrew = 8,
89: Umalqura = 23, // Um Al Qura (Arabic lunar) Vista or later
90:
91: // LOCALE_ICURRENCY
92: PositiveCurrencyPrefixNoSeparation = 0,
93: PositiveCurrencySuffixNoSeparation = 1,
94: PositiveCurrencyPrefixSeparation = 2, // one character separation
95: PositiveCurrencySuffixSeparation = 3, // one character separation
96:
97: // LOCALE_IDIGITSUBSTITUTION
98: DigitSubstitutionContextBased = 0,
99: DigitSubstitutionNone = 1, // use this setting for full unicode support
100: DigitSubstitutionNative = 2, // uses digits based on national conventions
101: // according to LOCALE_SNATIVEDIGITS
102:
103: //LOCALE_IFIRSTDAYOFWEEK
104: Monday = 0, // LOCALE_SDAYNAME1
105: Tuesday = 1, // LOCALE_SDAYNAME2
106: Wednesday = 2, // LOCALE_SDAYNAME3
107: Thursday = 3, // LOCALE_SDAYNAME4
108: Friday = 4, // LOCALE_SDAYNAME5
109: Saturday = 5, // LOCALE_SDAYNAME6
110: Sunday = 6, // LOCALE_SDAYNAME7
111:
112: // LOCALE_IFRISTWEEKOFYEAR
113: FirstDay = 0, // Week containing 1/1 even if single day
114: FirstFullWeek = 1, // first full week following 1/1
115: FirstWeek = 2, // first week with at least 4 days after 1/1
116:
117: // LOCALE_ILZERO
118: NoLeadingZero = 0, // .975
119: LeadingZero = 1, // 0.975
120:
121: // LOCALE_IMEASURE
122: Metric = 0,
123: US = 1,
124:
125: // LOCALE_INEGCURR
126: ParenthesisSymbolNumber = 0, // ($1.1)
127: NegativeSignSymbolNumber = 1, // -$1.1
128: SymbolNegativeSignNumber = 2, // $-1.1
129: SymbolNumberNegativeSign = 3, // $1.1-
130: ParenthesisNumberSymbol = 4, // (1.1$)
131: NegativeSignNumberSymbol = 5, // -1.1$
132: NumberNegativeSignSymbol = 6, // 1.1-$
133: NumberSymbolNegativeSign = 7, // 1.1$-
134: NegativeSignNumberSpaceSymbol = 8, // -1.1 $
135: NegativeSignSymbolSpaceNumber = 9, // -$ 1.1
136: NumberSpaceSymbolNegativeSign = 10, // 1.1 $-
137: SymbolSpaceNumberNegativeSign = 11, // $ 1.1-
138: SymbolSpaceNegativeSignNumber = 12, // $ -1.1
139: NumberNegativeSignSpaceSymbol = 13, // 1.1- $
140: ParenthesisSymbolSpaceNumber = 14, // ($ 1.1)
141: ParenthesisNumberSpaceSymbol = 15, // (1.1 $)
142:
143: // LOCALE_INEGNUMBER
144: Parenthesis = 0, // (1)
145: NegativeSignNumber = 1, // -1
146: NegativeSignSpaceNumber = 2, // - 1
147: NumberNegativeSign = 3, // 1-
148: NumberSpaceNegativeSign = 4, // 1 -
149:
150: // LOCALE_PAPERSIZE
151: USLetter = 1,
152: USLegal = 5,
153: A3 = 8,
154: A4 = 9,
155:
156: // LOCALE_ITIME
157: FormatAM_PM = 0,
158: Format24Hour = 1
160: };
161: }
162: }
You see, manipulating the Regional Options settings through the user interface had nothing to do with the purpose of his test; it was whether or not those changes in the NLS settings were propagated to the application under test, and whether the resultant output displayed correctly. The oracle to verify the output in this case was simply reading the string from the appropriate control in the application and comparing each character code point value with the expected character. For example, one test changed the date format from dd/mm/yyyy to yyyy-MM-dd. The automated oracle verified the year, month and day values in the correct format and order, and also checked whether the date separator characters in the 4th and 7th position in the string were Unicode values U+002D in this example (or other randomly generated Unicode character value(s)). This automated test was able to test and verify 31 different customizable NLS settings with multiple variables per setting to satisfy basic international sufficiency of this tester’s feature in a fraction of the time it would require a human, and with greater precision. Of course, this assumes that as a tester you have an in-depth understanding of the “system” on which you are tasked to test, and capable of designing effective tests from perspectives other than that of the end-user.
I try to constantly emphasize the emerging role of a software tester primarily focuses on analysis and design; analysis of the “system”, the tests, and the results of tests, and the design of effective tests with reasoned purpose and well defined goals. Professional testers provide value by enriching their organization’s intellectual knowledge repository and ultimately resolving hard problems. But, we can’t start to resolve the hard problem of effective UI test automation by perpetuating the medieval mentality that UI automation is merely mindlessly mimicking the clicks and keystrokes through the user interface because we don’t understand how the system works below the surface, or we can’t think intelligently about effective oracles capable of interpreting the results for some of our automated tests. The persistent prophets of pestilence will perpetually pule, but fortunately I see more and more professional software testers stepping up to meet increasingly complex technological challenges head on with increasing success. As I have said before, the only problems we can’t solve are those which we have not yet devised a solution.
Comments
- Anonymous
April 16, 2009
PingBack from http://blog.martinig.ch/?p=264