UITest framework in Visual Studio 2010 – Part 2
In Part 1 of this blog we saw UITest architecture and how it is used in Microsoft Test runner. In this blog we will see how it is used in CodedUITest and how to troubleshoot and get resilient playback.
Automated Functional Tests using CodedUITest
CodedUITest can be used to automate the manual test actions in the form of C#/VB code to play them back later. One can generate the code for manual testing actions on the application under test (AuT) using one of the two ways:
1. From the recorded strip stored in TFS (mentioned above)
(For more details refer: https://msdn.microsoft.com/en-us/library/dd286652(VS.100).aspx)
2. By doing recording using CodedUITestBuilder (For more details refer https://msdn.microsoft.com/en-us/library/dd286681(VS.100).aspx)
The generated code can either in the form of C# or VB.net syntax. Once code is generated the CodedUITestBuilder also allows users to add validation of the GUI based on UI Object properties. The generated code can be modified in order to enhance the testing.
Please take a look at the below blogs to know about creating CUIT tests
How to: Generate a Coded UI Test from an Action Recording
How to: Generate a Coded UI Test by Recording the Application Under Test
How to: Add UI Controls And Validation Code Using the Coded UI Test Builder
Customizing CodedUITest
This section elaborates on the playback settings and when it is advised to tweak these settings in code.
Continue On Error :
There might be instances where some actions which we record during playback may or may not appear during playback. For ex: Say we record actions on ‘Save password dialog’ during recording. This dialog might not appear during playback. To prevent playback from failing on this action & to proceed with the other actions we use the ContinueOnError flag. This flag is set to false by default implying that we would fail whenever playback fails to perform any action. We can make playback to continue executing by turning on this flag. This is done by the following set property statement:
Playback.PlaybackSettings.ContinueOnError = true;
Note : Make sure to turn this flag back to false for actions where potential failures are not expected.
SetProperty Verification:
Every time playback does a set-property action (SetValue in a textbox, setstate of a checkbox etc) the playback verifies the state of the control after performing the action. If this verification fails then the action playback fails. There might be certain applications where doing a setvalue might trigger some logic by which the state of the control changes. In these cases, the playback verification fails. To avoid playback failing in these cases, we can temporarily disable SetProperty Verification by the following set property statement
Playback.PlaybackSettings.SkipSetPropertyVerification = true;
Note: Make sure to turn this flag back to false when you don’t expect the application to behave in the manner stated above.
Search Time Out & Search Fail Fast:
Search timeout indicates how long the playback should wait to locate the control. By default this is set to 2 minutes. In the case of search using IE plugin, we don’t hit this timeout as the IE plugin implements a retry mechanism. The IE plugin does a 3 time retry for searching a control. Search inturn calls WFR(Wait For Ready) to make sure the document is in a ready state. Once the document is in a ready state Search in IE Plugin is very fast as accessing the DOM using mshtml is very quick. So the time taken for performing 3 retries is much lesser than the default search timeout of 2 minutes. By this retry mechanism the IE Plugin is reasonably confident that the search will not succeed and it fails fast after 3 retries.
The user however has an option to disable fail fast. If you are sure that the control will become available after a specified duration you can increase the search time out & disable the fail fast mechanism.
Playback.PlaybackSettings.SearchTimeOut = 180000; // Value in milliseconds
Playback.PlaybackSettings.ShouldSearchFailFast = false; // This would disable fail fast fast and playback would wait for the full duration of search timeout for the control to be available
WaitForReadyTimeOut:
WaitForReady is a mechanism to ensure that the controls are being searched only when the document is in a state where user actions are possible. This is accomplished by tracking a number of parameters like timers in a page, pending ajax requests, percentage of navigation etc into consideration. By default the waitforready timeout is 60 seconds. There is a fail fast mechanism in WFR similar to search where if the state of the page isn’t in a ready state and the state doesn’t change for 50% of WFR timeout (30 seconds) we exit WFR and continue to search. There might be applications which might take a long time to receive the ajax responses, or might take a long time for navigation etc. In these cases we can increase the WFRTimeout to make sure the page is in a ready state thereby ensuring that the actions we perform on the page doesn’t fail. We can change the WFR timeout by the following setproperty statement:
Playback.PlaybackSettings.WaitForTimeOut = 120000; // Setting the timeout to 2 mins.
WaitForControlEnabled
This is actually not a playback setting. This is a setting specific to a UITestControl. By using this setting we can block the playback thread until the control becomes enabled. If you are sure that the control will get enabled after a delay we can use this setting.
UITestControl.WaitForControlEnabled(int millisecondsTimeout)
Blocks the current thread until the control is ready to accept mouse/keyboard input, or the specified timeout expires.
MatchExactHeirarchy
This setting is specified to indicate whether to match exact hierarchy specified for searching the control. Each control is specified as a hierarchy of containing controls for the search. If this flag is set to true, all ancestors of the control will be searched in order to locate this control. If this flag is set to false, some of the ancestors may be skipped during the search. For example, if the hierarchy of a control is A;B;C;D if A is found but playback fails to find full hierarchy B->C->D under it, then directly search for D under A if this property is set to false. This is set to false by default. To turn on this feature you can set it by the following setproperty statement.
Playback.PlaybackSettings.MatchExactHeirarchy = true;
ImeLanguageList
IMEs are components that allow the user to enter the thousands of different characters used in East Asian languages using a standard keyboard. The IME engine attempts to guess which ideographic character or characters the keystrokes should be converted into and lists them in a dropdown box. The aggregator in the recorder module aggregates the keyboard actions performed with IME enabled into an appropriate setvalue action. Playback uses copy paste to play back keyboard related actions for these language controls. The list of IME languages for which IME ends composition on pressing the {Enter} key that are included by default are : Chinese (People's Republic of China), Chinese (Taiwan), Japanese and Yi.
For UITest framework to support an IME language, we need to add the IME enabled language to the config settings by adding the Locale Id of the language to the comma separated list of IMELanguages node in the MTM.exe.config file.
SendKeysAsScanCode
Playback by default sends keys to any application as Unicode. It might be necessary to send keys as scancode to certain applications. We can do this by using an over load of Keyboard.Sendkeys function.
Workarounds using Playback Settings for special cases
The configuring of playback settings listed in the above section show how they can be manipulated in a CodedUITest project. At the same time there are certain settings that can be set globally in each machine. In case of MTM, the user can configure the playback settings by editing the mtm.exe.config file or CodedUITestBuilder.exe.config for CUIT present at %VSINSTALLDIR%\Common7\IDE.
The table lists the different playback settings, a high level description of the setting and whether the setting can be configured via Coded UI Test(CUIT) and Microsoft Test and Lab Manager(MTM).
Playback Setting |
Description |
API support |
MTM/CUIT settings file |
ContinueOnError |
Setting to ensure the playback continues execution incase of a failure to playback any action |
Yes |
No |
SetPropertyVerification |
Setting to Skip verification of action after the playback of a setvalue action |
Yes |
No |
SearchTimeOut |
Time for which the playback tries to search for a control. Default value is 2 minutes. |
Yes |
Yes |
ShouldSearchFailFast |
Setting to indicate if search should fail fast |
Yes |
No |
WaitForReadyTimeOut |
Time for which the playback waits for the IE document to enter a ready state. Default value is 1 minute. |
Yes |
Yes |
WaitForControlEnabled |
Setting to indicate the duration for which playback has to wait for the control to get enabled |
Yes |
No |
MatchExactHeirarcy |
Setting to indicate where the playback search should find all the controls in the hierarchy of the target control. By default it is set to false. |
Yes |
Yes |
SendKeysAsScanCode |
To send keys as scancode format as opposed to the default Unicode format. |
Yes |
No |
DelayBetweenActions |
This indicates the amount of time the playback has to wait between two successive actions. By default it is 100 ms. |
Yes |
No |
DelayBetweenSendKeys |
This indicates the delay between sendkeys of 2 characters. Default value is 10ms. |
Yes |
Yes |
GlobalHotKeys |
List of global hot keys that aggregator should not aggregating along with other keys |
No |
Yes |
RecordImplicitHover |
Recording hovers is very prevalent in IE. If user wants to turn off implicit hover, they can set this to false. User has to explicitely use Ctrl+Shft+R to record hover. Even this is customizable by using HoverKeyModifier and HoverKey keys. |
No |
Yes |
How to get resilient playback?
Recording
1) Determine whether you will launching the application along with the test case or the testcase assumes the AuT is already launched
2) If you are launching the application in the beginning of the testcase, then create the shortcut to the AuT in desktop, quick launch bar or in the start menu. This will ensure that Launch Application is recorded.
3) Ensure that AuT is ready to take user actions before doing the action
4) Remember that recording is one time operation so that do mindful about what actions are you recordingDuring recording watch the recorded steps in the MTM/CUIT Builder window and ensure that right set of actions got recorded. You can even delete actions if they are unwanted.
5) If you are recording actions on OS components please be aware that it may fail while playingback on other OS. The recorder aggregates certain actions automatically like start menu, task bar, quick launch etc to enable playback on other OS but not all actions are aggregated.
6) Create marker segments by clicking pass button in MTM or GenerateMethod button in CUITBuilder if you have accomplished a set of actions that are related and can be grouped as one entity. This way you can decide to playback only those portions during playback.
7) Use shared steps so that you can reuse the recording in many testcases. Maintainance will be easier.
8) Once the testcase is done determine you want to close the AuT as part of the testcase. If not, try to bring the AuT to a known state so that the next test case can be played back or recorded.
Playback
1) Based on the action log, determine whether the AuT is launched as part of playback or you have to keep the AuT opened before playback. In many cases this can be confusion and you may get a Seacrh failure.
2) If the AuT is launched as part of the testcase, then ensure that the AuT is installed in the same location as in the time of recording. Otherwise playback will fail to launch the application
3) Ensure that the state of the application is same as it was recorded.
4) Do not lock the machine or screen saver is not on, playback need an active session
5) Play back of the strip will fail, if an object on which a user performs action during recording is not available during playback. For example, if you are getting a dialog box in your scenario which has an option to not to show this dialog in future and you wish to choose that option, please pause the recording by clicking on pause button in MTM and check the “Do not show this dialog” option and choose Ok or Cancel. Basically if you record an action on a dialog box which only appears during recording, playback will fail because that dialog box will not show up second time which playback will expect. Remember password, Enable AutoComplete are some examples for this.
6) Please do not move your mouse or perform any actions during playback, as this might end up in playback failure.
Supported & Non-supported configurations
Open multiple instances of the application along with some stray windows to test window searching. |
||
Some actions may fail |
||
Same as above |
||
Dialogs that was not present during recording will fail playback |
Exceptions & what they mean
Playback on IE can fail due to various reasons and the playback engine throws the best exception out of a variety of playback exceptions for the user to rootcause the reason for the failure.
Recording Time Failures
Possible Exception |
Cause |
Comment |
TechnologyNotSupportedException |
Recoding on a unsupported technology App |
If the user tries to record on applications on IE whose version is below 7 or on 64-bit IE, the recorder throws a technology supported exception. This is same for Mozilla FireFox and SilverLight |
ApplicationFromNetworkShareException |
If the AuT is opened from a network share |
AuT has to be launched from the local drive not from a network share |
Playback time failures
Playback does two major things in order to perform action on the control search and action on the control. Following are the possible reasons and their exceptions.
SearchFailure
Following are some of the suggested tips to debug the search failure during playback.
- See screenshot (stored in %temp%\uitestlogs\lastrun if logs are enabled) to see whether the control was actually visible or not.
- Verify the search properties and UI hierarchy (QueryID) using IE Developer Toolbar.
Possible Exception |
Cause |
Comment |
UITestControlNotFoundException |
The control does not exist or search properties are not enough to find the control. |
Using the Accessibility tools find out whether the control hierarchy changed or control properties changed. |
UITestControlNotAvailableException |
The control does not exist in the app now. |
Whenever the playback performs an action on a control, it is cached for future references. When the playback tries to act on this cached control which is no longer available, the playback throws this exception. |
Action Failure
Sometime though the search passes but the playback may not be able to do the action on the control. Following may be the possible reason/tips to debug the issue
- Verify if there is any other control with the same properties under the parent mentioned in the QueryID. Some time there may be more than one control with the same properties under a parent hence during playback the first control will be selected and the action may fail on it. User can work around it after adding the instance property in the code for the ambiguous control.
- User can use .DrawHighLight() method to actually see which control is chosen for the UITestAction. This method is available for all the UITestControl and will draw a rectangular highlighter around the control.
- Verify whether the control state is same as it was during recording e.g., SetValue was recorded on a text box but during playback the text box is read only.
Possible Exception |
Cause |
Comment/Possible workarounds |
FailedToLaunchApplicationException |
When the playback fails to launch the AuT |
Verify whether the LaunchApp was recorded on the original exes or on some other file e.g., sometime if a App launches SPLASH screen before the original AuT LaunchApp is recorded on splash screen, user can work around this in API playback by changing the path |
ActionNotSupportedOnDisabledControlException |
Playback is trying to perform action on a disabled control |
This is thrown when the playback try to do some action on the disabled control/read only controls. Following are some of the cases: · SetValue on a read only combobox/text box · Click/SetState on disabled buttons/checkboxes |
FailedToPerformActionOnHiddenControlException |
When the playback tries to act on a control which is present but is not visible |
A modal dialog might be blocking the control or there could be a UI bug where the control is hidden in the UI. |
FailedToPerformActionOnBlockedControlException |
The control exists but not actionable |
Thrown when a control is blocked in this case though the search passes but the required action will not be played back. Following might be some of the causes for this exception · The control is not visible in the view port and not scrollable even · Control is blocked due to some modal dialog on top of it |
DecodingFailedException |
May come in password text box |
When the decoding of encrypted password fails due to incorrect key value. |
PlaybackFailureException |
Default fall back exception for uncategorized issues |
For any other failure the playback engine would throw PlaybackFailureException. For example if Get/Set of some property fails for a control playback engine would throw PlaybackFailureException. |
Enable Trace logs for troubleshooting
Sometimes the exceptions thrown by the UITest framework may not be enough to debug a failure. In those instances you can enable RnP trace logs which contains debugging information to troubleshoot failures. Below blog contains more information on how to enable trace logs.
https://blogs.msdn.com/gautamg/archive/2009/11/29/how-to-enable-tracing-for-ui-test-components.aspx
Summary
In this blog we saw the UITest framework engine of VS 2010 which is used in Fast Forward and CodedUITest features. There will be more blogs that will be followed to explain each technology that this record & play engine supports.