using UIAutomation

Artikkeli
06/27/2006

With Beta 2 most developers inside Microsoft has started playing with Vista. Among a lot of uber-cool features one of the interesting feature is UIAutomation. This is the next-gen assistive/UI-Automation technology that succeeds Active Accessibility (MSAA).

I tried it out a bit and I liked its power and reach. However, the programming model is very unconventional and un-intuitive. I really hope that the folks have done enough user-study to understand if its understandable by average developers.

Lets take a look.

Why UIA

I don't have the official statement but my guess would be that MSAA failed to keep up with the expectations. MSAA served its purpose well. However it was designed for assistive purposes to help out visually impaired people or people with other disabilities. Soon everyone started using it to drive/automate UI and ran into all sorts of limitations. The fact that it was not documented well added to the confusion and let to lot of incorrectly implemented piece of UI.

UIA was designed grounds up to serve both automation and assistive needs.

What is UIA

Its a layer over all MS UI technologies and provides a single interface to automate all kinds of user interface. You can potentially write generic code to automate an application and run the same bits to drive an application written in Avalon (WPF), WinForm, ASP.NET (inside IE), Win32 etc. WPF controls implement UIA natively and all others implement MSAA and UIA reads into all of them and provides a seamless experience.

UIA has a plugin provider model where you can register client-side providers which acts as bridges and allows UIA to read into any assistive technology out there. So potentially you can create a bridge that'd allows UIA to read into Java Accessible applications. Currently it works with WPF, Win32, MSAA, Office controls. Since a lot of frameworks like Flash supports MSAA, it'd should work with UIA seamlessly.

Features

Multiple features make UIA very interesting. Specially the fact that it represents the whole desktop as a singly rooted tree. Its not relevant which technology is used to implement an UI, everything is expressed in terms of an AutomationElement in the desktop tree. So Visual Studio which is a native Win32 IDE comes under the desktop node and all the managed pieces inside VS appear as its children. Using crossbow if you host XAML (Avalon) inside a WinForm control it'll appear seamlessly in that tree. All the background work of converting technology specific details are done by UIA.

UIA also provides way to search the tree to locate nodes based on search criteria. It has it own query mechanism.

UIA supports notification of user actions using Events similar to the MSAA WinEvents.

Code

The primary step of automating UI is to locate specific controls that match a search criteria and then drive it. We'll first define three overloads of a function named GetElement that locates a control based on criteria's passed to it

 //Search for a elment just by name starting from root
private static AutomationElement GetElement(AutomationElement root, string name, bool recursive)
{
    PropertyCondition condName = new PropertyCondition(AutomationElement.NameProperty, name);
    return root.FindFirst(recursive ? TreeScope.Descendants : TreeScope.Children, condName);
}
//Search for the first occurance of a given type of control
private static AutomationElement GetElement(AutomationElement root, ControlType controlType)
{
    PropertyCondition condType = new PropertyCondition(AutomationElement.ControlTypeProperty, controlType);
    return root.FindFirst(TreeScope.Descendants, condType);
}

So the first one gets me any control with a given name (as in label, text) and the second gives me a control of a given type like ControlType.Button

Even though I don't use it here, its trivial to combine multiple conditions. So in case I wanted to use a condition that combines both of the above I would have done

 AndCondition andCond = new AndCondition(condName, condType);

Another interesting feature in UIA is that each of the AutomationElements supports patterns. The supported patterns vary from control to control and can be used to drive them. So a button has a invoke pattern which is equivalent to clicking it and edit boxes have a Value pattern that can be used to set values in them. So the following clicks on a button and sets the value of a edit box

 AutomationElement btn = GetElement(...);InvokePattern invPattern = btn.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;invPattern.Invoke();AutomationElement edit = GetElement(...);ValuePattern valPattern = edit.GetCurrentPattern(ValuePattern.Pattern) as ValuePattern;valPattern.SetValue("abc");

So with all of the above I can locate any button in say a calculator window and drive it. We can use the following code to drive calculator to do the calculation 7 * 7 - 7 = 42. At the end we'll also verify that the result in the calculator indeed matched the expected the most important answer 42. The code expects to have calculator already running.

 static void Main(string[] args)
{
    // Locate calculator window
    AutomationElement calculator = GetElement(AutomationElement.RootElement, "Calculator", false);
    // locate the button 7
    AutomationElement btn7 = GetElement(calculator, "7", true);
    InvokePattern btn7InvPat = btn7.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
    btn7InvPat.Invoke();
    // locate the button *
    AutomationElement btnMult = GetElement(calculator, "*", true);
    InvokePattern btnMultInvPat = btnMult.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
    btnMultInvPat.Invoke();
    // hit on 7 again
    btn7InvPat.Invoke();
    // locate and invoke -
    AutomationElement btnMinus = GetElement(calculator, "-", true);
    InvokePattern btnMinusInvPat = btnMinus.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
    btnMinusInvPat.Invoke();
    // hit on 7 again
    btn7InvPat.Invoke();
    // locate and invoke =
    AutomationElement btnEq = GetElement(calculator, "=", true);
    InvokePattern btnEqInvPat = btnEq.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
    btnEqInvPat.Invoke();
    // get the result edit box and verify the value is indeed 42
    AutomationElement editResult = GetElement(calculator, ControlType.Edit);
    string result = editResult.GetCurrentPropertyValue(ValuePattern.ValueProperty, false).ToString();
    Debug.Assert(result != "42.", result);
}

Code Quality

The UIA code looks very ugly and is not strongly typed. With generics in place there is no excuse for writing a framework that requires casting almost every other line. I think I'll use another post to discuss why I do not like the code.

Comments

Anonymous
July 24, 2006
Very interesting. Thanks a lot.
Anonymous
September 15, 2006
Thanks, Mr . abhinaba for wonderful information

I would be thankful if more resources & coding are available for UIA.

Regards
sachin
Anonymous
October 30, 2006
Dear Mr abhinaba,I am a tester. I use UIA in my test program. And I describe all the UI controls in my product by a Excel file. The test program reads the Excel file and knows how to find each control, so that in test case logic, no need to add any code for finding UI controls.But maybe it's more effective if I use XAML instead of Excel?Do you think developer and test programmer should use the same XAML?Regards,Friendkey
Anonymous
April 03, 2007
Based on what I've seen in this code, the only way to query for a given control is by it's name. Is there any way to achieve the same thing, querying by a control ID?Thanks so much,Chris
Anonymous
April 04, 2007
Hi there. Thank you so much for the info on UIA! That was a great post with some very useful code tidbits.I'm having some problems adding a reference for System.Windows.Automation.First of all, I'm running Windows XP SP2, and I have installed .NET 3.0, Windows SDK, and VS Extensions. I couldn't find System.Windows.dll anywhere. So I found a post that said, use WindowsBase.dll.That I do have. But unfortunately, when I try to add it as a reference in my project, it says that it's not a valid dll... I'm frustrated! :)Do you think you could help me figure this one out?Thanks so much,Chris
Anonymous
April 04, 2007
Chris, UIAutomation provides you a generic way to search and not "only by name". You can use AutomationElement.AutomationIdProperty for using Control ID for search.To solve you dll assembly reference issue. You need the following dllsUIAutomationClient.dllUIAutomationClientsideProviders.dllUIAutomationProvider.dllUIAutomationTypes.dllWindowsBase.dllAll of these are in GAC. Try adding all the reference and let me know if you still hit an issue
Anonymous
April 25, 2007
Hi,Please help me. I'm stuck up in this problem. I see all the above DLLs in the GAC. but not able to add the reference. what do I do? :(
Anonymous
May 09, 2007
Using UI Automation code I want to drive the Word 2007 UI.How will i do that?Let me know if any sample avaliable for Word 2007.
Anonymous
June 14, 2007
Hi Abhinaba, thanks for the update. I tried using the "AutomationElement.AutomationIdProperty" and tried to specify the control ID of the button "1" on the calculator (following your example). The control ID is, "0000007D", but it doesn't seem to find it...Any thoughts?Thanks,Chris
Anonymous
June 14, 2007
I'll answer my own question... you need to pass in the decimal equivalent to the hex value. It doesn't like the hex value. But the AutomationIdProperty works...
Anonymous
June 19, 2007
To Reference the assembly the UIAutomationClient.dll is located in the C:Program FilesReference AssembliesMicrosoftFrameworkv3.0 folder. You just have to browse to it.
Anonymous
October 15, 2007
Hi everyone, How can I discover Menu Strips using UISPY, and work with UI Automation? Another question, what is the best practice to wait the window open? I am using: while (myAutomationElement==null) { myAutomationElement=SearchForElement(blah bla) delay(1000) }
Anonymous
December 14, 2007
The comment has been removed
Anonymous
December 14, 2007
Thats way more complex than you think. Actually you summarized the feature of the product I'm working on :) UIA events are volutarily fired by UI and hence cannot be relied on. You should be looking into other techniques like hooking...
Anonymous
February 13, 2008
Hi, I too was trying to use UIAutomation to test my teams application. It's an application that mixes WPF and WinForms so UIAutomation seemed like a good approach. However we got stuck on simple things like waiting for objects etc. Some time ago I stumbled across a testing framework that uses UIAutomation among other technologies. The framework allowed us to record tests from inside Visual Studio and the way they designed the framewrork we can separate all UI mapping from the test logic. And the best part is that everything is recorded in plain .NET code. We've been in their beta program for some time now and that's a good idea if you want some support for your specific application testing needs. You can check it out at http://www.testautomationfx.com where the have a simple movie that shows of some of the features. They seem to be pretty good at answering support questions by mail but be sure to request a place in the beta program. The approach we use requires the developers to write some tests and it seems to be much easier now when they can do it in c#.
Anonymous
February 14, 2008
Hi, Some of our UI Controls are using infragistics. in my test GetElement can only find normal Controls. Do you have an idea how to get elements of infragistics controls?
Anonymous
February 14, 2008
Today UIA for IE is working over the minimal MSAA implementation that IE exposes. Especially for Infragistics kind of controls this is not sufficient. You'd need to create an IE plugin (or provider in UIA terminology) for UIA and then use over it. This plugin needs to be an adapter layer adapting the IE-DOM to the UIA plugin interface.
Anonymous
April 28, 2008
Is there anyway to catch the global cursor with a specific ControlType or anything like that. I've read that MSAA roles (ROLE_SYSTEM_CURSOR between them) has been substituted but I cannot find how to catch global cursor with UIAutomation. Thanks!
Anonymous
January 21, 2009
PingBack from http://www.keyongtech.com/1502697-required-word-2007-ui-automation
Anonymous
January 21, 2010
HI, Excelent Article. You have any Idea about OUTLOOK 2007 UI Automation. If Yes please reply to kuvanka@gmail.com Thanks, Sivaprasad
Anonymous
January 21, 2010
HI, Excelent Article. You have any Idea about OUTLOOK 2007 UI Automation. If Yes please reply to kuvanka@gmail.com Thanks, Sivaprasad
Anonymous
January 21, 2010
http://social.msdn.microsoft.com/Forums/en-US/windowsaccessibilityandautomation/thread/75bf4e62-b459-4bf6-976c-33feb450e36a
Anonymous
November 01, 2012
Hi Abhinava, Is there anyway to create automation peer element In VBA.I am automating one of my application using WPF in VBA.UI spy is not identifying one of my object.Could you please help in this regard?

Jaa

using UIAutomation

Comments

Lisäresursseja