Custom UI Automation Providers in Depth: Part 1

Creating a UI Automation provider for a custom control is not a difficult task. At least, we members of the UI Automation team don’t think so. Yet, it is one that engineers continue to find challenging. And so it seems worth writing a walkthrough. I’ll also post the full code samples on the MSDN Code Gallery so that you can explore and re-use them.

I am Michael Bernstein, the development lead for the Accessibility team in Windows. I want to start with a quick word about the philosophy of what we’re trying to do here. Programmatic accessibility is really all about taking a user interface and turning it into a programming interface. I’m sure you’ve all used programmatic interfaces of various kinds: library functions, APIs, web services, etc. So, how would you query a word processing program for data – like you can do with a database? Or how would you issue commands to a Control Panel – as you would to a web service? The problem is, the Control Panel or word processor doesn’t have anything to code against. So, that’s the problem we’re trying to solve.

We’ll need a particular example for our walkthrough. Most of the standard UI controls, like buttons and combo boxes, already have UI Automation interfaces. But a custom control would not. So, I created a custom control: the TriColor control. It’s a bit silly, but bear with me. The TriColor control lets you select one of three colors: Red, Yellow, or Green. It would be fabulous for an application that controlled a traffic light. Clicking on a color moves the selection, and it has some basic keyboard support for the up and down arrows to do the same thing. It’s not a complicated control, but it’s good enough to demonstrate the concepts here.

We’ll also need something to test how we’re doing. One of the most classic Accessibility tools is the screen reader, but using a screen reader for testing adds a layer between you and the programmatic interface you are creating. You’d like to see the interface itself. The tool our team uses most often is called Inspect Objects, and you can find it in the Windows 7 Platform SDK. So, if you don’t have that SDK, download it and make sure you can run the Inspect tool. The Inspect tool defaults to following both focus and cursor position, so hover your mouse over anything to inspect its properties. The menu item Options | Show Highlight Rectangle is very useful to show you exactly what you are inspecting. If you haven’t tried this before, inspect various parts of your desktop and see what kinds of properties show up.

Experiment: Go to the MSDN Code Gallery site and get the UIAControls-Part1.zip file.  Compile the TricolorControl program, run it, and inspect it with Inspect. What do you see?

Here’s what I see:

When I do it, I notice that even though I have done nothing with UI Automation yet, I do see a few Accessibility properties:

• The Control Type is ‘pane.’ (That just means a container.)
• The bounding rectangle is accurate.
• The RuntimeID property is there. (It’s correct, too.)
• The IsKeyboardFocusable and HasKeyboardFocus properties are both true. (If the program has focus.)

Well, that’s a good trick – I got UIA properties for free! How did that happen? Well, UIA can see that the control has a window handle, and the window itself has a bunch of properties. UIA picks these up as defaults. This is convenient – it saves you the trouble of providing them. (Although you can override them, as we’ll see.) The defaults are provided by the HWND provider, which exists for every control with a window handle.

However, we’re missing a whole bunch of information:

• What is the control’s name?
• The ControlType is wrong – this isn’t just a window pane. What is it really?
• What value is currently set (Red, Yellow, Green)?
• Where are the red, yellow, and green regions? How do they relate to the whole control?
• Can I change the value?

To answer these questions, we need to extend the control’s programmatic interface. And we’ll do that by implementing a UI Automation for the control, in the next post.

One last word about language: my sample uses C# and WinForms. WinForms is close enough to the Windows native APIs that the techniques I show here can be translated easily into C++. But the syntactic sugar of C# makes the examples shorter and easier to follow. When I’m doing something in C# that doesn’t translate easily in C++, I’ll point it out. I’m also not using WPF. WPF has a different system: it does a bunch of UI Automation work on behalf of regular controls and even custom controls. This is great – go WPF! – but it would undermine what I’m trying to demonstrate here. The MSDN content on implementing UI Automation in WPF is quite good, so read that if that’s your framework.

Next time: Creating the “Hello world!” of UI Automation providers.

Comments

• Anonymous
  October 04, 2010
  what are the basic functionalities of automation testing of custom controls?

• Anonymous
  July 17, 2013
  Hi Michael, I'm new for UIA, when I download your sample code, I cannot build it successfully and it seems that the project missing references interop.uiautomationcore.dll and interop.uiaautomationclient.dll. For interop.uiautomationcore.dll, I can resolve it following the post stackoverflow.com/.../using-c-sharp-and-ui-automation-to-grab-contents-of-unknown-control-type, but for interop.uiaautomationclient.dll, it does not work using the same way. Could you hlep to let me know how to make the project works? PS. I used VS2010 and covered the project when opening it. Thanks a lot!

• Anonymous
  July 10, 2014
  Hello, the link to sample code "UIACustomProviders" does not work anymore. Would you please update the links? Thank you!

• Anonymous
  April 19, 2015
  the link to sample code "UIACustomProviders" still doesn't work. Would you please update the links? thanks

• Anonymous
  October 19, 2015
  The code samples built for this series of posts are no longer available. If you'd like to learn about building a UI Automation Provider, these samples will be of interest: code.msdn.microsoft.com/UI-Automation-Simple-735d36a5 code.msdn.microsoft.com/UI-Automation-Fragment-6f0aaf20 code.msdn.microsoft.com/UIA-Document-Content-00f77d1e

  • Anonymous
    March 10, 2017
    Links to others parts of this article:https://blogs.msdn.microsoft.com/winuiautomation/2010/05/12/custom-ui-automation-providers-in-depth-part-2/https://blogs.msdn.microsoft.com/winuiautomation/2010/06/04/custom-ui-automation-providers-in-depth-part-3/https://blogs.msdn.microsoft.com/winuiautomation/2010/06/04/custom-ui-automation-providers-in-depth-part-4/https://blogs.msdn.microsoft.com/winuiautomation/2010/07/02/custom-ui-automation-providers-in-depth-part-5/https://blogs.msdn.microsoft.com/winuiautomation/2010/07/27/custom-ui-automation-providers-in-depth-part-6/

• Anonymous
  November 12, 2017
  The link to the sample code is still not working.However, I found this forum entryhttps://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/5afea385-fc4f-464b-a8c6-e658f9463ab6/sample-code-available-for-custom-uia-patterns-in-net?forum=windowsaccessibilityandautomationin which another developer (Ivan Danilov) provides a link to a modified version of the samplehttps://github.com/TestStack/uia-custom-pattern-managedThis might be a useful alternative to Michael's original code.I will check it out.