Learn about cool things you can do with UI Automation, and help teams around you at the same time
We can use the Windows UI Automation (UIA) API to build some really interesting tools which can help people in all sorts of ways. I have a bunch of UIA samples up at MSDN, which demonstrate different aspects of the API. As it happens, I've started using one of my own samples as a tool to help me at work. I've found that I've sent a few e-mails with typos in them, (for example, words ending with "ed" instead of "ing",) and I can miss these when re-reading e-mails before I sent them. These typos can be a real distraction when a recipient reads the e-mail, and there are some e-mails where I definitely don't want to the message to be lost because of typos in the content. So I downloaded my e-mail reading sample and tweaked it to work with Outlook instead of Windows Live Mail. Now, prior to sending some e-mails, I have my sample tool speak the contents of my e-mail to me. This nearly always leads me to update the e-mail before sending it, because it either highlights mistakes I've made, or makes it clearer to me that the content doesn't flow as smoothly as I'd like. So I now have another useful tool at my disposal, and it's all thanks to UIA.
As you start getting familiar with UIA yourself in preparation for building your own useful tools, you might consider helping app teams around you improve the quality of their products at the same time. You could write a little UIA code to learn how the Windows UIA API works, and with the results of that code, you might be able to find issues with the UI in apps near you. For example, perhaps your tool could quickly find Edit fields in forms which would be unusable to a customer who's blind, or apps with Button controls which can't be invoked by a customer who uses a screen reader with touch to interact with an app.
I should say that while taking the opportunity to have the results of your UIA experiments such that they can be useful to the teams around you might be worthwhile, you'll really want to include the powerful Windows SDK tools in your app verification process in order to find accessibility issues in your apps. For example, the AccChecker tool can quickly find a wide range of issues in UI, and so it's definitely worth learning how to get the most out of that tool.
But say you want to learn how to use UIA to get a set of properties from a collection of elements in your UI, and importantly - you want to achieve all this with a single cross-process call. Making many cross-process can really slow your app down, and so UIA's ability to cache lots of information during a single call to the target app is extremely useful.
To illustrate this, the code below shows how you can gather various properties from specific UI element types, and then examine the properties to see whether there's anything you want to draw attention to. To make sure this code worked, I built a simple app which used it. The app populated ListViews with the results of the checks it made, and when the user selected an item in the list, the element in the target app with the issue was highlighted on the screen using the bounding rect property which UIA had cached for me.
// Get the UIA Element at position (0, 0) on the screen.
tagPOINT point = new tagPOINT();
point.x = 0;
point.y = 0;
IUIAutomationElement elementWindow = _automation.ElementFromPoint(point);
if (elementWindow != null)
{
// We're only interested in Buttons, Lists, Text, Edit and Custom controls,
// so create a condition which will limit our search to only those types of
// controls.
IUIAutomationCondition conditionButton =
_automation.CreatePropertyCondition(_propertyIdControlType, _controlTypeIdButton);
IUIAutomationCondition conditionList =
_automation.CreatePropertyCondition(_propertyIdControlType, _controlTypeIdList);
IUIAutomationCondition conditionEdit =
_automation.CreatePropertyCondition(_propertyIdControlType, _controlTypeIdEdit);
IUIAutomationCondition conditionCustom =
_automation.CreatePropertyCondition(_propertyIdControlType, _controlTypeIdCustom);
IUIAutomationCondition conditionText =
_automation.CreatePropertyCondition(_propertyIdControlType, _controlTypeIdText);
IUIAutomationCondition condition1 =
_automation.CreateOrCondition(conditionButton, conditionList);
IUIAutomationCondition condition2 =
_automation.CreateOrCondition(condition1, conditionEdit);
IUIAutomationCondition condition3 =
_automation.CreateOrCondition(condition2, conditionText);
IUIAutomationCondition conditionControlType =
_automation.CreateOrCondition(condition3, conditionCustom);
// We're also only interested in elements which are on the screen, (which means
// the IsOffScreen property is false). So add that to the condition we'll use.
IUIAutomationCondition conditionIsOffScreen =
_automation.CreatePropertyCondition(_propertyIdIsOffscreen, false);
IUIAutomationCondition conditionControlVisible =
_automation.CreateAndCondition(conditionControlType, conditionIsOffScreen);
// Finally, we're only interested in controls which are in the UIA Control View.
// Those controls are easy for screen reader users to reach. So update our UIA
// condition to account for that.
IUIAutomationCondition conditionControlView =
_automation.ControlViewCondition;
IUIAutomationCondition conditionFinal =
_automation.CreateAndCondition(conditionControlVisible, conditionControlView);
// Create a cache request with all the properties that we want to examine after we've
// found the elements we're interested in. By using a cache, when we access the
// properties later, we'll not be making cross-process calls back to the app which
// presents the UI of interest.
IUIAutomationCacheRequest cacheRequest1 = _automation.CreateCacheRequest();
cacheRequest1.AddProperty(_propertyIdName);
cacheRequest1.AddProperty(_propertyIdControlType);
cacheRequest1.AddProperty(_propertyIdAutomationId);
cacheRequest1.AddProperty(_propertyIdBoundingRectangle);
cacheRequest1.AddProperty(_propertyIdIsExpandCollapsePatternAvailable);
cacheRequest1.AddProperty(_propertyIdIInvokePatternAvailable);
cacheRequest1.AddProperty(_propertyIdIsTogglePatternAvailable);
IUIAutomationCacheRequest parentCacheRequest = _automation.CreateCacheRequest();
parentCacheRequest.AddProperty(_propertyIdName);
// Now make the one and only cross-process call we make. This will return the
// collection of elements that meet the conditions we defined above.
IUIAutomationElementArray elementArray =
elementWindow.FindAllBuildCache(TreeScope.TreeScope_Descendants,
conditionFinal, cacheRequest1);
for (int i = 0; i < elementArray.Length; i++)
{
IUIAutomationElement element = elementArray.GetElement(i);
if (element.CachedName.Length == 0)
{
// The element has no name, so add it to our list of results.
AddItemToResultsList(listViewResults, element,
controlNameDictionary[element.CachedControlType]);
}
else if (element.CachedControlType == _controlTypeIdButton)
{
if (!element.GetCachedPropertyValue(
_propertyIdIsExpandCollapsePatternAvailable) &&
!element.GetCachedPropertyValue(
_propertyIdIInvokePatternAvailable) &&
!element.GetCachedPropertyValue(
_propertyIdIsTogglePatternAvailable))
{
// A button is not programmatically actionable.
AddItemToResultsList(listViewNotInvokableButtons, element,
element.CachedName);
}
}
else if (element.CachedControlType == _controlTypeIdText)
{
if ((element.CachedName.Length == 1) && (element.CachedName[0] >= 0xE000))
{
// A Text element has a name that might be unspeakable. (For example a
// glyph shown on a button). NOTE: This check is not valid for some
// worldwide languages.
AddItemToResultsList(listViewUnspeakableText, element,
element.CachedName);
}
}
}
}
I pointed my new tool to a test XAML app which contained a TextBox. I'd not given the TextBox an accessible name, and sure enough, the tool found the problem. The image below shows the tool highlighting the problem in the test app's UI.
In order for my tool to be able to appear above a Windows Store app's UI, I did need to specify that the app request UIAccess privileges in its app manifest. (In order for UIAccess to be granted, I also need to digitally sign my exe and run it from a secure folder.)
I've posted some options for fixing XAML apps which exhibit the accessibility issues detected by the code above, at Solutions to some common accessibility issues found in XAML apps.
Finally, I would stress how useful the SDK tools can be for detecting accessibility issues in apps. Experimenting with the code above can be a fun way to learn UIA while helping raise the quality of apps around you, but you can't beat pointing tools like AccChecker and Inspect to UI. Those tools can really help you deliver a great experience for all your customers, regardless of how those customers interact with your UI.