Windows 7 UI Automation Client API C# sample (e-mail reader) Version 1.0
In preparation for a demo of the native-code Windows UI Automation (UIA) API last year, I built a sample app which introduces most of the commonly used features of UIA. This included the following topics; properties, events, patterns, cache requests and conditions. The original C++ sample is up at https://code.msdn.microsoft.com/Windows-7-UI-Automation-9131f729, and a version I ported to C# is at https://code.msdn.microsoft.com/Windows-7-UI-Automation-0625f55e.
Probably the most interesting functionality that’s not mentioned in that sample relates to the Text Pattern. If a UIA provider supports the Text Pattern, then a client can perform a number of operations to move through the text shown in the provider and to access that text. For example, the client can move through the text, examining the text character-by-character, word-by-word, line-by-line or paragraph-by-paragraph. Details of the Text Pattern are up at https://msdn.microsoft.com/en-us/library/windows/desktop/ee815688(v=vs.85).aspx.
So I thought it would be interesting to build a new sample that uses the Text Pattern. As I considered this, I wondered whether I could build a very simple app which hardly does anything, but could still be a useful assistive technology (AT) tool in practice. While UIA is great for enabling UI automated tests, the API is also designed to support people building AT, and so I’m always very excited to build tools which relate to a real-world AT scenarios. (For example, building the tool to speak the text shown on the OSK prediction keys, https://code.msdn.microsoft.com/Windows-7-UI-Automation-433a961f.)
With this in mind, I decided that a future sample would demo some of the more powerful aspects of the Text Pattern, but my next sample would do the absolute minimum required to have the Text Pattern support a useful tool. What I settled on was an app to read e-mail text. Some people find it easier to comprehend an e-mail message if the text is spoken to them as they read it. And sometimes when important e-mail is being composed, if it’s spoken prior to being sent, that can help to detect issues with the text which aren’t caught by running a spell checker alone. So the new sample app should read both received e-mail and e-mail being composed.
I decided to build the app in C# and my first step was to copy one of my existing C# samples, https://code.msdn.microsoft.com/Windows-7-UI-Automation-6390614a. I then removed a few existing things which weren’t relevant to the new app, (for example the UIA event handler, and use of the Magnification API.) One handy aspect about my new app was that I didn’t have to account for the UIA threading rules. That is, if an app uses UIA to access its own UI, or it has UIA event handlers, then certain things must be done on background threads, (as described in my other samples). But since the new app doesn’t do any of that, it keeps things really simple.
I then updated the .cs file for the main app WinForm UI. There’s little work done here other than to use a SpeechSynthesizer to speak the text using the current default voice on the computer, and also to change the visuals for a button to start or stop the speaking of the text.
The more interesting .cs file is the one that actually uses UIA. It initializes and uninitializes an IUIAutomation object, (as any UIA client app does), and then has a single function to access the e-mail text. That function looks like this:
// Try to find a Windows Live Mail window for composing and reading e-mails.
// Using the Spy tool, the class of the main window can be found. This test
// app assumes there's only one Windows Live Mail window of interest.
IntPtr hwnd = Win32.FindWindow("ATH_Note", null);
if (hwnd != IntPtr.Zero)
{
// We found a window, so get the UIA element associated with the window.
IUIAutomationElement elementMailAppWindow = _automation.ElementFromHandle(
hwnd);
// Find an element somewhere beneath the main window element in the UIA
// tree which represents the main area where the e-mail content is shown.
// Using the Inspect SDK tool, we can see that the main e-mail content
// is contained within an element whose accessible name is "NoteWindow".
// So create a condition such that the FindFirst() call below will only
// return an element if its name is "NoteWindow".
IUIAutomationCondition conditionNote = _automation.CreatePropertyCondition(
_propertyIdName, "NoteWindow");
IUIAutomationElement elementNoteWindow = elementMailAppWindow.FindFirst(
TreeScope.TreeScope_Descendants,
conditionNote);
// As it happens, the actual element that supports the Text Pattern is
// somewhere beneath the "NoteWindow" element in the UIA tree. Using
// Inspect we can see that there is an element that supports the
// Text Pattern. Once we have that element, we can avoid a cross-process
// call to access the Text Pattern object by using cache request.
IUIAutomationCacheRequest cacheRequest = _automation.CreateCacheRequest();
cacheRequest.AddPattern(_patternIdTextPattern);
// Now find the element that supports the Text Pattern. This test app assumes
// there’s only one element that can be returned which supports the Text Pattern.
bool fTextPatternSupported = true;
IUIAutomationCondition conditionTextPattern = _automation.CreatePropertyCondition(
_propertyIdIsTextPatternAvailable,
fTextPatternSupported);
IUIAutomationElement elementMailContent = elementMailAppWindow.FindFirstBuildCache(
TreeScope.TreeScope_Descendants,
conditionTextPattern,
cacheRequest);
// Because the Text Pattern object is cached, we don't have to make a cross-process
// call here to get object. Later calls which actually use methods and properties
// on the Text Pattern object will incur cross-proc calls.
IUIAutomationTextPattern textPattern = (IUIAutomationTextPattern)
elementMailContent.GetCachedPattern(
_patternIdTextPattern);
// This test app is only interested in getting all the e-mail text, so we get that through
// the DocumentRange property. A more fully featured app might be interested in getting a
// a collection of Text Ranges from the e-mail. Each range might relate to an individual
// word or paragraph. Given that a provider which supports the Text Pattern allows a
// client to find the bounding rectangles of these ranges, the client could choose to use
// its own method of highlighting the text as the text is being spoken.
IUIAutomationTextRange rangeDocument = textPattern.DocumentRange;
// Pass -1 here because we're not interested in limiting the amount of text here.
strMailContent = rangeDocument.GetText(-1);
}
This app specifically only works with Windows Live Mail, but with a small adjustment the code above could target other window-based e-mail apps.
As Text Pattern samples go, this isn’t very interesting as all it does is access the text for the entire document as a single string, (through the DocumentRange property.) But I did find this an exciting experiment. In around an hour and a half I built a UIA app which contains very little code, but which could still be a useful AT tool for some people. My next step is to figure out which other parts of the Text Pattern and Text Range interfaces would help most in practice for an e-mail related AT like this, and to update the sample accordingly. For example, the AT tool might get call IUIAutomationTextRange::Move() to access each line in turn, and call GetBoundingRectangles() and render its own highlighting for the line being spoken. The app could also call IUIAutomationTextPattern::GetVisibleRanges() to make sure it only gathers text that’s shown on the screen when working with e-mail clients that handle hidden text.
The e-mail reading sample app is out at https://code.msdn.microsoft.com/Windows-7-UI-Automation-28b2b537.