Jaa

Finding the handle of the window that contains a UI Automation element

  • Artikkeli

Hi,

Someone asked a great UIA-related question earlier today at the UIA Forum. They pointed out that UIA has an API to get a UIA element associated with a Win32 window handle, (hwnd). This can be done by calling the ElementFromHandle() method. But what happens if you already have a UIA element, and you want to get the hwnd most closely associated with that element? There's no such thing in UIA as HandleFromElement() to get the hwnd that contains the UI element.

And a note on hwnds and UIA elements - with UIA you can have a whole hierarchy of UIA elements representing some piece of UI, and there might be only one hwnd associated with it. That hwnd would be associated with a UIA element at the root of the hierarchy, and that element would expose a property with the value of the hwnd. It can be interesting to point the Inspect SDK tool at some UI, and see which UIA elements have an hwnd associated with them. The UIA_NativeWindowHandlePropertyId property on the UIA element represents the hwnd associated with the element, if there is one.

For example, say I point the Inspect tool to a list item in the Start Screen in Windows 8.1. Inspect tells me that the UIA element representing the list item does not expose a UIA_NativeWindowHandlePropertyId property, and so this means there's no hwnd directly associated with that UIA element. 

Figure 1: Inspect reporting that a UIA element for a list item does not expose the UIA_NativeWindowHandlePropertyId property.

 

As I follow the ancestor chain up from the UIA element for the list item, I find the first ancestor that has a UIA_NativeWindowHandlePropertyId property is the UIA element associated with the Start Screen itself. 

 

Figure 2: Inspect reporting that the UIA element for the Start Screen does expose the UIA_NativeWindowHandlePropertyId property.

 

So this brings us to the question asked at the UIA forum. How can the hwnd associated with the ancestor be found, starting from the descendant UIA element, and in the most performant way possible? Below was my response to that question.

Thanks,

Guy

 

This is a really interesting question.

In this situation, it's tempting to try calling something like FindFirstBuildCache() passing in TreeScope.TreeScope_Ancestors and a condition to say that we want the first ancestor who has a NativeWindowHandle which is not zero. But as  https://msdn.microsoft.com/en-us/library/windows/desktop/ee696030(v=vs.85).aspx points out, TreeScope_Ancestors isn't a valid scope for a call to FindFirstBuildCache(). Similarly it might be tempting to call BuildUpdatedCache() on the element, with a cacheRequest whose scope is TreeScope_Ancestors, to build up a cache of ancestors' NativeWindowHandles in a single call. But as  https://msdn.microsoft.com/en-us/library/windows/desktop/ee684019(v=vs.85).aspx says, TreeScope_Ancestors can't be specified when setting up a cache request.

In some cases it might be possible to use Win32 calls to access the hwnd of interest. For example, if you know the UIA element is visible on the screen, and is not obscured, then based on the element's bounding rectangle, a call to the Win32 WindowFromPoint() could return whatever hwnd is at the point. But in general, that's not going to be a reliable approach because bounding rect's could overlap with other bounding rects, or the element of interest could be completely obscured by another window.

So the most reliable approach will be to use a TreeWalker. But as you point out, this has potential to impact performance. If the UIA element that you start off with is many levels deep beneath the ancestor element with the NativeWindowHandle, then you'll be making a cross-process call with every call to GetParentElementBuildCache(). While that would eventually get you the element you're after, there is an approach with a better performance.

You can call NormalizeElementBuildCache() to have the ancestor found, and its properties of interest returned, with a single cross-proc call. The details at https://msdn.microsoft.com/en-us/library/windows/desktop/ee671483(v=vs.85).aspx mention how the function can be useful when accessing an element of interest after hit-testing, but it seems useful in any situation where you want to access a particular ancestor.

I just wrote the following test code...

 

// For this test, just get the focused element as the starting element.
IUIAutomationElement elementOfInterest = _automation.GetFocusedElement();

// Build up a condition to access elements whose NativeWindowHandle is not zero.
int propertyIdWindowHandle = 30020; // UIA_NativeWindowHandlePropertyId

IntPtr propValueWindow = IntPtr.Zero;

IUIAutomationCondition conditionNativeWindowHandleZero = 
    _automation.CreatePropertyCondition(
        propertyIdWindowHandle, propValueWindow);

IUIAutomationCondition conditionNativeWindowHandleNOTZero = 
    _automation.CreateNotCondition(conditionNativeWindowHandleZero);

// Now create a TreeWalker based on that condition.
IUIAutomationTreeWalker treeWalker = 
    _automation.CreateTreeWalker(conditionNativeWindowHandleNOTZero);

// Set up a cache request such that when we find an ancestor of interest,
// we don't have to make more cross-process calls to get the data we're
// interested in. For this test, cache the name and bounding rect.

int propertyIdBoundingRectangle = 30001; // UIA_BoundingRectanglePropertyId
int propertyIdName = 30005; // UIA_NamePropertyId

IUIAutomationCacheRequest cacheRequestTest = 
    _automation.CreateCacheRequest();

cacheRequestTest.AddProperty(propertyIdName);
cacheRequestTest.AddProperty(propertyIdBoundingRectangle);

// Given that we only need the cached data, don't have UIA keep a
// live reference around to the ancestor.
cacheRequestTest.AutomationElementMode = 
    AutomationElementMode.AutomationElementMode_None;

// Now get the first ancestor with a not-zero NativeWindowHandle.
IUIAutomationElement elementAncestorWithWindow = 
    treeWalker.NormalizeElementBuildCache(
        elementOfInterest, cacheRequestTest);

Debug.WriteLine("Barker test: ancestor with window handle is \"" + 
    elementAncestorWithWindow.CachedName + "\"");

 

When I ran that code, I moved keyboard focus around the various descendant elements within the TaskBar, Start Screen, Task Switcher and Alarms app. When I did that, I found the ancestors returned had the names "Running applications", "Start menu", "Task Switching" and "Alarms".

So this approach seemed to access and return all the data I needed with a single cross-proc call, keeping the app high-performance.

Comments

  • Anonymous
    February 04, 2016
    Thanks for your code. I got tripped up on CreatePropertyCondition which was failing.   Found that the VARIANT type for window handle is VT_I4 I had tried to use a VARIANT of type VT_INT_PTR, thinking as this example alluded to, that the window handle could be 64-bits on 64-bit Windows.   After all an HWND is a HANDLE which is a PVOID.   However, this page says that Window handles are always 32-bit. msdn.microsoft.com/.../aa384203%28v=vs.85%29.aspx Also, the docs for the UI Automation properties mentions VT_I4 for the window handle property. msdn.microsoft.com/.../ee684017(v=vs.85).aspx