UIA Custom Patterns: Part 3
In part 2, I talked about how to declare and implement custom methods in UI Automation and promised I would do one more part to wrap it all up. I'll introduce two new objects that play an important role in custom patterns and describe how to register the custom patterns. The samples for this post are here.
I used this image in the last post, but it's worth repeating here:
The Client Pattern Instance Object
When a client of your custom pattern uses the pattern, the client just sees the interface, like IValuePattern (in the case of the standard UIA pattern) or IColorPattern, the example I've been using in this series. But interfaces don't come from nowhere - they have to exist on some object. We'll call that object the client pattern instance object, and you have to implement it. Since UI Automation doesn't know what your custom interface will be, there's no way for it to implement an unknown interface. The sole responsibility of the client pattern instance object is to implement your custom interface. You can see it on the left side of the diagram above.
When I implemented two client pattern instance objects, I noted some similarities, so my sample uses a common base class, called CustomClientInstanceBase. The base class does two jobs:
- Store the IUIAutomationPatternInstance pointer, which is the interface that your client pattern instance object will need to do its job.
- Provide some helper methods to make retrieving custom properties and calling custom methods easier.
Here's one of the helper methods:
// Get a current property value for this custom property
protected object GetCurrentPropertyValue(UiaPropertyInfoHelper propInfo)
{
UiaParameterHelper param = new UiaParameterHelper(propInfo.UiaType);
this.patternInstance.GetProperty(propInfo.Index, 0 /* fCached */,
param.GetUiaType(), param.Data);
return param.Value;
}
In this case, I noticed that getting a custom property involves pretty similar steps: declaring a UIAutomationParameter struct, initializing it, and calling IUIAutomationPatternInstance.GetProperty() with the right index and structure pointers. So it seemed like a good idea to encapsulate it.
But here's a real example of a client pattern instance object, the one for my Color Pattern example:
// Pattern instance class: wrap up a IUIAutomationPatternInstance and implement the
// custom pattern interface on top of it.
public class ColorProviderClientInstance : CustomClientInstanceBase, IColorPattern
{
public ColorProviderClientInstance(IUIAutomationPatternInstance patternInstance) : base(patternInstance)
{
}
#region IColorPattern Members
public int CurrentValueAsColor
{
get
{
return (int)GetCurrentPropertyValue(ColorSchema.GetInstance().ValueAsColorProperty);
}
}
public int CachedValueAsColor
{
get
{
return (int)GetCachedPropertyValue(ColorSchema.GetInstance().ValueAsColorProperty);
}
}
public void SetValueAsColor(int value)
{
CallMethod(ColorSchema.GetInstance().SetValueAsColorMethod, value);
}
#endregion
}
As you can see, it isn't very long. It derives from CustomPatternInstanceBase for convenience and then implements IColorPattern. Visual Studio will helpfully add empty bodies for the methods for me if I ask nicely, and then I implement the methods by calling through to either IUIAutomationPatternInstance.GetProperty() or IUIAutomationPatternInstance.CallMethod().
[It occurs to me that the sample might have had more educational value without the base class -- you could see more easily exactly what each method is doing -- but it would have been a lot longer and it wouldn't have been the way that I actually write code. I guess removing the base class can be left as an exercise to the reader.]
So, there's my client pattern instance object.
The Pattern Handler Object
The other object you will need to implement is the pattern handler object. This object implements the IUIAutomationPatternHandler interface and has two responsibilities:
- Create client pattern instance objects when requested -- always in the client process.
- Dispatch method calls to the provider -- always in the provider process.
You can see this object on the right side of the diagram above.
Creating client pattern instances is trivially easy. Dispatching method calls is a little more tricky, but I talked about this in the previous part of this series: it's just a matter of taking the incoming parameters from UIA, passing them to your custom provider, and then packing up any out-parameters.
This is simple enough that I can just paste the whole implementation:
/// <summary>
/// Pattern handler class: creates pattern instances on client side and dispatches
/// calls on the provider side.
/// </summary>
public class ColorProviderHandler : Interop.UIAutomationCore.IUIAutomationPatternHandler
{
#region IUIAutomationPatternHandler Members
public void CreateClientWrapper(Interop.UIAutomationCore.IUIAutomationPatternInstance pPatternInstance, out object pClientWrapper)
{
pClientWrapper = new ColorProviderClientInstance(pPatternInstance);
}
public void Dispatch(object pTarget, uint index, IntPtr pParams, uint cParams)
{
// Parse the provider and parameter list
IColorProvider provider = (IColorProvider)pTarget;
UiaParameterListHelper paramList = new UiaParameterListHelper(pParams, cParams);
// Dispatch the method/property calls
if (index == ColorSchema.GetInstance().ValueAsColorProperty.Index)
{
paramList[0] = provider.ValueAsColor;
}
else if (index == ColorSchema.GetInstance().SetValueAsColorMethod.Index)
{
provider.SetValueAsColor((int)paramList[0]);
}
else
{
throw new InvalidOperationException();
}
}
#endregion
}
Now that I have this object done, I have both the client side and provider side done, so I can register my pattern.
Registering a Custom Pattern
To register a custom pattern, you just call IUIAutomationRegistrar.RegisterPattern. You only need to do this once in the lifetime of the process. You pass a UIAutomationPatternInfo structure full of information about your pattern:
- A GUID to identify your pattern.
- A human-readable programmatic name for the pattern.
- The client and provider IIDs for the custom pattern interfaces.
- A list of custom properties for the pattern, as an array of UAutomationPropertyInfostructs.
- A list of custom methods for the pattern, as an array of UIAutomationMethodInfostructs.
- A list of custom events for the pattern, as an array of UIAutomationEventInfostructs.
- A pointer to a single instance of your custom pattern handler object. (Since it is stateless, only one instance is needed.)
RegisterPattern() will then return an integer pattern ID assigned to your custom pattern, a property ID for the IsXxxxPatternAvailable property (every good pattern has one!), a list of assigned property IDs for your custom properties, and a list of assigned event IDs for your custom events.
(Why isn't there a list of method IDs, too? Well, clients and providers need to know the event IDs when they call RaiseAutomationEvent(), and they need to know property IDs if they call IUIAutomationElement.GetCurrentPropertyValue. But clients and providers never use method IDs -- they just call the methods directly on the interfaces. So, it is enough to just number the properties from [0 ... M-1], if there are M properties, and then methods from [M .. M+N-1], if there are N methods. We call these numbers method indices. Only the client pattern instance object, the pattern handler, and UIA ever see the method indices, so there's no need to return them to the caller during pattern registration)
At this point, you are done. Clients can query for your custom pattern, UIA will make the connection between the client process and the provider process, and your pattern handler will be dispatching calls on the provider side.
A Few Useful Notes
While building this sample, I noticed a few subtle things worth mentioning.
Custom Pattern Schemas are a Kind of Singleton
When working with custom patterns, there is a set of data that you use exactly once, like the registration data and outputs. The pattern ID that you get when you register a custom pattern is essentially a global variable - it will not change for the lifetime of your process, but it also isn't a constant. This reminded me of the Singleton design pattern, which is what I used. Again, I found that some repetitive tasks were best handled by a base schema class, and then the custom patterns had a schema that derived from it.
The base schema class mostly defines what it means to be a custom pattern schema, and also encapsulates the registration procedure:
/// <summary>
/// Base class for defining a custom schema.
/// Responsible for defining the minimum info for a custom schema and
/// registering it with UI Automation.
///
/// This class is not required by UIA and doesn't correspond to anything in UIA;
/// it's a personal preference about the right way to represent what is similar
/// between various schemas and what varies.
/// </summary>
abstract public class CustomPatternSchemaBase
{
// The abstract properties define the minimal data needed to express
// a custom pattern.
/// <summary>
/// The list of properties for this pattern.
/// </summary>
abstract public UiaPropertyInfoHelper[] Properties { get; }
/// <summary>
/// The list of methods for this pattern.
/// </summary>
abstract public UiaMethodInfoHelper[] Methods { get; }
/// <summary>
/// The list of events for this pattern.
/// </summary>
abstract public UiaEventInfoHelper[] Events { get; }
/// <summary>
/// The unique ID for this pattern.
/// </summary>
abstract public Guid PatternGuid { get; }
/// <summary>
/// The interface ID for the COM interface for this pattern on the client side.
/// </summary>
abstract public Guid PatternClientGuid { get; }
/// <summary>
/// The interface ID for the COM interface for this pattern on the provider side.
/// </summary>
abstract public Guid PatternProviderGuid { get; }
/// <summary>
/// The programmatic name for this pattern.
/// </summary>
abstract public string PatternName { get; }
/// <summary>
/// An object that implements IUIAutomationPatternHandler to handle
/// dispatching and client-pattern creation for this pattern
/// </summary>
abstract public IUIAutomationPatternHandler Handler { get; }
/// <summary>
/// The assigned ID for this pattern.
/// </summary>
public int PatternId
{
get { return patternId; }
}
/// <summary>
/// The assigned ID for the IsXxxxPatternAvailable property.
/// </summary>
public int PatternAvailablePropertyId
{
get { return patternAvailablePropertyId; }
}
/// <summary>
/// Helper method to register this pattern.
/// </summary>
public void Register()
{
if (!this.registered)
{
// Get our pointer to the registrar
Interop.UIAutomationCore.IUIAutomationRegistrar registrar =
new Interop.UIAutomationCore.CUIAutomationRegistrarClass();
// Set up the pattern struct
UiaPatternInfoHelper patternInfo = new UiaPatternInfoHelper(
this.PatternGuid,
this.PatternName,
this.PatternClientGuid,
this.PatternProviderGuid,
this.Handler
);
// Populate it with properties and methods
uint index = 0;
foreach (UiaPropertyInfoHelper propertyInfo in this.Properties)
{
patternInfo.AddProperty(propertyInfo);
propertyInfo.Index = index++;
}
foreach (UiaMethodInfoHelper methodInfo in this.Methods)
{
patternInfo.AddMethod(methodInfo);
methodInfo.Index = index++;
}
// Add the events, too, although they are not indexed
foreach (UiaEventInfoHelper eventInfo in this.Events)
{
patternInfo.AddEvent(eventInfo);
}
// Register the pattern
int[] propertyIds;
int[] eventIds;
UiaPatternInfoHelper.RegisterPattern(registrar,
patternInfo,
out this.patternId,
out this.patternAvailablePropertyId,
out propertyIds,
out eventIds);
// Write the property IDs back
for (int i = 0; i < propertyIds.Length; ++i)
{
this.Properties[i].PropertyId = propertyIds[i];
}
for (int i = 0; i < eventIds.Length; ++i)
{
this.Events[i].EventId = eventIds[i];
}
this.registered = true;
}
}
int patternId;
int patternAvailablePropertyId;
bool registered;
}
The classes with names like UiaPropertyInfoHelper are wrappers that I wrote for the official UIAutomationPropertyInfo struct to take care of initialization and cleanup. I also found that they were a convenient place to store the registered property ID, once I registered the pattern.
My custom property schema for Color Pattern looks like this:
/// <summary>
/// Declaration of the pattern schema, with all of the information UIA needs
/// about this property.
/// </summary>
public class ColorSchema : CustomPatternSchemaBase
{
static readonly ColorSchema instance = new ColorSchema();
static public ColorSchema GetInstance()
{
return ColorSchema.instance;
}
public readonly UiaPropertyInfoHelper ValueAsColorProperty =
new UiaPropertyInfoHelper(
new Guid("48F45D48-37A1-4480-B5A7-198315D2F2A0"),
"ValueAsColor",
UIAutomationType.UIAutomationType_Int);
public readonly UiaMethodInfoHelper SetValueAsColorMethod =
new UiaMethodInfoHelper(
"SetValueAsColor",
true /* doSetFocus */,
new UiaParameterDescription[] {
new UiaParameterDescription("value", UIAutomationType.UIAutomationType_Int)
});
public override UiaPropertyInfoHelper[] Properties
{
get { return new UiaPropertyInfoHelper[] { ValueAsColorProperty }; }
}
public override UiaMethodInfoHelper[] Methods
{
get { return new UiaMethodInfoHelper[] { SetValueAsColorMethod }; }
}
public override UiaEventInfoHelper[] Events
{
get { return new UiaEventInfoHelper[] { }; }
}
public override Guid PatternGuid
{
get { return new Guid("CDF2D932-6043-47ef-AB48-1CA756678B0C"); }
}
public override Guid PatternClientGuid
{
get { return typeof(IColorPattern).GUID; }
}
public override Guid PatternProviderGuid
{
get { return typeof(IColorProvider).GUID; }
}
public override string PatternName
{
get { return "ColorPattern"; }
}
public override IUIAutomationPatternHandler Handler
{
get { return new ColorProviderHandler(); }
}
};
As you can see, this is just filling in the slots that the base schema left. I liked this: it felt like a clean division of labor between what was common between custom patterns and what varied, and enabled me to avoid repeating myself.
To be sure, you don't have to do it this way. Global variables are completely legitimate in C++ to do this job. But in C#, this felt like a reasonable way to do it.
Five basic datatypes
To save you some trouble, I should note that I found five basic datatypes work for custom method parameters:
- int
- bool
- double
- string (BSTR),
- UI Automation element (IUIAutomationElement/IRawElementProviderSimple).
UIAutomationType seems to show more usable types than this, but some of them are not supported for custom properties and methods.
These five do give you the ability to do almost anything you want -- you could take any data type and have your client pattern instance object and Dispatch method cooperate to serialize/de-serialize to a string using an algorithm like UUEncode/UUDecode.
General properties and pattern properties
In part 1 of this series, we talked about registering custom properties, which a provider implements in their IRawElementProviderSimple.GetPropertyValue() method. Let's call them general properties. You can create as many of these as you want. You can also have custom properties that are part of your pattern interfaces, like my ValueAsColor property. Let's call these pattern properties.
It took me a while to realize that pattern properties are not dispatched through GetPropertyValue(). Instead, they are dispatched through your pattern handler's dispatch method as though they were a call to a method with a single out-parameter of the correct type. Like so:
public void Dispatch(object pTarget, uint index, IntPtr pParams, uint cParams)
{
// Parse the provider and parameter list
IColorProvider provider = (IColorProvider)pTarget;
UiaParameterListHelper paramList = new UiaParameterListHelper(pParams, cParams);
// Dispatch the method/property calls
if (index == ColorSchema.GetInstance().ValueAsColorProperty.Index)
{
paramList[0] = provider.ValueAsColor;
}
So, the client can access these properties in two different ways: the client can call IUIAutomationElement.GetCurrentPropertyValue or the client can call the appropriate method on the pattern interface, like IColorPattern.get_CurrentValueAsColor. But either way, the call gets dispatched through the Dispatch() method on the provider side. On reflection, this makes sense: it would be inconvenient to have to implement the property in two different functions just to support the client's two different ways of asking. But it did take me some time to sort that out.
Also, I ran into a limitation in the current implementation of UI Automation: you cannot have more than two pattern properties. UI Automation will allow you to register a pattern with more than two pattern properties, but they will not work properly. I found a workaround for this issue: I created additional methods that each had a single out-parameter and used those methods (call them psuedo-properties) just as if they were real properties. You can see how this works in the sample file called TestPattern.cs.
Wrapping Up
The sample on Code Gallery has two full custom patterns (the Color Pattern described in this series and a Test Pattern that I used to try out various datatypes) plus a set of unit tests for them. The unit tests helped me while coding, and also serve as samples of how to consume custom patterns. The Code Gallery sample also proves that UIA Custom Patterns can be used from managed code. (They are easier to use from native code, but hopefully my samples can be re-used to make managed code usage equally straightforward.) The next time that you have extra Accessibility data that doesn't fit well into UI Automation's existing design, or you need special test-automation hooks, consider using UIA Custom Patterns!
Comments
Anonymous
April 17, 2011
Fantastic articleAnonymous
July 22, 2011
Hi Michael, I am very grateful for the tutorial. It has been a big help. However, I'm having trouble getting the two Interop projects in the example solution to build. Is there any advice you could offer for setting up the build in VS 2010 ThanksAnonymous
July 25, 2011
Keynan, You'll need to have the Windows 7 SDK installed. I also noticed that Visual Studio 2010 seems to ignore my custom build steps. But if you look at UiaCoreInterop.vcproj, you can see the five steps that are used to build, starting with MIDL. UIAutomationCore.tlb should be generated by MIDL.exe from the Windows 7 SDK version of UIAutomationCore.idl, by the first step in UiaCoreInterop. You can try walking the steps in UiaCoreInterop.vcproj by hand – the command lines are all in there. Thanks, MichaelAnonymous
July 26, 2011
The comment has been removedAnonymous
July 26, 2011
The comment has been removedAnonymous
July 25, 2013
Thank you for this tutorial, but i do have a problem running the sample. I created the Interop.UIAutomationCore.dll as you suggested. Unfortunately any time i tried to run the sample, i get NullReferenceException (when and attempt is made to register ColorPattern): // Call register pattern registrar.RegisterPattern( ref patternData, out patternId, out patternAvailableId, patternData.cProperties, pPropertyIds, patternData.cEvents, pEventIds); I've tried to debug it but couldn't go past this exception. I also searched by registry for to see if required classes are on the machine. I couldn't find entries for "IID_IUIAutomationRegistrar is defined as 8609c4ec-4a1a-4d88-a357-5a66e060e1cf" nor "CUIAutomationRegistrar". I'm using Win2008R2. I also installed update but still get same exception. other UIA functionalties are fine via UIACOMWrapperAnonymous
October 23, 2013
@Yomi I had same problem and setting Platform=x86 solved it. Seems that COM object can't be created from x64 code.