UI Automation in Silverlight - Simulating User Interactions
I was recently tasked with automating Silverlight Rich Internet Applications (RIAs) in our immediate group. Some tools out there provide limited assistance in this regard; for example, you can write unit tests against your Silverlight controls for in-proc testing. You can read more about that approach here: https://www.jeff.wilcox.name/2008/03/31/silverlight2-unit-testing/
Unfortunately, my requirement is to enable scenario automation. We must simulate a user that goes through several user flows both within and outside of our RIA; move the mouse somewhere, click on a thing, go to a 3rd party authentication provider, start typing some keys, and so on. The Silverlight unit testing framework doesn't quite address this requirement.
UI automation in Silverlight has been a hotly followed topic around here. With the release of Silverlight Beta 2, we started seeing some accessibility stubs come into play. More correctly, we started seeing the WPF way of doing UI automation start to trickle in. If you want to follow along, you're going to want to grab UISpy (https://blogs.msdn.com/windowssdk/archive/2008/02/18/where-is-uispy-exe.aspx).
The Microsoft UI Automation (UIA) assemblies were released with the .NET Framework 3.0. Traditionally, we've had various COM wrappers to work with Microsoft Active Accessibility (MSAA), which isn't going to get you very far with your Silverlight application. If you've done any sort of UI automation in WPF, then you're going to feel right at home. So, without dwelling on that further, let's start working with UIA!
First, let's start a new Test project in Visual Studio 2008. The UIA namespaces that you'll need to start automating applications in Silverlight are System.Windows.Automation and System.Windows.Automation.Providers. You'll need to add references to these assemblies that ship with .NET 3.0+ to get the relevant parts:
- UIAutomationProvider.dll
- UIAutomationClient.dll
- UIAutomationClientsideProviders.dll
- UIAutomationTypes.dll
Let's say that we need to automate a Silverlight control that we own. We'll need to override the OnCreateAutomationPeer method (from the Control class) to return our own Peer type that handles the accessibility functions of the control. This is important, because the accessibility functions will be key in letting us automate our application.
Assume a hypothetical Search control that consists of a text box and a search button:
public partial class SearchBar : Control
{
...
public SearchBar()
{
this.GotFocus += (sender, args)
=>
{
this.SearchText.Focus();
};
InitializeComponent();
}
protected override AutomationPeer OnCreateAutomationPeer()
{
return new SearchBarAutomationPeer(this);
}
}
We've overridden the OnCreateAutomationPeer, which will get called by anything that's inspecting your control tree for accessibility functions (and consequently, your automation functions). The Peer object will be responsible for returning your combination of controls in a manner that is coherent to anything that needs accessibility.
In the process of doing so, we've also wired up the GotFocus handler to set up our controls' default .Focus() behavior.
Let's take a look at what we need to implement the SearchBarAutomationPeer class:
public class SearchBarAutomationPeer : FrameworkElementAutomationPeer, IValueProvider
{
public SearchBarAutomationPeer(SearchBar searchBar) : base(searchBar)
{
}
Our Peer class needs to derive from FrameworkElementautomationPeer to provide all the methods that we're going to need to work with. IValueProvider maps out to interacting with the TextBox component of our custom control. You can learn more about the Provider interface mapping to individual components here: https://msdn.microsoft.com/en-us/library/system.windows.automation.provider.aspx
We need to give our control a class name and an accessibility identifier to find it in the control tree. To do this, we must override GetAutomationIdCore() and GetClassNameCore() from FrameworkElementAutomationPeer.
protected override string GetAutomationIdCore()
{
return "SearchBar"; // You're going to want to make this unique. ;)
}
protected override string GetClassNameCore()
{
return "SearchBar";
}
protected override bool IsKeyboardFocusableCore()
{
return true;
}
IsKeyboardFocusableCore is an important override to add in, as well; without it, our calls to SetFocus() on the control will fail. We should also think about implementing our Provider interface. The SearchBar that we passed into our constructor maps out to the base.Owner property. Casting base.Owner to SearchBar is going to get tedius, so we'll add a property to make working with that easier as well.
public SearchBar SearchBar
{
get
{
return (SearchBar)base.Owner;
}
}
#region IValueProvider Members
public bool IsReadOnly
{
get
{
return this.SearchBar.SearchText.IsReadOnly;
}
}
public void SetValue(string value)
{
this.SearchBar.SearchText.Text = value;
}
public string Value
{
get
{
return this.SearchBar.SearchText.Text;
}
}
#endregion
If we take a look at our control in UISpy, it should now look something like this:
Identification
ClassName: "SearchBar"
ControlType: "ControlType.Custom"
Culture: "(null)"
AutomationId: "SearchBar"
LocalizedControlType: "custom"
Name: "SearchBar"
ProcessId: "2276 (iexplore)"
RuntimeId: "42 197110 6"
IsPassword: "False"
IsControlElement: "True"
IsContentElement: "True"
Visibility
BoundingRectangle: "(356, 286, 949, 36)"
ClickablePoint: "830,304"
IsOffscreen: "False"
ControlPatterns
Value
Value: ""
IsReadOnly: "False"
The "Value" property under ControlPatterns automagically comes from the IValueProvider interface, mapping out to the value of our underlying TextBox. Slick, huh?
So we've done some plumbing to enable our Silverlight control. Let's take a look at our test method looks like:
[TestMethod]
public void TestMethod1()
{
Process process = System.Diagnostics.Process.GetProcessesByName("iexplore").First();
AutomationElement browserInstance = System.Windows.Automation.AutomationElement.FromHandle(process.MainWindowHandle);
TreeWalker tw = new TreeWalker(new PropertyCondition(AutomationElement.ClassNameProperty, "SearchBar"));
AutomationElement searchBar = tw.GetFirstChild(browserInstance);
myElement.SetFocus();
Thread.Sleep(1000);
searchBar.SetFocus();
Thread.Sleep(1000);
SendKeys.SendWait("Hello, world!");
}
You might be asking yourself a couple of questions at this point: "Why did I implement IValueProvider?" for example. Well, the snippet above simulates user input. If that isn't your thing, in comes the ValuePattern. As an aside, I found interacting with the ValuePattern/TryGetCurrentPattern/etc and found the whole experience to be a bit clunky. You can see what I mean below:
[TestMethod]
public void TestMethod1()
{
Process process = System.Diagnostics.Process.GetProcessesByName("iexplore").First();
AutomationElement myElement = System.Windows.Automation.AutomationElement.FromHandle(process.MainWindowHandle);
TreeWalker tw = new TreeWalker(new PropertyCondition(AutomationElement.ClassNameProperty, "SearchBar"));
AutomationElement searchBar = tw.GetFirstChild(myElement);
object valuePattern;
searchBar.TryGetCurrentPattern(ValuePattern.Pattern, out valuePattern);
((ValuePattern)valuePattern).SetValue("Hello, world!");
}
This is by no means a comprehensive guideline, but it should be enough to get those of you out there interested in UI automation going.
One caveat: applications with Windowless enabled show up as one huge control if you're looking in UISpy. Hopefully, support for accessibility (and subsequently, automation) will be in RTW builds of Silverlight. If you want to perform UI automation on your Silverlight application today, you'll have to do it without Windowless.
Comments
Anonymous
July 11, 2008
I work closely with Gabriel and to a very lesser extent helped with this implementation. I and another co-worker spent boundless hours doing research and investigation, and before I left for Parental Leave, came to the conclusion that with the current implementation and technology, just couldn't get anything tangible. I really, really hope this helps other teams both internally and externally as much as this is going to help us. Also, in theory, you should be able to build your peers in a seperate library if you need that kind of code isolation.Anonymous
July 15, 2008
Putting your AutomationPeer objects in a different project could potentially result in a circular dependency; CreateAutomationPeer would require the types in the external assembly and the external assembly would require the types in the base library for a proper override. It's probably best to bake them into your own control libraries. If you're extending someone else's library, then you shouldn't have much issue.Anonymous
July 20, 2008
The comment has been removedAnonymous
July 21, 2008
Thread.Sleep isn't a necessity, though keeping automation code synchronized with the UI is traditionally a problem. I haven't really hit any synchronization issues with the functionality in the UIA namespaces. The key problem I'm running into right now is that the AutomationPeer approach only gets you so far. For example, several classes I'd be interested in are sealed, so you can't throw in your own automation peer. And, as you noted, you can't really "listen" for events (though you can call things that should trigger events from the providers in the AutomationPeer).Anonymous
September 27, 2008
（摘要自：http://blogs.msdn.com/gisenberg/archive/2008/07/12/ui-automation-in-silverlight-simulating-user...Anonymous
January 08, 2009
Could you share a zip illustring all this? Thanks a lot!Anonymous
May 05, 2009
UI Automation/Accessibility in Silverlight 2, tools and resources summaryAnonymous
May 14, 2009
More options: http://www.artoftest.com/community/blogs/09-05-11/Intro_to_Silverlight_Automation.aspxAnonymous
May 27, 2009
Hello, I'm trying to follow your article by writing the code, but I'm not able to run the test. Do you happen to have working sample project? If so, is it possible to send it to me? My e-mail address is: tesicg@yahoo.com. Thank you in advance. GoranAnonymous
May 27, 2009
Can i get sample project for developing Automation peer for custom control. I am trying do this for Infragistics Grid Control. Please email me to veenashivaprasad@hotmail.comAnonymous
October 26, 2009
Thanks for this post. Please could you make the code downloadable?