UI Automation client doesn't see all dynamic elements of WPF application

Question

Hi there. I have a problem while automating a WPF App using Windows UI Automation client API. After attaching to the app process I can see all static visible elements on UI using Windows UI Automation client, but after expanding some multiple-level nested lists or re-arranging already existing UI elements (i.e. after filtering), some of new elements (including re-arranged already visible UI elements) are not visible to UI Automation client anymore. Inspect.exe tool doesn't see those elements as well. But those elements are visible to me on the screen and they work absolutely fine (I can click onto them, drag-and-drop them and so on). Basically all of problematic elements represent the DTOs received from the server side,i.e. they are dynamic elements. The static elements are always visible to UI Automation client.

Neither refresh of the whole element tree inside Inspect.exe tool, nor re-attaching to the process of the application helps. In some cases Inspect.exe sees the old element tree (even after clicking the "Refresh" button), which used to be there before the re-arranging of UI elements. Only restart of the application itself helps to let such elements be visible to both UI Automation client and Inspect.exe tool.

Those issues became more frequent after the Windows updates (and corresponding update of UIAutomationCore.dll) during the recent couple of months. The elements which were visible to UI Automation 2 months ago are now somehow fully invisible to its client and thus can't be interacted with using client API.

Does anyone have any idea or suggestion on how to find the root cause of this issue? It has most probably something to do with caching and/or refreshing the COM, because all static elements are always visible and accessible to both Windows UI Automation client and Inspect.exe tool. It could be something specific to WPF as well, but I haven't found any specific clues. Because restart of the application helps, it tells me that the root cause might be lying in the injected UIAutomationCore.dll.

Thanks in advance!

Answer 1

Hi,@Tarik_P.

This seems to be a problem with ItemsControl. I have made three different view switches for ListBox, but I could still get each data. But for ItemsControl, even if I don't switch its view, I could't get its elements through AutomationElement.

Three view switching tests： 1.Window1(Contains ListBox) and Window2 are switched in the following way:

public partial class Window1 : Window
{
private void Button_Click_1(object sender, RoutedEventArgs e)
{
   	 new Window2().Show();
  	  this.Close();
}
}

2. In MainWindow, switch between UserControl1(Contains ListBox) and UserControl2 in the following way:

<ContentControl x:Name="MyContentControl"></ContentControl>

public bool change = false;
private void Button_Click(object sender, RoutedEventArgs e)
{
  if (change)
 {
         MyContentControl.Content = new UserControl2();
  }
  else { 
         MyContentControl.Content = new UserControl1();
   }
  change = !change;
}

3.In MainWindow, switch between Page1(Contains ListBox) and Page2 in the following way:

<Frame x:Name="MyFrame" Grid.Row="0"></Frame>

public bool change = false;
private void Button_Click(object sender, RoutedEventArgs e)
{
            if (change) {
                MyFrame.Navigate(new Uri("Page1.xaml", UriKind.Relative));
            }
            else
            {
                MyFrame.Navigate(new Uri("Page2.xaml", UriKind.Relative));
            }
            change = !change;
}

I will describe the third switch in more detail here.

WPF Program--WPF App(.NET Framework). MainWindow.xaml

    <Grid>
        <Grid.RowDefinitions>
            <RowDefinition Height="7*"></RowDefinition>
            <RowDefinition Height="*"></RowDefinition>
        </Grid.RowDefinitions>
        <Frame x:Name="MyFrame" Grid.Row="0"></Frame>
        <Button  Grid.Row="1" HorizontalAlignment="Center" VerticalAlignment="Center" Width="100" Height="30" Content="Click Me" Click="Button_Click"></Button>
    </Grid>

MainWindow.xaml.cs

    public partial class MainWindow : Window
    {
        public bool change = false;
        public MainWindow()
        {
            InitializeComponent();
            MyFrame.Navigate(new Uri("Page1.xaml",UriKind.Relative));
        }

        private void Button_Click(object sender, RoutedEventArgs e)
        {
            if (change) {
                MyFrame.Navigate(new Uri("Page1.xaml", UriKind.Relative));
            }
            else
            {
                MyFrame.Navigate(new Uri("Page2.xaml", UriKind.Relative));
            }
            change = !change;
        }
    }

Page1.xaml

<Page 
      xmlns:vm="clr-namespace:WpfApp1.ViewModel"
>

    <Page.DataContext>
        <vm:Page1ViewModel></vm:Page1ViewModel>
    </Page.DataContext>
    <Page.Resources>
        <CollectionViewSource x:Name="MyList" x:Key="MyList" Source="{Binding list}"  Filter="CollectionViewSource_Filter"></CollectionViewSource>
    </Page.Resources>
    <StackPanel>
        <TextBlock Text="HAHA"></TextBlock>
        <ListBox x:Name="MyListBox" ItemsSource="{Binding Source={StaticResource MyList}}">
            <ListBox.ItemTemplate>
                <DataTemplate>
                    <Grid>
                        <Grid.ColumnDefinitions>
                            <ColumnDefinition Width="*"/>
                            <ColumnDefinition Width="*"/>
                            <ColumnDefinition Width="*"/>
                        </Grid.ColumnDefinitions>
                        <TextBlock Text="{Binding Id}" Margin="10,0" Grid.Column="0" Height="30" ></TextBlock>
                        <TextBlock Text="{Binding Name}" Margin="10,0" Grid.Column="1" Height="30"></TextBlock>
                        <TextBlock Text="{Binding Age}" Margin="10,0" Grid.Column="2" Height="30"></TextBlock>
                    </Grid>
                </DataTemplate>
            </ListBox.ItemTemplate>
        </ListBox>
        <Button x:Name="MyButton" Width="100" HorizontalAlignment="Left" Height="20" Content="Fileter" Click="Button_Click"></Button>
    </StackPanel>
</Page>

Page1.xaml.cs

public partial class Page1 : Page
{
    public Page1()
    {
        InitializeComponent();
        collectionViewSource = this.Resources["MyList"] as CollectionViewSource;
    }

    public bool isFilter = true;

    public CollectionViewSource collectionViewSource;

    private void CollectionViewSource_Filter(object sender, FilterEventArgs e)
    {
        if (e.Item is Person person && person.Age <= 20)
            e.Accepted = true;
        else
            e.Accepted = false;
    }

    private void Button_Click(object sender, RoutedEventArgs e)
    {
        isFilter = !isFilter;
        if (isFilter)
            collectionViewSource.Filter += CollectionViewSource_Filter;
        else
            collectionViewSource.Filter -= CollectionViewSource_Filter;
    }
}

Page2.xaml

<StackPanel>
    <TextBlock x:Name="MyTextBlock" Text="AA"></TextBlock>
    <Button Content="Click Me" Click="Button_Click"></Button>
</StackPanel>

Page2.xaml.cs

public partial class Page2 : Page
{
    public Page2()
    {
        InitializeComponent();
    }

    private void Button_Click(object sender, RoutedEventArgs e)
    {
        MyTextBlock.Text = "BB";
    }
}

Page1ViewModel.cs

    public class Page1ViewModel
    {
        public ObservableCollection<Person> list { get; set; } = new ObservableCollection<Person>()
        {
             new Person(){Id=1,Name="AA",Age=18},
             new Person(){Id=2,Name="BB",Age=19},
             new Person(){Id=3,Name="CC",Age=20},
             new Person(){Id=4,Name="DD",Age=21},
             new Person(){Id=5,Name="EE",Age=22},
        };
    }

Person.cs

    public class Person
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public int Age { get; set; } 
    }

UI Automation Client Program--Console App(.NET Framework)

class Program
{
    static void Main(string[] args)
    {
        // Waiting for WPF
        Thread.Sleep(5000);

        // Get the root element of the desktop
        AutomationElement desktop = AutomationElement.RootElement;
        // Find the main window of the target WPF application
        AutomationElement mainWindow = desktop.FindFirst(TreeScope.Children,
            new PropertyCondition(AutomationElement.NameProperty, "MainWindow"));  

        if (mainWindow!=null)
        {
            while (true)
            {
               
                Console.WriteLine("-----------------------------------------------------------------------------------------");

                // Find the button and click it(Dynamically load filters via a button)
                AutomationElement button = mainWindow.FindFirst(TreeScope.Descendants,
                    new PropertyCondition(AutomationElement.AutomationIdProperty, "MyButton"));

                if (button != null)
                {
                    InvokePattern invokePattern = button.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
                    invokePattern?.Invoke();
                }
                else
                {
                    Console.WriteLine("Unable to find Button");
                }
               
                AutomationElement page = mainWindow.FindFirst(TreeScope.Descendants, new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Pane));
                AutomationElement listBox = page.FindFirst(TreeScope.Descendants, new PropertyCondition(AutomationElement.AutomationIdProperty, "MyListBox"));

                if (listBox != null)
                {   // Get all items in the ListBox
                    AutomationElementCollection listItems = listBox.FindAll(TreeScope.Children, new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.ListItem));
                    // Print the value of each item
                    foreach (AutomationElement listItem in listItems)
                    {
                        AutomationElementCollection textItems = listItem.FindAll(TreeScope.Descendants, new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Text));

                        foreach (AutomationElement textItem in textItems)
                        {
                            Console.Write(textItem.Current.Name + "  ");
                        }
                        Console.WriteLine();
                    }
                }
                else
                {
                    Console.WriteLine("Unable to find ListBox");
                }

                //Execute every five seconds
                Thread.Sleep(10000);
            }
        }
        else
        {
            Console.WriteLine("Unable to find target application window");
        }
    }
}

In the test, ItemsControl cannot be obtained even if the view is not switched. This situation is also described in this document. You could consider using ListBox or use the method described in the document to achieve the effect you want.

---

If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.