Using Reflection to automate testing of an .exe file

项目
12/19/2007

Hi all,

My name is Kirill Osenkov and I am a new member of the C# IDE QA team. I'm hoping to post articles related to C#, testing and other interesting areas to our teams blog. I also have a personal blog over at https://kirillosenkov.blogspot.com, where I post random thoughts about .NET, design and developer tools.

Today I'd like to discuss a technique to programmatically access the internals of .NET executables, and invoke them from code, and not through the user interface. We'll use reflection to test one of the C# LINQ samples that ship with Visual Studio 2008.

Why do samples need automated testing anyway?

If you open VS 2008 and go to Help -> Samples -> Samples on Disk, you'll see a link to a .zip file containing all of the C# samples, similar to this: \Program Files\Microsoft Visual Studio 9.0\Samples\1033\CSharpSamples.zip. In folder LinqSamples, there is a WinFormsDataBinding project, and we'd like to write a test for this sample that ensures that the sample works fine.

First, you might ask, why bother and write a test for the samples at all? You can just open it, hit F5, and spend 20 seconds to test the sample functionality. Well, during the development of a feature such as LINQ a lot of things can change often - the syntax, the libraries etc. You don't want to manually open each sample and verify that it still works every time our developers change a feature. It is nice to have a set of automated tests that can ensure:

if the sample still works, this means that the changes to the product feature (LINQ) haven't introduced new bugs in most common scenarios (samples actually are perfect tests for common usage scenarios!).
if the sample is broken, we'll investigate and either find a bug in the feature or update the sample to correctly use the feature.

In any case, we won't forget to update the samples to correctly work with the latest version of the product, and if the sample tests pass, this is generally a good indication of a working product - awesome for quick sanity check of main functionality.

Different ways to test a sample

When you build the sample, you get a .NET executable. If it exposes public types and members, you can just add the assembly as a reference to your test and directly call those members. Such "access points" that are specifically there to facilitate testing are called test hooks and their purpose is to expose an API externally to allow tests to manipulate the functionality. If a product provides good, easy to use test hooks, it is said to be testable, which means you can easily write a test program to invoke the functionality of the product.

But what if the sample wasn't specifically designed to be testable? Mostly, samples tend to be small, simple pieces of code, designed only to demonstrate some basic concepts. We don't want to make the samples more complicated by providing test hooks, as it might confuse and distract those who study the sample. So, we come to a constraint - we can't change the sample to be testable - we have to live with what we have. And often, we only have private types and members - adding the sample assembly as a reference to our test project won't help anymore. So the problem boils down to: how to invoke private API's of a .NET assembly from another assembly?

A scenario to test

We have a WinFormsDataBinding project. If you build and run it, it will display a form with a list of employees displayed in a grid:

Our goal is to pretend that we are a user who:

opens the main form
changes the last name of employee with ID 1
submits the changes to the database
closes the main form
reopens the form
verifies that the last name of employee #1 was indeed changed
changes the last name back to the old one
submits the changes
closes the form

Actually, only steps 1-6 are necessary to verify the basic functionality of the LINQ to SQL data binding. Steps 7 and 8 are only necessary to make the test repeatable - this means we can run the test multiple times without side effects. This is a quality of a good test: repeatable means atomic and side-effects-free.

Using Reflection

In our test project, we first want to open the main form. All we have is an .exe file on disk, which is built when you compile the WinFormsDataBinding project. To be able to access the internals of the .exe, we must first load the assembly into memory (into our application domain, to be more specific):

             // Load an assembly from file
            string fileName = Path.Combine(Application.StartupPath, "WinFormsDataBinding.exe");
            Assembly winFormsDataBinding = Assembly.LoadFile(fileName);

Don't forget to add using System.Reflection; to be able to use reflection.

1. Open the form

The main form is represented by the WinFormsDataBinding.EmployeeForm type in the WinFormsDataBinding assembly. Let's just load the type from the assembly:

             Type formType = winFormsDataBinding.GetType("WinFormsDataBinding.EmployeeForm", true, true);

Now that we have the type of the form, we can create an instance of this type (a Form to show). Let's create a method that will do that for us:

         Form ShowForm(Type formType)
        {
            Form newForm = Activator.CreateInstance(formType) as Form;

            newForm.Show();
            newForm.Refresh();
            Application.DoEvents();
            return newForm;
        }

We use System.Activator.CreateInstance to create a new instance of the form, and then we show it on the screen and redraw, so that we actually see that the form is there. I don't include any error handling to simplify the code (normally you would like to surround the first line in a try-catch block in case something goes wrong).

2. Change the cell in the data grid

Let's write a method for this:

         void SetCellText(Type formType, string newName)
        {
            Form newForm = ShowForm(formType);
            DataGridView dataGrid = FindDataGrid(formType, newForm);
            DataGridViewCell cell = FindCell(dataGrid);
            cell.Value = newName;
            PushSubmitChangesButton(formType, newForm);
            CloseForm(newForm);
        }

First we use the ShowForm method to create an instance of the form and show it. Then there are two additional steps involved: we have to find the data grid object on the form and then find the required cell in the data grid. Here's the FindDataGrid method:

         DataGridView FindDataGrid(Type formType, Form newForm)
        {
            FieldInfo dataGridField = formType.GetField(
                "employeeDataGridView", 
                BindingFlags.NonPublic | BindingFlags.Instance);
            DataGridView dataGrid = dataGridField.GetValue(newForm) as DataGridView;
            return dataGrid;
        }

We use the GetField method on the Type class to find a private field called 'employeeDataGridView'. Then we extract the value of this field from our Form object, which gives us a reference to the DataGridView object.

Now we want to find a cell in this grid:

         /// <summary>
        /// Finds the required cell in the grid 
        /// (last name of employee #1)
        /// </summary>
        DataGridViewCell FindCell(DataGridView dataGrid)
        {
            int rowNum = FindRow(dataGrid);
            DataGridViewCell cell = dataGrid.Rows[rowNum].Cells[2];
            return cell;
        }

        /// <summary>
        /// Finds the row number with ID = 1 in the grid
        /// </summary>
        int FindRow(DataGridView dataGrid)
        {
            for (int i = 0; i < dataGrid.Rows.Count - 1; i++)
            {
                if (dataGrid.Rows[i].Cells[0].Value.ToString() == "1")
                {
                    return i;
                }
            }
            return 0;
        }

If we go back to our SetCellText method, we'll see that actually changing the cell's value is easy:

             cell.Value = newName;

3. Submit the changes to the database

To submit the changes, we must programmatically push the 'Submit changes' button on our form. The method PushSubmitChangesButton does that:

         void PushSubmitChangesButton(Type formType, Form newForm)
        {
            MethodInfo submitChanges_Click = formType.GetMethod(
                "submitChanges_Click", 
                BindingFlags.Instance | BindingFlags.NonPublic);
            submitChanges_Click.Invoke(newForm, new object[] { null, null });
        }

Again, we use reflection to fish out a private method from the form's type, and then we invoke the method on the newForm object (passing nulls as arguments).

4. and 5. Closing and reopening the form

Closing and reopening the form is easy, we just call Close() on the form object and call ShowForm to show a new instance of the form.

6. Read the contents of the data grid cell

The GetCellText method is very similar to SetCellText:

         string GetCellText(Type formType)
        {
            Form newForm = ShowForm(formType);
            DataGridView dataGrid = FindDataGrid(formType, newForm);
            DataGridViewCell cell = FindCell(dataGrid);
            string result = cell.Value.ToString();
            CloseForm(newForm);
            return result;
        }

7. 8. and 9. Complete the test

Finally, here's the complete code of our test:

             string fileName = Path.Combine(Application.StartupPath, "WinFormsDataBinding.exe");
            Assembly winFormsDataBinding = Assembly.LoadFile(fileName);

            Type formType = winFormsDataBinding.GetType("WinFormsDataBinding.EmployeeForm", true, true);
            
            SetCellText(formType, "Smith");
            string newName = GetCellText(formType);
            if (newName != "Smith")
            {
                throw new Exception("It didn't rename");
            }
            SetCellText(formType, "Davolio");

Summary

In this tutorial we've used reflection to:

load an assembly
find a type in it
create an instance of this type
read a field on that instance
call a method on that instance

Please let me know if you have any questions or want me to clarify anything.

Thanks,

Kirill

Comments

Anonymous
December 03, 2010
Great article!Very clearly explained...Thank you, and keep the good work!
Anonymous
December 26, 2011
clearly explained and a helpful post

通过