Configuration matrix testing using Visual Studio Lab Management

One of the common issues regularly faced by our users is how to test their application in both IE and Firefox. Visual Studio Lab Management can help in solving this issue for automated Coded UI test as part of Lab E2E Workflow flow.

To accomplish this with Lab Management, we will need

1. Visual Studio 2010 Feature Pack 2 (To be installed on developer box, build box, test controller and test agents)
2. Customized build template derived from LabDefaultTemplate. (Attached to this post)

Lab Management uses Test Settings to define the Role of the environment to use for running test and also for data collection. If there are more than one machines of the Execution role in the environment, then the tests are distributed among them. For configuration matrix testing between IE and Firefox we want to be sure that all our tests run both on IE and Firefox.

So to accomplish our goal we will be using multiple test settings and multiple automated test runs within our environment.

Let us setup the environment now, to do that we do the following:

1. Create a Lab Environment for our application with the additional requirement that we add one more Web Client machine and name the roles for the two machines WebClientIE and WebClientFF.

    

2. Create two test settings by going to the test settings page in Lab Management for this environment; name them TestSettingIE and TestSettingFF. In TestSettingIE, we set WebClientIE as the execution role and enable the data collectors on other roles as required (you should disable data collectors on WebClientFF role as this machine will not participate in the test run for this test setting). In TestSettingFF, we set WebClientFF as the execution role and enable data collectors on the other roles.

    

3. Login into the WebClientFF role machine and we need to set the environment variable CodedUITestCurrentBrowser to specify to the Test Agent to run the Coded UI tests in Firefox. Restart the Test Agent process.

    

4. Make sure all other pre-requisites for deploying the application are also done and we can then take a checkpoint for our daily Lab E2E workflow build.

 With these steps our environment is ready. Now we will do the setup of the test cases.

1. After installation of Feature Pack 2, we open our test project from source control and record a new Coded UI test. This will add the required references for running our Coded UI test against Firefox. After this we check-in our test project back to source control. (A good idea at this point will be to make sure that all tests which we plan to run on both IE and FF are run on your development box with Firefox as the execution browser to avoid any test issues later.)

Now we need to setup the build:

1. Please download the attached LabDefaultTemplate_ConfigTest.xaml; We need to check it in the source control in the BuildProcessTemplates directory. This template is the default Lab build template with the changes to execute two test runs. You can open the file in Visual Studio to understand the changes.

    

    

2. Now we define a new build and select our LabDefaultTemplate_ConfigTest as the build template (to do so you will need to select the add existing xaml option).

    

    

3. We enter all the parameters for the environment created earlier in this build as we normally do for Lab E2E workflow builds, the test setting selected on the Test page of the workflow parameters wizard will be ignored  by our build.

4. We need to enter the test settings on the Process page of build under the values TestSettingRun1 and TestSettingRun2, we need to enter the values as ID of the test settings which we can get from the Test Settings page of Lab Management. We enter the ID for our TestSettingIE and TestSettingFF in the two fields.

    

Now we can run this build and it will go through the build, deploy & test phases. It will create and execute two test runs by the name BuildNumber & BuilNumber_2, we can check them under Test > Analyze Test Runs page of Microsoft Test Manager.

 

With this we have accomplished our goal of executing same set of tests on both IE and Firefox using Lab E2E workflow.

For running the test in Firefox, we used a feature introduced in Feature Pack 2 (more info) to capture Coded UI test in IE and run them in Firefox. If you want to run the configuration matrix across various versions of IE, we do not need to install Feature Pack 2, we just need to have the different versions of IE installed on the two WebClient machines we have in our environment.

You can also extend this testing to beyond two runs by modifying the build template to include more test runs and creating the additional web client machines and test settings in the environment.

LabDefaultTemplate_ConfigTest.xaml

Comments

• Anonymous
  February 12, 2013
  Have you got an updated version for TFS 2012?  The automated configuration of test agents seems to intefere with a setup like this.

• Anonymous
  February 12, 2013
  With TFS 2012, you have to manually login on the 2nd role to run the test agent as process after environment creation. But if you are using snapshot/restore functionality, this does not work as auto-configuration of agent will restore it back. If multiple-browser testing support is what you are looking, have you checked:msdn.microsoft.com/.../jj835758.aspx and skydrive.live.com/redir

• Anonymous
  February 12, 2013
  Thanks for the reply. Yes I was referring to multi-browser testing, unfortunately the CUIT approach doesn't work for multiple versions of IE (without multiple environments) which is really what I want to test. I was hoping to be able to use your work around, but with auto config it would appear not. Currently investigating disabling agents via code to force the next test run to run in a particular machine, but i might just be stuck making 4+ lab environments.

• Anonymous
  February 14, 2013
  I found a potential solution using a simple Test Controller plugin - bzbetty.blogspot.co.nz/.../tfs-extensibility-filtering-lab.html