Test Configurations - specifying test platforms
Your application's users will probably install it on computers that have a wide variety of configurations, with different operating systems, different web browsers, different speeds of internet connection, and other variations. You will therefore want to perform at least some of your tests in environments that have those different configurations.
You might draw up a schematic matrix of the combinations that you want to test:
.png "Configuration variables and configurations")
Requirements
- Visual Studio Ultimate, Visual Studio Premium, Visual Studio Test Professional
Running tests with configurations
When you look at the list of tests that are ready to run, one of the columns is titled Configuration. This column specifies a particular combination of hardware and software on which you should run the test. For example, it might specify Windows Phone 7 or Windows 8 and Internet Explorer 9. If your application can be installed on different server platforms, the configurations might also specify features of the server.
.png "Running test cases with different configurations")
Notice that the test plan can call for some test cases to be run under more than one configuration. Each pair of test case and test configuration is called a test point.
When you run a particular test point, you should work on a machine that has the specified configuration. Typically, this means that before you install the application for testing, you have to set up a computer with the correct operating system and other specified characteristics.
You will probably want to run all the tests on one configuration in a batch. You can filter and sort the test list by configuration:
.png "Sorting and filtering test points by configuration")
For manual testing, the test configurations are simply instructions to the manual tester about how to set up the test environment. But some projects don’t require multiple configurations. In those projects, you can simply ignore the configuration settings.
Plan the configurations for a test case
In the test plan, you can set the configurations to be used for one or more test cases:
.png "Opening test configurations for test cases")
In the Select Test Configurations window, select All configurations. Indicate which configurations you want:
.png "Selecting test configurations for a test case")
If you want to change a long list of tests at the same time, select Enable range selection and fill mode. Click in the box at the top of the range, SHIFT+click in the box at the bottom, and then select or clear the box at the top.
Note
Setting the configurations alters only the configurations of this appearance of the test case. If the same test case appears in another suite or another plan—for example in the test plan for another sprint—then the configurations assigned to it in the other plan will not be affected.
Set the default configurations for your test plan
Instead of setting the configurations for each test case separately, set the default configurations for in the properties of your test plan. The defaults apply both to new test cases and to test cases that you copy from a previous plan.
To set the defaults, choose Testing Center, Plan, Contents. Select the test plan at the root of the tree. Then choose the drop-down selection at Default configurations.
.png "Setting default configurations for a test plan")
You can also set the configurations in the Properties of the plan.
Note
The default configurations only have effect when you add a test case to a plan. Changing the defaults doesn't affect test cases that are already in the plan.
Create a test suite for multiple configurations
Typically, you don't want to run every test case with multiple test configurations. Instead, you select just a few tests to make sure that your application runs on other configurations. For example, you might run all your tests on Windows 8, but also run just a few of them on Windows 7 and Windows XP.
The easiest way to do this is to create a separate suite for multiple configurations. Set its default configurations separately from the test plan. Copy the selected tests to the new suite. In the new suite, those test cases will be assigned multiple configurations.
Each test suite usually inherits its default configurations from the test plan, but you can set them individually by selecting the suite:
.png "Setting default configurations for a test suite")
To copy a test case into a test suite, simply drag it onto the new suite. It will then be referenced from more than one suite, and will have different configurations in the different suites.
Create more configurations
A small number of test configurations is automatically set up when you create your team project, but you can create more in Test Configuration Manager.
Each test configuration specifies settings for a number of configuration variables, such as Operating System and Browser. You can define new variables, and new permitted values for each variable.
.png "Managing test configurations in a test plan")
You can delete a test configuration if it isn't referenced in a test result or in a test plan. Otherwise, you can set its state to inactive.
Creating new configuration variables
Create variables for each aspect of the test environment that you want to specify. For example, you can create variables for client device type, Server operating system, Network speed, and database edition. Notice that these are not aspects of your system under test. They are variable choices that your application's users might make for the platform on which they will install your system.
.png "Creating and editing configuration variables")
Creating a new configuration
.png "Creating a test configuration")
If you want to stop a configuration from appearing as an option when you set the configurations of test cases, set its State to Inactive. However, this does not affect existing assignments of configurations to test cases.
External resources
Guidance
Testing for Continuous Delivery with Visual Studio 2012 – Chapter 3: Lab Environments