How to: Configure and Run Scheduled Tests After Building Your Application
You can run tests after a build has completed to evaluate the quality of your build. These tests are often called build verification tests (BVTs) or smoke tests. These tests typically consist of a broad suite of tests that are used to verify key areas of an application in a particular build. A build is considered a success if all the tests in the BVT have passed.
You can use one or more types of automated tests as part of your build verification test. You can run the following types of tests:
Unit tests
Database unit tests
Coded UI tests
Web performance tests
Load tests
Generic tests
Creating BVTs uses Visual Studio, Team Foundation version control, and Team Foundation Build. First, you use Visual Studio to assign test categories to your tests that you want to use for your BVT. Next, you check in the automated tests marked with the test categories to source control by using Team Foundation version control. You then add the test category filter in your build definition. Finally, you use Team Foundation Build to queue a build which will then run your tests if the build is successful.
Note
The client computer must have Team Explorer installed, and your Visual Studio user session must be connected to a Team Foundation Server computer. For information about how to connect to Team Foundation Server, see How to: Connect to a Team Project in Team Foundation Server.
This topic describes all the procedures that are required to create and run build verification tests:
Create Test Categories for Your BVT
Check In Your Build Verification Tests
Create a Build Definition
Add the computers or user accounts for build and test agents to the TeamTestControllerUsers group
Set Up Your Agent to Run Coded UI Tests
Run the BVT Build Definition
Note
You can use test lists to run your build verification tests, but test categories are recommended for use over the test lists functionality from earlier versions of Microsoft Visual Studio 2010, unless you have to create a check-in policy which requires a test list. For more information about how to create a test list, see How to: Create a Test List.
Create Test Categories for Your BVT
You can use logical operators to create filters using & (AND), | (OR) and !(NOT) to select the tests to run based on the categories assigned to the tests. You may want to create multiple test categories that you can use for your build definition to create flexibility in the way you select your tests.
To create test categories for your test methods
On the Test menu, click Windows, and then select Test View.
The Test View window displays
Select a test.
In the properties pane of the test that you selected, click the ellipsis (**…)**in the Test Categories column. The Test Category window will be displayed.
Type the name of your new test category in the Add New Category field.
Click Add.
Click OK. The new test category will be assigned to your test and it will be available to the other tests in the Test View window.
To create more test categories, repeat steps 4 to 6.
For more information about how to assign test categories by adding an attribute to your test method, see How to: Group and Run Automated Tests Using Test Categories.
Check in Your Build Verification Tests
In this procedure, you check in all the files of your solution. This will include the test categories that you added to your test methods.
To check in your build verification tests to source control
Connect to a Team Foundation Server computer. For more information, see How to: Connect to a Team Project in Team Foundation Server.
If your solution is not already in source control, add it to source control. For more information, see Add Files to Version Control.
Click View and then click Pending Checkins. The Pending Checkin window is displayed.
Check in all the files of your solution. For more information, see Check In Pending Changes.
Note
You might have a specific team process that governs how BVTs are created and managed. For example, the process might require that you verify your build locally before you check in that code and the tests that run on it.
After the check-in operation is finished, a padlock icon is displayed next to each file in Solution Explorer to indicate its Checked In status. For more information, see Identify Version Control Item Status in Solution Explorer.
Your checked-in tests are available to use in a build. You can now create a build definition that contains the tests you want to run in your BVT.
Create a Build Definition
To create the BVT build definition
In Team Explorer, click your team project.
Right-click Builds and then click New Build Definition.
The New Build Definition tab is displayed.
Enter the information for your new build definition**.** For more information, see Create a Basic Build Definition.
Specify the name to associate with the build definition in the Build definition name text box.
(Optional) Add an appropriate description in Description.
Click Workspace.
The Working folders table includes the source control folder for the team project for which you are creating the new build definitions and a local folder on the build agent. The local folder on the build agent is listed in the Local Folder column. All the workspace paths on the build agent are mapped relative to the default root directory shown.
To copy an existing workspace to the list of working folders, click Copy Existing Workspace to open the Select a Workspace dialog box.
The workspace you select is normalized to the common root directory on the build agent, $(SourceDir). SourceDir is an environment variable that expands to $(BuildDir)\Sources.
You can also click an empty table cell in the Source Control Folder, and then click the ellipses (…) to browse to a source control folder to add as a working folder. The source control folder you select is also normalized to the common root directory on the build agent.
Click Build Defaults.
On the Build Defaults pane, you can select build controller, if one exists, from the Build controller drop-down list. You can optionally click Manage to open the Manage Build Controllers dialog box.
In the Copy build output to the following drop folder type the UNC path, such as (\\server\share) location. The built binaries and log files will be located in this folder as soon as the build finishes. For more information about how to set up a drop folder, see Set Up Drop Folders.
If you plan to run coded UI tests, Web performance tests, or load tests as part of your build definition, you must use the output from the build that is in this location to launch or install your application. To automatically install your application after the build completes and before the tests are run, you can either use the lab default template that can deploy your application to a virtual environment or you can modify this lab default template to deploy your application to a physical environment.
Note
Before you complete this step, you must have created a public folder where the TFSService account has full rights. For more information about Team Foundation service accounts, see View Team Foundation Server Services.
Click Process.
In the Required section, click Items to Build. Then click the ellipses (…).
The Items to Build dialog box is displayed.
Click Add. Then locate the solution or project you want to build in the version control tree, and then click OK.
On the Items to Build dialog box, click OK.
To add a test category of tests to run when the build has completed, open the Basic section. In the Automated Tests section, open Test Assembly, and then click Category Filter. Enter the filter you require to select your test methods based on your test category.
Note
The test category filter consists of one or more test category names separated by the logical operators '&', '|', '!', '&!'. For example, ShoppingCart&SmokeTest will run all tests with a test category of ShoppingCart and SmokeTest. Or you can just select all tests in one category by entering SmokeTest. (The logical operators '&' and '|' cannot be used together to create a test category filter.)
To specify the search pattern for locating test assemblies, click Test Assembly Filespec. Type your search string. For example, **\*test*.dll if your dlls all contain the word "test" in their names.
Note
This search string will search recursively through directories looking for any dlls that match *test*.dll in the Binaries directory. For more information about this, see Define Your Build Process.
(Optional) To select a test settings file to use when you run the tests, open Automated Tests, open Test Assembly and click TestSettings File and then click the ellipses (…).
The Browse dialog box is displayed. Locate the .test settings file that contains the test settings that you want to use, and then click OK.
If your test setting file uses a test controller and test agents, see the following procedure: Add user accounts or computers for build and test agents to the TeamTestControllerUsers group.
If you are running coded UI tests, see the following procedure to set up your agents based on your test settings file: Set Up Your Agents to Run Coded UI Tests.
Note
You can use the Agent Settings in the Advanced section to select a specific agent to use. For example, if you are running coded UI tests and must select an agent that is running as an interactive process, you can select it here.
For more information about test settings files, see Create Test Settings to Run Automated Tests from Visual Studio.
Note
There are two default test settings files. Local.testsettings collects only system information, by default. If you want to also collect IntelliTrace data, and collect test impact analysis data to use to determine tests that are recommended to be run based on build changes, you must select the test settings file that is named TraceAndTestImpact.testsettings.
(Optional) To run load tests as part of the build process, you must set up a load test results repository and configure your test controller specified in your test settings to use that repository. For more information, see How to: Select a Load Test Results Repository.
To save your new build definition, click Save.
Your new build definition appears in the Team Explorer window under the Builds folder.
Note
If you do want to add a test list to run when the build has completed instead of a test category, you can add command line arguments to do this. For more information about command line arguments, see Running Automated Tests from the Command Line.
Add the user accounts or computers for build and test agents to the TeamTestControllerUsers group
If your test settings file that you added to your build definition uses a test controller and test agents, you must add the computers for any build or test agents used to the TeamTestControllerUsers security group on the test controller computer. For example, if you want to run coded UI tests as part of your build process, you must add these computers to this security group.
Warning
If your build agents or test agents are set up to use a different user and not the Network Service account, you must make sure that this domain user account is added to the TeamTestControllerUsers group instead.
To add the users or computers for build and test agents to the TeamTestControllerUsers group
From the test controller computer, click Start, click Control Panel, click Administrative Tools, and then click Computer Management.
The Computer Management dialog box is displayed.
Open Local Users and Groups and then click Groups.
The groups are displayed.
To add the users or computers, right-click TeamTestControllerUsers and point to Add to Group.
The TeamTestControllerUsers Properties dialog box is displayed.
Look at the Members list to see whether the domain user account or computers are already added. If they are not, click Add.
The Select Users, Computers, or Groups dialog box is displayed.
By default, only the users or groups are shown. To add computers, click Object Types, select Computers and then click OK.
To add a computer, type the name of the computer in Enter the object names to select, and then click OK.
Warning
You must add both the computer where the build agent is running and the computer for your test agent.
To add a domain user account, verify the location is correct, type the name of the user account in Enter the object names to select, and then click OK
Important
Repeat this step to add all the user accounts that you require.
To apply your changes, click OK.
Set up your agents to run coded UI tests
If you want to run coded UI tests as part of your scheduled tests after building your application, you must do one of the following:
Use a test settings file that specifies a test controller and the roles for your application to run your tests. Create these test settings using Microsoft Visual Studio 2010. For any one of the test agents assigned to the role that runs your tests in the test settings, you must follow the steps in this procedure to set up the test agent to run as a process instead of a service. For more information about test controllers and test agents, see Setting Up Test Machines to Run Tests or Collect Data and Installing and Configuring Visual Studio Agents and Test and Build Controllers.
Use a test settings file that does not specify a test controller. If you do this, then you must set up your build agent service to be able to interact with the desktop. Select the property for the service to Allow the service to interact with the desktop. This enables the build agent to run the coded UI test.
Warning
If you are running a coded UI test that starts a browser, the service account for the build service is used to start that browser. This service account must be the same as the user account that is the active user on this computer. If it is not the same user account, the browser will not start.
To set up your test agents to run coded UI tests
- To set up your test agents to run coded UI tests, follow the steps in this topic: How to: Set Up Your Test Agent to Run Tests that Interact with the Desktop.
To run the BVT using Team Build
To run the BVT using Team Build
In Team Explorer, click your team project.
Right-click Builds and then click Queue New Build.
(Optional) Change the build location and directory.
Click Queue.
The build starts and the Build Explorer dialog box is displayed.
When the build has completed, click Completed to see the details.
To view the details of the build, double-click the build in the list.
A new tab is displayed with the build information. You can view the status of the test run.
To view the test result details, click the arrow to open the test and then click View Test Results.
Select a folder to store your test results locally.
The test results are now displayed in the Test Results window.
For more information, see Building the Application.
See Also
Concepts
Defining Test Categories to Group Your Tests
Defining Test Lists to Group Your Tests
Testing Application Performance and Stress