Guidance for Build, Deploy and Test Workflows
This topic describes the recommended approaches for your build, deploy, and test workflow. You can determine the best approach to use based on your needs. However, your needs might change during the phases of your project cycle. At the beginning of your project, you might just build your application on a nightly basis. As your project progresses, you might want to test the quality of your build by adding unit tests to this workflow. These are often called build verification tests or smoke tests. When your test teams are ready to run tests, you might want to automate the workflow to include deploying your application. You can then run manual or automated tests on the latest version of the application that you have deployed. Or, you might want to build, deploy and run automated tests as your complete workflow to verify the quality of your build when it is deployed.
You can run automated tests as part of these workflows from the assembly for a test project using mstest.exe. Alternatively, you can run automated tests from a test suite in your test plan using tcm.exe. To run automated tests from a test plan, you must do these tasks:
Create a test plan and test suite: Defining Your Testing Effort Using Test Plans.
Associate your automated tests with test cases in a test suite: How to: Associate an Automated Test with a Test Case.
Create a physical or virtual environment: Environments.
Use the information in the following sections to set up your software components for the tasks that you need in your workflow:
Requirements
The following section describes the requirements to use build, deploy and test as part of your workflow: Requirements.
Build
If you just want to build your application, you can use the default build template to do that. For more information about how to set up a build, see Building the Application.
Build and Test
If you want to run automated tests as part of your build process, you can select the tests to run from your test project by using the default build template. If the tests pass, then you consider that build is successful. Build and Run Automated Tests
Important
This runs tests as part of the build process by using mstest.exe. For more information about mstest.exe, see MSTest.exe Command-Line Options.
Build and Deploy
If want to run manual tests from your test plan using the latest build of your application, you can build and then deploy your application to a physical or virtual environment. Build, Deploy and Run Manual Tests from a Test Plan
Build and Deploy and Test
If you want to check the quality of the application after deployment, you can build, deploy your application and run automated tests from a test plan using a physical or virtual environment. Build, Deploy and Run Automated Tests from a Test Plan Using an Environment
Important
This runs tests as part of the build process by using tcm.exe. For more information about tcm.exe, see tcm: Importing and Running Automated Tests for a Test Plan from the Command Line.
Requirements
Based on the tasks that you want in your workflow, you can install the required components for each task on physical or virtual machines. The following illustration shows an example of how you might choose to install your software based on the information in this section:
.png "Requirements for Build, Deploy and Test")
Software Requirements for All Tasks
- Team Foundation Server
Software Requirements for Build
Build controller
At least 1 build controller is required for each team project collection
Build agent
At least 1 build agent is required for each build controller
Software Requirements to Deploy
1 build controller (can use the same build controller that is used to build the application)
For a virtual environment: 1 build agent and 1 lab agent on each machine in the virtual environment
Software Requirements to Test with Environments
Test Controller
At least 1 test controller is required for each team project collection
Test Agents
1 test agent is required on each machine in the environment
For a virtual environment: 1 lab agent on each machine in the virtual environment (in addition to the test agent)
For more information about how these components work together to run tests, see Setting Up Test Machines to Run Tests or Collect Data.
For more information about how to install these components, see Installing and Configuring Visual Studio Agents and Test and Build Controllers.
The following sections can help you decide on which machines to install your software components. Then you can determine how many machines you will need for your setup.
Machine Requirements for Build Controllers and Build Agents
To determine the machines that you require, you must base your topology on the following information:
You must have one build controller for each team project collection.
You can only have one build controller on each computer.
In a typical installation, the tasks that the build agent performs can place heavy demands on the processor. This could significantly decrease the performance of your Team Foundation Server. In this case, you will want to put your build agent on a separate computer from your Team Foundation Server.
You can install the build controller on the same computer as your Team Foundation Server. If the build controller manages many active build agents, this can use more of the system’s memory and you might decide to install the build controller on a separate computer.
For example if you have three team project collections and each one has applications that must be built, you need three computers for the three build controllers or two extra computers if you install one build controller on your Team Foundation Server.
Important
For more information about the topologies you can use to set up your build controller and build agents, see Build System Topology Examples.
Machine Requirements for Test Controllers and Test Agents
To reduce communication issues between software components, we recommend that the test controller machine and Team Foundation Server machine are in the same domain. For more information about how to install components in different domains or in workgroups, see Requirements for Workgroups and Multiple Domains.
To determine the machines that you require for test controllers, you must decide on the topology that you require based on the following information:
If you are testing with a physical or virtual environment, you need a test controller.
You can use a single test controller with multiple physical or virtual environments.
You must register a test controller with the team project collection that contains your environments.
Each test controller can only be registered with a single team project collection.
Each test controller that you require must be installed on a separate machine.
For example, if you have three team project collections for which you want to create environments, you must have at least three test controllers. This requires at least three machines.
Note
If you have many environments to support in a team project collection and need more than one test controller, you can register multiple test controllers with the same Team Foundation Server.
Reducing Machines Required For Build Controllers and Test Controllers
If you need multiple build controllers and test controllers, you can reduce your machine requirements by installing a test controller on the same machine as a build controller. For example, if you have three team project collections A, B and C and each one requires a build controller and test controller, you can install the build controller and test controller on the same machine. By using this approach, you would only need three machines for the controllers instead of six machines.
Build and Run Automated Tests
To use build and test as your workflow tasks, you must provide the following information in the default build definition that you create:
The test assembly that contains your tests
(Optional) The test categories to use if you want to select specific tests from the assemblies
The test settings to use to run your tests
For more information about test settings, see Setting Up Machines and Collecting Diagnostic Information Using Test Settings.
The build agent uses mstest.exe to run the tests from the test assembly that is located in the drop folder as shown in the following illustration. You can use the default test settings (local.testsettings) from the solution for your application to run your tests locally using the build agent.
Note
To use your local.testsettings file as part of your build process, you must check in this file to your Team Foundation Server.
You cannot collect information by using diagnostic data adapters when you run tests with just a build agent. If you want to use diagnostic data adapters with this scenario, you must install a test agent on the same machine as your build agent. Alternatively, you can use a physical or virtual environment, as described in the following section: Build, Deploy and Run Automated Tests from a Test Plan.
.png "Build and Test with No Test Agent")
For the detailed steps about how to build and run tests for your workflow, see How to: Configure and Run Scheduled Tests After Building Your Application.
Warning
We do not recommend that you use a test controller with your test settings to run tests remotely as part of the build definition. If you want to use a test controller and test agents, you should use a physical or virtual environment, as described in the following section: Build, Deploy and Run Automated Tests from a Test Plan.
If you are just learning about how to use Visual Studio, Team Foundation Server and Team Foundation Build to define a workflow, you can install all the components on one machine. But we do not recommend this when you have more users of this system and you are running multiple builds.
Note
To run coded UI tests, we recommend that you first deploy your application to a physical or virtual environment. By using this approach, if you have to run manual steps to investigate a test failure, you can use this same environment. To run coded UI tests without deploying your application, you must configure the build agent that is used to be able to interact with the desktop. In addition, you must install a test agent on the build machine and use the latest build from the drop folder for your testing. For more information about how to set up your build agent, see Set up your agents to run coded UI tests.
Build and Deploy
To build and deploy your application for your workflow, you must use a physical or virtual environment.
Virtual Environments
To use a virtual environment that uses Visual Studio Lab Management to build and deploy, you use the lab default template. This lab template enables you to do the following actions:
Select the virtual environment that you want to use
Select a snapshot for that environment to use as your starting point for deployment
Select the build definition or build to use to deploy your application
Add scripts to run to deploy the application from the drop folder
Take a snapshot of the virtual environment after deployment as part of the build and deploy workflow
You can use the virtual environment to run manual tests using the build that was deployed onto this environment or you can run automated tests. For more information about how to build and deploy an application to a virtual environment, see How to: Deploy an Application on a Virtual Environment. For more information about how to run manual tests using a virtual environment, see How to: Run Manual Tests and Create Reproducible Bugs With Virtual Environments.
Physical Environments
If you want to deploy your application to a physical environment, you can customize the template for a build definition. For more information about how to customize a build process template, see Create and Work with a Custom Build Process Template.
You can use this environment to run manual tests or automated tests from your test plan. For more information about how to run these tests, see Running Manual Tests Using Test Runner and Essential Guide for Running Automated Tests from a Test Plan.
Build, Deploy and Run Automated Tests from a Test Plan using an Environment
To run automated tests that are part of a test plan in your build, deploy and test process, you must use a physical or virtual environment. The build definition uses tcm.exe to run your tests. When you run these tests as part of your workflow, a test run is created for these tests and the test results can be reviewed and analyzed using Microsoft Test Manager. For more information about how to analyze your test runs, see How to: Analyze Test Runs Using Microsoft Test Manager. This also enables you to view historic data about the quality of your builds. You can determine when there are consistent issues with the quality of the builds and which areas of the application have tests that fail.
You can use physical or virtual machines when you create a physical environment for testing, or you can use Visual Studio Lab Management to create a virtual environment. A virtual environment enables you to deploy your application by using each machine in the environment in a known state from an existing snapshot. In addition, you can take snapshots of the environment after the application is deployed so that you can return to this known state for testing a bug or running manual tests. A virtual environment gives you much more flexibility. For more information about how to create these environments, see Creating a Physical Environment to Use for Testing and Creating Virtual Environments.
Virtual Environments
To deploy your application to a virtual environment, you can use the default lab template definition that is provided with Team Foundation Build. To use a virtual environment, you must have Visual Studio Lab Management. This lab template enables you to do the following actions:
Select the virtual environment that you want to use
Select a snapshot for that environment to use as your starting point for deployment
Select the build definition or build to use to deploy your application
Add scripts to run to deploy the application from the drop folder
Select the test suites, test configuration and test settings for your test run
Take a snapshot of the virtual environment after deployment as part of the build and deploy workflow
For more information about how to set up your build definition for the default lab template to build, deploy and test with a virtual environment, see How to: Configure and Run Scheduled Tests After Building and Deploying Your Application.
.png "Build, Deploy and Test with a Virtual Environment")
Physical Environments
To deploy your application to a physical environment and run automated tests, you can customize the template for a build definition. For more information about how to customize a build process template, see Create and Work with a Custom Build Process Template.
.png "Build, Deploy and Test With a Physical Environment")