Share via

Visual Studio Test Agent Deployment task

  • Article

TFS 2017 | TFS 2015

Note

In Microsoft Team Foundation Server (TFS) 2018 and previous versions, build and release pipelines are called definitions, runs are called builds, service connections are called service endpoints, stages are called environments, and jobs are called phases.

TFS 2017 and earlier

Use this task to deploy and configure the test agent to run tests on a set of machines. The test agent deployed by this task can collect data or run distributed tests using the Visual Studio Test task.

Demands and prerequisites

This task requires the target computer to have:

  • Windows 7 Service Pack 1 or Windows 2008 R2 Service Pack 2 or higher
  • .NET 4.5 or higher
  • PSRemoting enabled by running the Enable-PSRemoting PowerShell script

Windows Remote Management (WinRM)

This task uses Windows Remote Management (WinRM) to access on-premises physical computers or virtual computers that are domain-joined or workgroup-joined.

To set up WinRM for on-premises physical computers or virtual machines

Follow the steps described in domain-joined

To set up WinRM for Microsoft Azure Virtual Machines

Azure Virtual Machines require WinRM to use the HTTPS protocol. You can use a self-signed Test Certificate. In this case, the automation agent will not validate the authenticity of the certificate as being issued by a trusted certification authority.

  • Azure Classic Virtual Machines. When you create a classic virtual machine from the Azure portal, the virtual machine is already set up for WinRM over HTTPS, with the default port 5986 already opened in the firewall and a self-signed certificate installed on the machine. These virtual machines can be accessed with no further configuration required. Existing Classic virtual machines can be also selected by using the Azure Resource Group Deployment task.

  • Azure Resource Group. If you have an Azure Resource Group

already defined in the Azure portal, you must configure it to use the WinRM HTTPS protocol. You need to open port 5986 in the firewall, and install a self-signed certificate.

To dynamically deploy Azure Resource Groups that contain virtual machines, use the Azure Resource Group Deployment task. This task has a checkbox named Enable Deployment Prerequisites. Select this to automatically set up the WinRM HTTPS protocol on the virtual machines, open port 5986 in the firewall, and install a test certificate. The virtual machines are then ready for use in the deployment task.

Arguments

Argument Description
Machines A comma-separated list of machine FQDNs or IP addresses, optionally including the port number. The maximum is 32 machines (or 32 agents). Can be:
- The name of an Azure Resource Group.
- A comma-delimited list of machine names. Example: dbserver.fabrikam.com,dbserver_int.fabrikam.com:5986,192.168.34:5986
- An output variable from a previous task.
Admin Login The username of either a domain or a local administrative account on the target host(s). This parameter is required when used with a list of machines. It is optional when specifying a machine group and, if specified, overrides the credential settings defined for the machine group.
- Formats such as username, domain\username, machine-name\username, and .\username are supported.
- UPN formats such as username@domain.com and built-in system accounts such as NT Authority\System are not supported.
Password The password for the administrative account specified above. This parameter is required when used with a list of machines. It is optional when specifying a machine group and, if specified, overrides the credential settings defined for the machine group. Consider using a secret variable global to the build or release pipeline to hide the password. Example: $(passwordVariable)
Protocol The protocol that will be used to connect to the target host, either HTTP or HTTPS.
Agent Configuration - Username Required. The username that the test agent will use. Must be an account on the test machines that has administrative permissions.
- Formats such as username, domain\username, machine-name\username, and .\username are supported.
- UPN formats such as username@domain.com and built-in system accounts such as NT Authority\System are not supported.
Agent Configuration - Password Required. The password for the Username for the test agent. To protect the password, create a variable and use the "padlock" icon to hide it.
Agent Configuration - Run UI tests When set, the test agent will run as an interactive process. This is required when interacting with UI elements or starting applications during the tests. For example, Coded UI or Selenium tests that are running on full fidelity browsers will require this option to be set.
Agent Configuration - Enable data collection only When set, the test agent will return previously collected data and not re-run the tests. At present this is only available for Code Coverage. Also see FAQ section below.
Advanced - Test agent version The version of the test agent to use.
Advanced - Test agent location Optional. The path to the test agent (vstf_testagent.exe) if different from the default path.
- If you use a copy of the test agent located on your local computer or network, specify the path to that instance.
- The location must be accessible by either the build agent (using the identity it is running under) or the test agent (using the identity configured above).
- For Azure test machines, the web location can be used.
Advanced - Update test agent If set, and the test agent is already installed on the test machines, the task will check if a new version of the test agent is available.
Control options See Control options

The task supports a maximum of 32 machines/agents.

Supported scenarios

Use this task for:

  • Running automated tests against on-premises standard environments
  • Running automated tests against existing Azure environments
  • Running automated tests against newly provisioned Azure environments

The supported options for these scenarios are:

  • TFS
    • On-premises and Azure Pipelines

  • Build and release agents
    • Hosted and on-premises agents are supported.
    • The agent must be able to communicate with all test machines. If the test machines are on-premises behind a firewall, an Azure Pipelines Microsoft-hosted agent cannot be used because it will not be able to communicate with the test machines.
    • The agent must have Internet access to download test agents. If this is not the case, the test agent must be manually downloaded, uploaded to a network location accessible to the agent, and the Test Agent Location parameter used to specify the location. The user must manually check for new versions of the agent and update the test machines.

  • Continuous integration/continuous deployment workflows
    • Build/deploy/test tasks are supported in both build and release workflows.

  • Machine group configuration
    • Only Windows-based machines are supported inside a machine group for build/deploy/test tasks. Linux, macOS, or other platforms are not supported inside a machine group.
    • Installing any version of Visual Studio on any of the test machines is not supported.
    • Installing any older version of the test agent on any of the test machines is not supported.

  • Test machine topologies
    • Azure-based test machines are fully supported, both existing test machines and newly provisioned test machines.
    • Machines with the test agent installed must have network access to the TFS instance in use. Network-isolated test machines are not supported.
    • Domain-joined test machines are supported.
    • Workgroup-joined test machines must use HTTPS authentication configured during machine group creation.

  • Usage Error Conditions
    • Using the same test machines across different machine groups, and running builds (with any build/deploy/test tasks) in parallel against those machine groups is not supported.
    • Cancelling an in-progress build or release that contains any build/deploy/test tasks is not supported. If you do cancel, behavior of subsequent builds may be unpredictable.
    • Cancelling an ongoing test run queued through build/deploy/test tasks is not supported.
    • Configuring the test agent and running tests as a non-administrator, or by using a service account, is not supported.
    • Running tests for Universal Windows Platform apps is not supported. Use the Visual Studio Test task to run these tests.

Example

More information

Open source

This task is open source on GitHub. Feedback and contributions are welcome.

FAQ

When would I use the Enable Data Collection Only option?

An example would be in a client-server application model, where you deploy the test agent on the servers and use another task to deploy the test agent to test machines. This enables you to collect data from both server and client machines without triggering the execution of tests on the server machines.

How do I create an Azure Resource Group for testing?

See Using the Azure Portal to manage your Azure resources and Azure Resource Manager - Creating a Resource Group and a VNET.

Do I need an agent?

You need at least one agent to run your build or release.

I'm having problems. How can I troubleshoot them?

See Troubleshoot Build and Release.

I can't select a default agent pool and I can't queue my build or release. How do I fix this?

See Agent pools.

My NuGet push task is failing with the following error: "Error: unable to get local issuer certificate". How can I fix this?

This can be fixed by adding a trusted root certificate. You can either add the NODE_EXTRA_CA_CERTS=file environment variable to your build agent, or you can add the NODE.EXTRA.CA.CERTS=file task variable in your pipeline. See Node.js documentation for more details about this variable. See Set variables in a pipeline for instructions on setting a variable in your pipeline.

I use TFS on-premises and I don't see some of these features. Why not?

Some of these features are available only on Azure Pipelines and not yet available on-premises. Some features are available on-premises if you have upgraded to the latest version of TFS.

Help and support