Running tests in parallel using VSTS Release Management

Release Management now supports running the tasks inside a phase in parallel on multiple agents.  In this blog post, I will walk you through how this can be leveraged to run tests in parallel.

Scenario:  I have a dotnet core project. I want to test it against Windows, Linux, iOS. Additionally, I want these tests to run in parallel.

Step 1:  Create a new variable TargetOS variable and initialize it with Windows_NT ,linux ,darwin.

Step 2. Add two dotnet core tasks for restore and test.

Step 3: Select Multi-Configuration from Type of parallelism.

Step 4: Specify Multipliers as TargetOS and set Maximum number of agents as equal to more than the OS in TargetOS variable.

If you have lots of runtimes and want to run those in parallel, but you want to limit the concurrency so that you do not starve another build/release for an agent. Then you should set Maximum number of agents to the value equal to the desired number of agent it is allowed to take simultaneously.

Step 5:  In demand specify Agent.OS -equals $(TargetOS)

Depending on the number of OS specified in TargetOS variable,  test runs will be created. Here three test run will be created each having the corresponding value of variable TargetOS variable. For example, the test run for Windows_NT will have TargetOs as Windows_NT.

We use this variable in demand to target appropriate agent. For Windows demand will evaluate to  Agent.OS -equals Windows_NT. This test will run on the agent with the capability Agent.OS as Windows_NT.

Also, make sure that the agent capabilities match the demands after the variable is replaced.

Step 6: Queue the release

This will create three test run and each will target agent with respective OS. The first test run is for iOS, second for Windows and third for Linux. We can check this in the name of Multiplier Node.

If your tests take very long to run and you want to cut the time, there is an upcoming version of VS Test task which can slice the tests in a single test run, each of which can run parallelly.  To use this new task, you can use Multi-Agent parallel execution. If you chose to run release with Parallel execution as Multi-Agent, the test is simply duplicated to run on multiple agents and each agent pull the appropirate slice from the server.

Comments

• Anonymous
  March 10, 2017
  Can this be used when any kind of task, like "Run a PowerShell script" task? Also is this available with on-premise VSTS?
  • Anonymous
    December 06, 2017
    This setting is at phase level and independent of tasks. Yes this is available on on-Prem VSTS