Controlling build result with code coverage percentage (using build vNext)

<Edit, added script for VSO>

I had written a blog about on this topic for XAML build definitions and became curious to check if it is possible to achieve the same using vNext build definitions also. I was able to do it and this blog covers what I did.

The PowerShell script used here can be modified – to check more parameters of the build. It’s relatively easier than XAML build definition and it’s easy to have stricter and custom checks in vNext.

While this blog covers Code coverage percentage as the benchmark to determine the status of build, we can use lot more (check the IBuildDetail class for the list of parameters).

To get started pick a build definition that has the “Visual Studio Build” and “Visual Studio Test” steps configured. Make sure the test step has the Code Coverage Enabled box ticked in advanced.

Now we should go ahead and add a new task “PowerShell”. But before we do that, we need to create the PowerShell script and check it into the source control. Note that, we have separate scripts for On-Prem and VSO.
Thanks to a lot of readers bringing it to our notice!

Script for TFS ( On-Prem )

param(

           # Getting the control percentage as an argument

           [int] $desiredCodeCoveragePercent = 95

)

 

Write-Host "Desired Code Coverage Percent is " -nonewline; Write-Host $desiredCodeCoveragePercent

 

# Load Assemblies we use. Make sure you change the version as per need.

[Reflection.Assembly]::Load(“Microsoft.TeamFoundation.Client, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”) | Out-Null

[Reflection.Assembly]::Load(“Microsoft.TeamFoundation.TestManagement.Client, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”) | Out-Null

[Reflection.Assembly]::Load(“Microsoft.TeamFoundation.TestManagement.Common, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”) | Out-Null

[Reflection.Assembly]::Load("Microsoft.TeamFoundation.Build.Client,Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”) | Out-Null

 

# Getting a few environment variables we need

[String] $CollectionUrl = "$env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI"

[String] $BuildUrl = "$env:BUILD_BUILDURI"

[String] $project = "$env:SYSTEM_TEAMPROJECT"

 

[int] $coveredBlocks = 0

[int] $skippedBlocks = 0

[int] $totalBlocks = 0

[int] $codeCoveragePercent = 0

 

 

# Get the Team Project Collection, then test and build service

$teamProjectCollection = [Microsoft.TeamFoundation.Client.TfsTeamProjectCollectionFactory]::GetTeamProjectCollection($CollectionUrl)

$tcmService = $teamProjectCollection.GetService([Microsoft.TeamFoundation.TestManagement.Client.ITestManagementService])

$buildServer = $teamProjectCollection.GetService([Microsoft.TeamFoundation.Build.Client.IBuildServer])

[Microsoft.TeamFoundation.TestManagement.Client.ITestManagementTeamProject] $tcmProject = $tcmService.GetTeamProject($project);

 

 

# Getting all the test run as part of the build

$totalRuns = $tcmProject.TestRuns.ByBuild($BuildUrl)

 

# Getting Code coverage class details

[Microsoft.TeamFoundation.TestManagement.Client.ICoverageAnalysisManager] $coverageAnalysisManager = $tcmProject.CoverageAnalysisManager

 

# Looping through all the results

 foreach ($currentRun in $totalRuns)

{

           $sourceBlocks = $coverageAnalysisManager.QueryTestRunCoverage($currentRun.Id, 1)

           foreach ($currentBlock in $sourceBlocks)

           {

                      $modules = $currentBlock.Modules

                      foreach($module in $modules)

                      {

                                            $coveredBlocks += $module.Statistics.BlocksCovered

                                 $skippedBlocks += $module.Statistics.BlocksNotCovered 

                      }

 

           }

}

 

 $totalBlocks = $coveredBlocks + $skippedBlocks;

 

 if ($totalBlocks -eq 0)

{

           $codeCoveragePercent = 0

           Write-Host $codeCoveragePercent -nonewline; Write-Host " is the Code Coverage. Failing the build"

           exit -1

 

}

 

$codeCoveragePercent = $coveredBlocks * 100.0 / $totalBlocks

Write-Host "Code Coverage percentage is " -nonewline; Write-Host $codeCoveragePercent

 

 

if ($codeCoveragePercent -le $desiredCodeCoveragePercent)

{

           Write-Host "Failing the build as CodeCoverage limit not met"

           exit -1

}

Write-Host "CodeCoverage limit met"

Script for Visual Studio Team Services( VSO )

This script for VSO is different because of the way we authenticate the user account. Also we would be using REST API to get the data. This script would work for both hosted and on premise build agents.

Before you can use this script, three values need to be set according to your environment. I have highlighted them.

1. The first two edits are going to be the placeholders for user name and password/token used to connect to VSO.

For getting this basic authentication settings you can check here - https://www.visualstudio.com/en-us/integrate/get-started/auth/overview

You can go under your profile

Then go under security and select personal access tokens.

You will have to generate a new token and give in the user name and the token info.

I would recommend using tokens, but you can go with Alternate access credentials also.

2. The next edit is going to be your account.

Once this is done, you can use the script.

param(

       # Getting the control percentage as an argument

       [int]$desiredCodeCoveragePercent=95

)

Write-Host"Desired Code Coverage Percent is "-nonewline; Write-Host $desiredCodeCoveragePercent

 

# Setting a few values

[int]$coveredBlocks=0

[int]$skippedBlocks=0

[int]$totalBlocks=0

[int]$codeCoveragePercent=0

 

# Getting a few environment variables we need

[String]$buildID="$env:BUILD_BUILDID"

[String]$project="$env:SYSTEM_TEAMPROJECT"

 

# Setting up basic authentication

$username=<basicauthfromVSO>

$password=<TokenfromVSO>

 

$basicAuth= ("{0}:{1}"-f$username,$password)

$basicAuth=[System.Text.Encoding]::UTF8.GetBytes($basicAuth)

$basicAuth=[System.Convert]::ToBase64String($basicAuth)

$headers= @{Authorization=("Basic {0}"-f$basicAuth)}

 

$url="https://<account>.visualstudio.com/DefaultCollection/"+$project+"/_apis/test/codeCoverage?buildId="+$buildID+"&flags=1&api-version=2.0-preview"

Write-Host$url

 

$responseBuild= (Invoke-RestMethod-Uri$url-headers$headers-MethodGet).value |selectmodules

 

foreach($modulein$responseBuild.modules)

{

       $coveredBlocks+= $module.statistics[0].blocksCovered

       $skippedBlocks+= $module.statistics[0].blocksNotCovered

}

 

$totalBlocks=$coveredBlocks+$skippedBlocks;

if ($totalBlocks-eq0)

{

       $codeCoveragePercent=0

       Write-Host$codeCoveragePercent-nonewline; Write-Host" is the Code Coverage. Failing the build"

       exit-1 

}

 

$codeCoveragePercent=$coveredBlocks*100.0/$totalBlocks

Write-Host"Code Coverage percentage is "-nonewline; Write-Host$codeCoveragePercent

 

if ($codeCoveragePercent-le$desiredCodeCoveragePercent)

{

       Write-Host"Failing the build as CodeCoverage limit not met"

       exit-1

}

        Write-Host"CodeCoverage limit met"

 

 

I have commented the scripts to make it self-explanatory to a small extent.

The important aspects of the scripts are

1. It takes one argument – this would be the benchmark value of Code Coverage percentage. This gives us the flexibility to change the control value in the build definition, without having to modify the script. Also this can help you re-use the script across multiple build definition

2. We do a comparison of the control value and the code coverage of the build. If it is less than the control value we exit with -1 and this would fail the build.

If we create the new task and then set the path for the PowerShell script we are good to go.

The argument is going to be the amount of code coverage expected. I tested the definition against a build that has 13% code coverage.

· When I set the argument to 10%, the build succeeds

· When I set the argument to 95%, the build fails as expected

Content created by – Venkata Narasimhan
Content reviewed by – Romit Gulati
Feel free to let us know if you needed any clarifications/improvements.

Comments

• Anonymous
  November 20, 2015
  Hi, I followed your instruction and get the following error? Exception calling "GetService" with "1" argument(s): "TF30063: You are not authorized to access Do I need to grant more permission to my default build agent?

• Anonymous
  November 20, 2015
  Hi, I got the following error when running script, am I missing something? Exception calling "GetService" with "1" argument(s): "TF30063: You are not authorized to access

• Anonymous
  November 24, 2015
  Hey Eric! We've added the script to add VSO as well. That should help!

• Anonymous
  April 26, 2016
  Hey, great article! However, I keep getting this error:##[error]Exception calling "GetTeamProject" with "1" argument(s): "Object reference not set to an instance of an object."I tried but I can't figure out where it's coming from. Any help would be greatly appreciated!I'm using TFS on-prem btw