Share via

How to: Fail a Build When Tests Fail

  • Article

Got an email from Martin Woodward this morning asking:

What would be your preferred way to fully fail the build on test failure rather than partially succeed the build?

A bit of background - Team Build 2008 marks builds as Succeeded if no errors are encountered during the build process, Partially Succeeded if compilation succeeds but an error is encountered at some other stage of the build process (e.g. a unit test fails), and Failed otherwise.  Depending on where you are in your development process, this logic may or may not make sense to you.  For example, if you are two weeks away from shipping your product, a unit test failure might be just as important to you as a compilation failure. 

The design of Team Build 2008 doesn't really allow for setting the overall status of the build directly - the setting of the build status is what indicates to TFS that the build is "complete", which triggers the build queue logic to start up the next build, etc.  As such, you really shouldn't ever set this property of the build yourself!

You can get around this by setting the properties Team Build uses to determine the overall status of the build, however - CompilationStatus, TestStatus, and an internal logger property that keeps track of whether any errors have occurred.  Note the above algorithm for determining status one more time - Succeeded if no errors are encountered, Partially Succeeded if CompilationStatus is Succeeded but an error occurs at some other stage, and Failed otherwise.  So - if you want the overall status of the build to be Failed, you'll need to set CompilationStatus to Failed.

So - here is my recommended approach:

   <Target Name="AfterTest">

    <!-- Refresh the build properties. -->
    <GetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
                        BuildUri="$(BuildUri)"
                        Condition=" '$(IsDesktopBuild)' != 'true' ">
      <Output TaskParameter="TestSuccess" PropertyName="TestSuccess" />
    </GetBuildProperties>

    <!-- Set CompilationStatus to Failed if TestSuccess is false. -->
    <SetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
                        BuildUri="$(BuildUri)"
                        CompilationStatus="Failed"
                        Condition=" '$(IsDesktopBuild)' != 'true' and '$(TestSuccess)' != 'true' ">

  </Target>

The idea here is to check whether or not tests have succeeded (see my previous post here on this topic) and then update the CompilationStatus (and thus the overall Status) of the build accordingly.  If you want the build to halt immediately in addition to marking the status as Failed, you can use an Error task, and you'll probably also want to move the logic to the AfterTestConfiguration target (to avoid running tests for each configuration before halting the build).

Comments

  • Anonymous
    November 06, 2007
    The SRL Team Blog on Start working with eclipse and VSTS (basic needs). Tiago Pascoal on TFS: Fetching...
  • Anonymous
    November 17, 2007
    The comment has been removed
  • Anonymous
    November 19, 2007
    Ah yes - there was a bug in earlier versions of the SetBuildProperties task where these properties were of their real enum types (BuildPhaseStatus/BuildStatus) rather than strings.  I had assumed - wrongly - that MSBuild could manage the conversion for me.  I believe the issue should be fixed in Orcas Beta 2 - let me know if I am incorrect here.  If so, it is definitely fixed in the RTM version, which should be released here shortly...  Sorry for the confusion!
  • Anonymous
    January 04, 2008
    Aaron,We're getting some funky behavior with this using MaxProcesses=2.With MaxProcesses=2, we're getting TestSuccess=False using the code below.  All unit test have passed.Thoughts?tim <Target Name="AfterTest">   <GetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)">     <Output TaskParameter="TestSuccess" PropertyName="TestSuccess" />   </GetBuildProperties>   <Message Text="TestSuccess='$(TestSuccess)'" />
  • Anonymous
    January 29, 2008
    The comment has been removed
  • Anonymous
    May 12, 2008
    As promised, here are some more details on other SP1 changes for TFS Build.&#160; 3. Detect test results.
  • Anonymous
    October 15, 2008
    Not sure if this is because I am on TFS 2008, but simply specifying the following in a <PropertyGroup> achieves the same.<TreatTestFailureAsBuildFailure>true</TreatTestFailureAsBuildFailure>