Udostępnij za pośrednictwem

VSTS Gems- Identify flaky tests

  • Artykuł

VSTS is a great platform, but did you know about its gems?
Follow this short post series where we undercover some of its coolest features.

Gem

Identify flaky tests

<click> on image for animation.

View flaky tests in the Test hub:

Area

Test

TFS Availability

Future

Value

  • Increase confidence in automated test process effectiveness
  • Save time & Shorten the MTTD (Mean time to detect) of the Flaky tests

Full Description

Sometimes tests are flaky - they fail on one run and pass on another without any changes. Flaky tests can be frustrating and will undermine confidence in test effectiveness - causing failures to be ignored and bugs to slip through.  You can now configure the Visual Studio Test task to re-run failed tests. The test results then indicate which tests initially failed and then passed on re-run. Pay attention to flaky tests, they are telling you something :)

The Visual Studio Test task can be configured to control the maximum number of attempts to re-run failed tests and a threshold percentage for failures (e.g. only re-run tests if less than 20% of all tests failed) to avoid re-running tests in event of wide spread failures.

You can refer to DevOps at Microsoft on Eliminating flaky tests

Find more gems on the Rangers blog

Find more gems on Twitter