How Test Case result is computed in Test Runner?

“I’ve NOT marked any test step BUT test case result is auto-computed as passed sometimes and failed sometimes.“

“I’ve marked a few test steps to pass / fail but auto-computed test case result is not what I had expected.”

Have you also facedsimilar kind of issues around auto-computation of test case result? Do you find test case auto-computation logic confusing sometimes? if yes then it won’t be any longer confusing for you after reading this post :)

Before understanding test result auto-computation logic, you need to understand difference between the simple test steps and validation test steps. While creating a test case in Microsoft Test and Lab Manager you can list the action and the expected result for each test step in your manual test case:

• Under Action, there is action that the tester takes to perform this test step.
• Under Expected Result (Optional) , there is result that user should expect after the action has been performed.

Simple test steps just have an action assigned to them and expected results is NOT set for them. If you set expected results for a test step, it automatically becomes a validation test step (shown by the red check mark on the step icon). E.g. in test case shown below, 2^nd and 4^th are validation test steps whereas 1^st and 3^rd are normal test steps:

In Test Runner also validation test steps are differentiated from normal test steps by an extra red check mark on the test step result icon as shown below:

So now as you understand the difference between validation test steps and normal test steps, this is simple algo used for auto-computation of test case results in Test Runner:

• If your test case has ZERO validation test steps then your test case result will be auto-computed as “Passed” unless you’ve explicitly marked one / more test steps as failed.
• If your test case has one / more validation test steps then your test case result will be auto-computed as “Passed” if and only if:
  • ALL validate test steps as marked as passed
  • NO other test step is marked as failed

In short, it is mandatory to mark Validation Test Steps. If you don’t mark any validate test step to pass / fail then it is considered as FAILED so please do mark ALL validation test steps in your test case to ensure that test case result is auto-computed properly.

Comments

• Anonymous
  June 23, 2010
  Thank you, that explains a lot. however what if I marked a test step as failed, but because it is not relevant to the final outcome , I still want to pass the test?

• Anonymous
  June 23, 2010
  There is an option to mark TC directly to pass / fail / blocked.  Please click on TC result icon / use drop down available on right hand side of Test case title in Test Runner for it.