Share via

Analyzing Test Results in MTM - Part II

  • Article

Introduction

In the previous blog post Analyzing Test Results in MTM– Part 1 we have seen how to find the run and a snapshot of various test run analysis features. In this blog post we will look at Analyzing a test failures using Test case result view and how to get a final report. In the sample run provided in the blog post there were 11 inconclusive results and 5 failures. We have analyzed 11 inconclusive results in the previous post using Test run view. Mow let’s analyze the 5 failures.

 

Test case result view

There are 5 failures in the run and let’s analyze each one of them using the above features. Since these are genuine failures in the run we need to open each test result to get more information on the failure from the logs. As usual user can double-click on the test result or Click “View Results� Button to view the test case results. The below analysis screen comes up where it displays all the data that the user is looking for. Browse through the below screenshot and take a look at the data displayed in each of these fields.

Test_Case_Result_View

In the above screenshot you can see related items Combo Box which will let you open the Test plan, Test run and the test case that the test result belongs to.

Related_Items

Attachments

Based on the data collectors turned on in the test settings, user will find all the log files associated to the run in the attachment section. If you have enabled video collector, you shall even look at what exactly happened during the test execution. Users can also write custom data collectors to gather specific information that they wanted from the test machine which will help them analyze the results. Note that if the attachments are huge and coming from multiple agents there may a time lag for all the attachments to get into TFS. In these cases you will attachments with 0KB of size, this means the files are getting uploaded to the server.

Attachments

Even the TRX file has been saved here. Opening the TRX file will open it VS IDE. User can investigate the failure and then update the analysis information in the results individually.

TRX

Test result History

Users can view the past test result history in the same view where they can find if the same issue has happened earlier. \

Test_Result_History

Update analysis information

Based on the investigation users can update the test result with analysis information. The below screenshot shows updated fields. Like this we shall analyze all the other 4 failures.

 

Update_Analysis_Information

 

When is analysis complete?

Notice that unless there are no failures with failure type value “None� or Resolution type of “None� or “Needs Investigation�, the analysis is not complete. Since all this analysis information is preserved, it is meaningful to analyze all run failures and update all the fields with appropriate details so that if the same issue happens in future it can easily be auto-analyzed. Also all the analysis data gets moved to warehouse where users can get historical reports about the past test runs and can get cool and meaning reports using excel reporting feature.

 

Final Analysis chart

After the analysis of the tests the final results looks like this. The user get to see some pie charts which gives a snapshot of the results based on failed test and analysis of those tests

Final_Analysis_Chart

 

Summary

With this blog post Analyzing Test Results in MTM series is complete. In this blog we have seen following in detail

- Analyzing a test failures using Test case result view

- Analysis complete indicator and final report

Click here to get to Analyzing Test Results in MTM - Part I

 

Author – Venkatesh Sargunesan

Test Lead – CodedUITest

Comments

  • Anonymous
    June 25, 2010
    I am using TFS, MTM and Coded UI Test for my application. Requirements are in TFS with Parent-Child relationships. For every requirement, I have created test cases(sometimes one test case per requirement and often many test cases per requirement). In creation of automation scripts, ideally it should be an automated Test Method associated with each test case(assuming 100% coverage) but in my application test cases are so granular that I am unable to create automation script for a single test case but for a group of test cases( which combines to form a feature of application). In this case, following problem occurs: Linking: How to link my automation Test Method to a group of test cases? Is there a way I can link my group of manual test cases to an automation script which combines to form a scenario/feature? Any responses will be appreciated. If anyone else has a similar problem/solution or other approach to this problem, please share. Thanks

  • Anonymous
    May 25, 2011
    Currently I am not seeing any options to select resolution while analyzing any failed test case. It is only showing None. Is this permission issue? Please suggest .

  • Anonymous
    January 31, 2012
    I'm having the same issue as Ankur, in that I'm not seeing any options for resolution other none for a failed test case. This is an analysis of a manual test run.

  • Anonymous
    April 16, 2013
    Analyzing Test Results in MTM– Part 1 Mow let’s analyze the 5 failures There are 5 failures in the run and let’s analyze Click “View Results” Button Notice that unless there are no failures with failure type value “None” or Resolution type of “None” or “Needs Investigation”, the analysis is not complete what is with the gibberfied characters? did you use the wrong encoding? ...