Diagnostic Data Adapters: Changing how Developers and Testers work together (Part 1 of 2 - The Test Impact Collector)

Last time I talked about remote execution and introduced the concept of Diagnostic Data Adapters in Visual Studio 2010. In this post and the next, we will jump right into working with two of our favorite Diagnostic Data Adapters: the Test Impact Collector and the Diagnostic Trace Collector. Both of these components have a part to play in how developers and testers interact in Visual Studio 2010, making it easier for each discipline to unlock the potential of the other. While a key pieces of the diagnostic data adapter story is the freedom for teams to create their own adaptors and tailor their data collection to better suit their needs, we plan to include several diagnostic data adapters that we feel will be useful to many teams and projects right out of the box. The remainder of this post (and the next one) will go into detail about these two great diagnostic data adapters and how they help address a couple perpetual sticking points between developers and testers: “What did you test?” and “No Repro.”

The first point we address is the eternal question “What did you test?” This question is often asked by developers of testers, testers of each other, and management of everyone whenever a bug makes it into your customer’s hands. There are many ways different groups attempt to answer this question, but the Test Impact Collector aims to bridge the gap between developers and testers to show you what needs to be tested and which tests need to be run. In the world of development and unit tests it is a fairly straightforward effort to enable code coverage and see what parts of the code have been called by your tests. The hassle of collecting coverage data, and providing consistent coverage through manual tests, is beyond the expectation for many testers. The development team often knows only by feature area what the test team has been up to, and vice versa for the test team to know what the development team has been trying to fix. The Test Impact Collector serves a vital role here by identifying for development which code changes will or will not be exercised by the test team’s existing test catalog, while it also helps testers identify which tests from that catalog they need to run in order to validate the new behavior their developers have included in a given build.

Configuration of the Test Impact Collector is very simple. Manual testers using Microsoft Test and Lab Manager (codename Camano) will simply run tests according to the plan configured by their test manager or a specialist tester. Configuring the test plan simply requires creating a Test Settings for the tests to use and marking the appropriate diagnostic data adapters for use during a test run, as seen above. Each diagnostic data adapter exposes configuration options. For the Test Impact Collector there are two lists that can be used as either white or black lists of processes and modules to either include or exclude from instrumentation during the test run. The Test Impact Collector will collect the names of methods exercised during the tests. This allows the team to focus their efforts on just their code and reduce the number of modules that will be instrumented at the start of the test case, or, to eliminate modules that are known to not be of interest during the test.

This diagnostic data adapter allows Visual Studio to inform developers when their code changes will not be covered by the test team’s existing tests. Knowing this before the change is made will allow the development team to take extra care in reviewing their work or warning the test team before new or unexpected code shows up in the application under test. From the perspective of the test team, the Test Impact Collector helps answer the question “Which tests do I need to run in order to test the new changes?” Collections of test cases may be grouped by feature areas, but these sets may be much larger than necessary to test small changes within those areas. Rather than expecting teams to run all the tests within a given area, the Test Impact Collector helps teams reduce the number of tests they must run without leaving out any tests that do exercise the newly changed product code.

The above battery of test cases to run can be reduced to the below list with the help of the Recommended Tests option within Microsoft Test and Lab Manager.

The Recommended Tests feature takes into account results from the Test Impact Collector as well as the changes to the product source between any known builds associated with that Team Foundation Server project. These configuration options help test teams keep pace with new development by always running the test cases that will put the new changes through their paces.

Gone then are the days, when the test teams mindlessly run an entire battery of tests for a new build, irrespective of the new changes that showed up there. Now testing will be much more efficient and leveraged. As each new build comes out, the system will know the exact changes that have occurred, and the Test Impact collector will flag for the testers which subset of their test cases they need to run to validate the new functionality.

That’s pretty neat!

(Staying on the topic of Diagnostic Data Adaptors, and how Developers and Testers work together, I will talk about the Diagnostic Trace Collector in the next post).

Comments

• Anonymous
  April 10, 2009
  PingBack from http://asp-net-hosting.simplynetdev.com/diagnostic-data-adapters-changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-collector/
• Anonymous
  April 12, 2009
  Thank you for submitting this cool story - Trackback from DotNetShoutout
• Anonymous
  May 19, 2009
  I am very excited proud and proud to introduce the new testing features you’ll find in dev10. In addition
• Anonymous
  June 10, 2009
  Following up on Brian Harry’s blog , I would like to provide a bit more details on Test Impact Analysis
• Anonymous
  August 15, 2010
  Though the feature is cool, the question still remains. Developers asking of testers- why did you not include my module or ensure you remember to test it at some point and hope that module really is tested at some point before making it into the hands of customers. You could though very well use this for intermittent testing cycles (more often than not!) than the final ones but that is purely a tester's (team's) call.Makes sense?-Phani
• Anonymous
  March 10, 2015
  "No Microsoft .NET applications were launched during the test run. Launch applications from Explorer or a new command prompt after a test case has started. If no .NET application is being tested, disable the Test Impact and IntelliTrace diagnostic data adapters." I am getting the above error, although my application is .NET based and I am launching my application using Explorer and its deployed on IIS. Please suggest a solution to this.