Condividi tramite

Customize your Test Case work item template to enable query on test suites, test plans, test categories and execution statistics

  • Articolo

[The following is a guest blog entry from a Microsoft internal user]

At our team, we use Microsoft Test Manager 2010 to manage our automated test cases. It provides great flexibility to compose test suites based on a static set of tests (static suite) or based on a TFS query (query-based suite). However, after using the system for a while, we find ourselves hoping for even more flexibility to compose query-based suites based on test categories or test suite membership. The default “Test Case” work item template does not have fields to capture these information, so they are not available for query out of the box. With some work item template customization and custom tools written using TFS API, it could all be implemented with a few hours of work! In this blog post, we would like to share with you our customizations, so you can extend them to further enhance your test management experience with Microsoft Test Manager.

Here is what we are trying to accomplish:

  • Enable creation of query based suites based on test category (annotated by “TestCategory” attribute)

    • Example: all tests with test category of "Daily Smoke"

  • Enable creation of query based suites based on a combination of static test suite membership and other work item attributes

    • Example: a subset of tests belong to “Test Suite A” whose owner is “John Doe”

  • Enable creation of tracking query or query based suites based on automation test execution statistics

    • Example: tests with average pass rate below 30%
    • Example: tests with average test run time over 20 minutes 

Here is the elements of our solution:

  1. Customize the “Test Case” work item template to add a set of fields to capture the additional information: test suite membership, test categories, and test execution statistics, etc.
  2. A command line program that updates the TFS TCM periodically to keep the information in #1 up-to-date

Work item customization 

Let’s first look at how the “Test Case” work item customization looks like. Below is a screenshot of our custom fields in the “Process Editor” power tool.

 

And with “Process Editor”, we also customized the form to show these new fields on the UI. Here is how they look like in our template:

Information Update Tool

 We wrote a tool using the TFS and TCM API to enumerate through the test plans and test suites and update the information in the test cases if they have changed. Only static suite membership is collected, otherwise the results might be unpredictable. The tool also looks for test assemblies in the latest build drop folder and reads the metadata in the assembly to get a list of test categories. If the test categories have changed, the tool will update the test case custom fields. Finally, the tool queries and computes the automation execution statics and write them to the test case so you can easily query for them. This is an area that you might want to further customize from the sample code. For example, you might want to be more selective about what tests you want to count as part of your average test run time. A test may run into failures very quickly and results in a much shorter test runtime that would skew the results.

We use a Windows scheduled task to run this tool every 30 minutes. You can determine what would be an appropriate interval for your team. The TFS connectivity information and custom field information is configured in the app.config file, and you would need to run the tool with a credential with appropriate TFS access.  

The sample code is attached with this blog post. They are provided as is and for illustrative purpose only.

Conclusion

Here is a glance of what you can do with the added fields. We hope this information would help you get some ideas on how to better organize and tracking your tests. We hope you could see that TFS offers a very flexible platform and a robust set of APIs that gives you the freedom to implement a process that works best for your team. I would like to thank the VS ALM Team for giving me access to this forum to share this with the community.

Joe Chiu

Software Design Engineer in Test, Interactive Entertainment Business

TfsTestcaseUpdaterSample.zip

Comments

  • Anonymous
    September 21, 2011
    This is fabulous thanks for sharing.

  • Anonymous
    January 29, 2015
    The comment has been removed

  • Anonymous
    August 26, 2015
    This is really cool! I am surprised I did not run into it 4-5 years ago :-). One issue with importing test results into a field is that Work Item History likely gets updated each time. You will get a mighty big history and potentially push DB size outside expected ranges due to such frequent field churn. An alternative would be to log test results then put a link or an Excel query to pull stats from the warehouse or other Test Result repository separate from work items. Thanks for sharing your customizations!