How to keep your Test Plans / Suites in sync with your Unit Test assemblies
In my last post I gave an overview of what automated testing looks like in VS / TFS 2010 with the new testing features. In 2010 we’ve added Test Plans which contain Test Suites which you can think of as a folder hierarchy to separate and organize your tests within. Test Suites contain Test Cases (well, Test Points really, you can see my previous post here if you’re interested in the distinction) which you can run from Microsoft Test Manager (MTM.exe). Test Cases can either be Manual or Automated, for a Test Case to be automated you need to associate the Test Case work item with one of the VSTS 2010 automated test types which includes Unit Tests, Web Performance Tests, Load Tests, Coded UI Tests, etc. The most common scenario here is probably to associate your Unit Tests with Test Cases.
To manually create that association you perform the following steps:
- Open your Test Project and Connect to your TFS 2010 Server
- Create or open an existing Test Case
- Switch to the Associated Automation tab
- Click the … button next to the Automated test name textbox
- Choose the test in the list you want to associate the work item Test Case with, in this simple example I only have one so I’ll go ahead and choose that and hit OK, then you’ll see the following data filled in.
- Save the Test Case.
Now when you add this Test Case to a Test Suite in Camano, associate your Test Plan with a build, and then Run the Test MTM will go to the corresponding build share to find the test assembly and run the associated unit test.
As you can see there’s a point where things quickly become unmanageable if you’re forced to manually manage these associations. My teams Unit Test assembly contains approximately 650 Unit Tests that we run through Microsoft Test Manager every night. We’re pretty well locked down now of course but there was a long time during the product cycle during which we were adding new tests and removing deprecated tests quite regularly. It would have been very time consuming to keep our MTM Test Suites in sync with these changes to our Unit Test assembly (and that’s not even considering the amount of time it would have taken to manually create 650 Test Case work items and then associate each one with the corresponding Unit Test just to get started). We saved a lot of time using the TCM.exe command line tool that is included with the product to do this for us so I wanted to dive into that process here.
First, here’s a tip you may find useful if you are going to be using TCM.exe against the same server / project frequently.
- Navigate to your VS Install Root\Common7\IDE
- Open TCM.exe.config
- Set the TeamFoundationServer and TeamProject app settings there, so you don’t have to specify them on every command you run.
Here’s a set of fake Unit Tests I’ll use to demonstrate some of the features of TCM.exe:
[TestMethod]
[TestCategory("BVT")]
[Priority(1)]
public void Apple()
{
//Test something
}
[TestMethod]
[TestCategory("BVT")]
[TestCategory("NotLocFriendly")]
[Priority(1)]
public void Banana()
{
//Test something
}
[TestMethod]
[TestCategory("BVT")]
[Priority(1)]
public void Pear()
{
//Test something
}
[TestMethod]
[TestCategory("NotLocFriendly")]
[Priority(2)]
public void Kiwi()
{
//Test something
}
[TestMethod]
[Priority(1)]
public void Lemon()
{
//Test something
}
[TestMethod]
[TestCategory("BVT")]
[Priority(1)]
public void Orange()
{
//Test something
}
This set of tests includes tests across 3 different Priority levels and 2 Test Categories (including some tests not in a Test Category). If you’re not familiar with the [TestCategory] attribute, I believe it’s new to this release. It’s a nice way to mark your unit tests in such a way that you can query them from MSTest.exe so you can easily run certain subsets of your tests without having to list them all out individually. We also support this in TCM.exe for the testcase import commands.
In my Test Plan I have a Suite Hierarchy like this:
Here goes first let’s check out the command we’ll use. As you can see in the help output below we can import our tests with as little input as simply pointing to the assembly with the “/storage” option. That would import all of the Test Methods in the entire assembly though and it would only create the associated work items rather than actually add them to our suites.
We’ll want to use the /syncsuite and /category options. That runs us into our first issue though, we don’t know our suite id’s, and they aren’t displayed in the MTM UI anywhere. We can get the Test Plan Id from the UI by going to the Test Plan Properties page. TCM.exe includes the ability to query this information from the server though, since I already know my Test Plans Id from the UI, I can do:
Now I have the Suite Id’s I need for the /syncsuite option and can proceed.
And we can see that Orange, Pear, Apple, and Banana were Added, here’s what our Test Suite now looks like:
What happens when I run the command again? We don’t create an additional set of Test Case work items, we just update the existing ones.
That’s it! I’ll skip going through populating the other Test Suites this way, hopefully you get the idea from the example above. Instead, let’s take a quick look at what the Test Case that the import command created looks like.
The “Assigned To” was set to the account I’m running the import as, if you specify the “Owner” attribute on your Unit Test it’ll use that value instead. Priority was set to the value on the Unit Test attribute also. You can also see that the Associated Automation info was set for us just as it was in the manual example at the beginning of this post.
That’s it for this post, hope you find it useful.
Comments
Anonymous
March 12, 2010
You're absolutely right! Test Categories are new in the 2010 Team Test framework. Here's a blog post about how it relates to running them as part of an automated build as well: <a href="http://www.edsquared.com/2009/09/25/Test+Categories+And+Running+A+Subset+Of+Tests+In+Team+Foundation+Server+2010.aspx">Test Categories and Running a Subset of Tests in Team Foundation Server 2010 Build</a>Anonymous
March 12, 2010
(Redo - since there is no HTML allowed in comments and the link above is broken. Feel free to delete the previous comment!) You're absolutely right! Test Categories are new in the 2010 Team Test framework. Here's a blog post about how it relates to running them as apart of an automated build in TFS 2010: http://www.edsquared.com/2009/09/25/Test+Categories+And+Running+A+Subset+Of+Tests+In+Team+Foundation+Server+2010.aspxAnonymous
April 12, 2010
Off Topic - I am using VSTS 2008 (Test edition), I testing a simple asp.net website which just have a ‘Default.aspx’ page with a text “Testing” and this is how my ‘web.config ’ looks like: <?xml version="1.0"?> <configuration> <appSettings/> <connectionStrings/> <system.web> <compilation debug="true"/> <authentication mode="Windows"/> <sessionState mode="InProc" cookieless="true" timeout="2"/> </system.web> </configuration> Every time I record a webtest, I get the below error:
- Failed WebTest6 TestProject3 1 primary requests and 0 dependant requests failed
- Response URL Validation The value of the ExpectedResponseUrl property 'http://localhost/WebSite/(S(sfu4s2y522la2vavuro5jb45))/Default.aspx' does not equal the actual response URL 'http://localhost/WebSite/(S(we2x0y45w3hk0ormurslyn45))/Default.aspx'. QueryString parameters were ignored. Due to the above error, VSTS is not able to “detect any dynamic parameters to promote” and hence I cannot run my webtest because the “ cookieless sessionID” gets expired after the timeout. Please suggest any pointers… I googled but no luck so far…..