Running selective unit tests in VS 2012 RC using TestCaseFilter

New unit testing platform in Visual Studio 2012 provides a new way to selectively execute test based on filtering condition through TestCaseFilter. TestCaseFilter can as specified as a string while executing tests through command line (vstest.console.exe), Team Build (when running test using “Visual Studio Test Runner”) or through Test platform client API.

Support for adapters to leverage this feature is provided in test platform. MSTest adapter shipped with Visual Studio 2012 for executing managed test provide basic filtering as described below (more enhanced filtering expression support to be added in future versions).

 

 

Specifying TestCaseFilter:

1. Command Line

   New command line runner for executing tests in VS 2012 (vstest.console.exe) has an optional command line parameter /TestCaseFilter for specifying filtering expression. /TestCaseFilter cannot be specified with /Tests argument.
   E.g. vstest.console.exe test.dll /TestCaseFilter:”TestCategory=Nightly”

2. Team Build

   When adding a test activity in build definition, if user selects to execute test using “Visual Studio Test Runner” then user get to specify test case filter (optional can be left blank to run all tests).

3. Client API

   When using Test Platform Client API for executing test then filtering expression can be specified in TestRunCriteria.TestCaseFilter property.
   E.g.
   TestRunCriteria runCriteria = CreateTestRunCriteria(); runCriteria.TestCaseFilter = “TestCategory=Nightly”;

 

 

Syntax for filtering expression:

Adapters can choose their own syntax for filtering or take advantage of support provided by Test Platform by MSTest adapter syntax. Following is syntax for MSTest adapter for managed test execution. Invalid filtering expressions will be ignored and all tests will be executed.
   

Operators supported in RC are:

1. = (equals)

2. != (not equals)

3. ~ (contains or substring only for string values)

4. & (and)

5. | (or)

6. ( ) (paranthesis for grouping)

   Expresssion can be created using these operators as any valid logical condition. & (and) has higher precedence over | (or) while evaluating expression.

E.g.    

"TestCategory=NAR|Priority=1"
"Owner=vikram&TestCategory!=UI"
"FullyQualifiedName~NameSpace.Class"
"(TestCategory!=UI&(Priority=1|Priority=2))|(TestCategory=UI&Priority=1)"

"Priority~1" // Invalid as priority is int not string

Properties supported by MSTest adapter for filtering are

1. Name=<TestMethodDisplayNameName>
2. FullyQualifiedName=<FullyQualifiedTestMethodName>
3. Priority=<PriorityAttributeValue>
4. TestCategory=<TestCategoryAttributeValue>
5. ClassName=<ClassName> (Valid only for unit tests for Windows store apps, currently not available for classic MSTest)

 

Using TestCaseFilter in Test Adapters

Adapters can leverage default format of TestCaseFilter that MS Test uses. Parsing and matching is done by ITestCaseFilterExpression provided by test platform.

Following method from IRunContext is for getting an implementation of ITestCaseFilterExpression.

 ITestCaseFilterExpression GetTestCaseFilter(IEnumerable<String> supportedProperties, Func<string, TestProperty> propertyProvider);

ITestCaseFilterExpression provides following method to match test case (only if MSTest format is used for TestCaseFilter)

 bool MatchTestCase(TestCase testCase, Func<string, object> propertyValueProvider);

 

E.g.

    1: /// <summary> 
    2: /// Supported properties for filtering 
    3: /// </summary> 
    4: private static Dictionary<string, TestProperty> supportedPropertiesCache;
    5:  
    6: supportedPropertiesCache = new Dictionary<string, TestProperty>(StringComparer.OrdinalIgnoreCase); 
    7: supportedPropertiesCache[“Priority”] = PriorityProperty; 
    8: supportedPropertiesCache[“TestCategory”] = TestCategoryProperty; 
    9: supportedPropertiesCache[“FullyQualifiedName”] = TestCaseProperties.FullyQualifiedName; 
   10: supportedPropertiesCache[“Name”] = TestCaseProperties.DisplayName; 
   11: supportedPropertiesCache[“MyProperty”] = MyProperty;
   12:  
   13: ITestCaseFilterExpression filterExpression = runContext.GetTestCaseFilter(supportedPropertiesCache.Keys, (p) => PropertyProvider(p));
   14:  
   15: foreach (TestCase currentTest in tests) 
   16: { 
   17:     // Skip test if not fitting filter criteria. 
   18:     if (null != filterExpression && filterExpression.MatchTestCase(currentTest, (p) => PropertyValueProvider(currentTest, p)) == false) 
   19:     { 
   20:         continue; 
   21:     }
   22:  
   23:     // Execute test
   24:  
   25: }
   26:  
   27: /// <summary> 
   28: /// Provides value of TestProperty corresponding to property name 'propertyName' as used in filter. 
   29: /// Return value should be a string for single valued property or array of strings for multi valued property (e.g. TestCategory) 
   30: /// </summary> 
   31: private static object PropertyValueProvider(TestCase currentTest, string propertyName) 
   32: { 
   33:     TestProperty testProperty; 
   34:     if (supportedPropertiesCache.TryGetValue(propertyName, out testProperty)) 
   35:     { 
   36:         // Test case might not have defined this property. In that case GetPropertyValue() 
   37:         // would return default value. For filtering, if property is not defined return null. 
   38:         if (currentTest.Properties.Contains(testProperty)) 
   39:         { 
   40:             return currentTest.GetPropertyValue(testProperty); 
   41:         } 
   42:     } 
   43:     return null; 
   44: }
   45:  
   46: /// <summary> 
   47: /// Provides TestProperty for property name 'propertyName' as used in filter. 
   48: /// </summary> 
   49: private static TestProperty PropertyProvider(string propertyName) 
   50: { 
   51:     TestProperty testProperty = null; 
   52:     supportedPropertiesCache.TryGetValue(propertyName, out testProperty); 
   53:     return testProperty; 
   54: }

 

 

If you've any feedback or issue then please do send us on Connect or in the Forums.

Comments

• Anonymous
  July 18, 2013
  We are using Team Build with TFS 2012.  I tried using ClassName as a filter, but the following error appeared in the log: "No tests matched the filter because it contains one or more properties that are not valid (ClassName). Specify filter expression containing valid properties (TestCategory, Priority, FullyQualifiedName, Name) and try again."