Jaa

ClassCleanup May Run Later Than You Think

  • Artikkeli

In addition to the well-known TestInitializeAttribute, the Visual Studio Team System unit testing framework also includes both initialization and clean-up attributes for both individual test cases, classes and assemblies. Although the use of these attributes may seem straightforward, some methods decorated with these attributes may not run in the order you would intuitively think (or maybe my intuition is just weird). Let's do a little quiz! Consider these two test classes:

 [TestClass]

 public class TestClass1

 {

     public TestClass1() { }

  

     [AssemblyInitialize]

     public static void InitializeAssembly(TestContext ctx)

     { Debug.WriteLine("AssemblyInitialize"); }

  

     [AssemblyCleanup]

     public static void CleanupAssembly()

     { Debug.WriteLine("AssemblyCleanup"); }

  

     [ClassInitialize]

     public static void InitializeClass(TestContext ctx)

     { Debug.WriteLine("TestClass1: ClassInitialize"); }

  

     [ClassCleanup]

     public static void CleanupClass()

     { Debug.WriteLine("TestClass1: ClassCleanup"); }

  

     [TestInitialize]

     public void InitializeTest()

     { Debug.WriteLine("TestClass1: TestInitialize"); }

  

     [TestCleanup]

     public void CleanupTest()

     { Debug.WriteLine("TestClass1: TestCleanup"); }

  

     [TestMethod]

     public void MyTestCase1()

     { Debug.WriteLine("TestClass1: MyTestCase1"); }

 }

  

 [TestClass]

public class TestClass2

{

    public TestClass2() { }

 

    [ClassInitialize]

    public static void InitializeClass(TestContext ctx)

    { Debug.WriteLine("TestClass2: ClassInitialize"); }

 

    [ClassCleanup]

    public static void CleanupClass()

    { Debug.WriteLine("TestClass2: ClassCleanup"); }

 

    [TestInitialize]

    public void InitializeTest()

    { Debug.WriteLine("TestClass2: TestInitialize"); }

 

    [TestCleanup]

    public void CleanupTest()

    { Debug.WriteLine("TestClass2: TestCleanup"); }

 

    [TestMethod]

    public void MyTestCase2()

    { Debug.WriteLine("TestClass2: MyTestCase2"); }

}

There are two test classes that are basically identical. The only real difference is that TestClass1 hosts the AssemblyInitialize and AssemblyCleanup methods in addition to the other methods, which basically all just write to the debug window that they are executing. What will be the output if you run all (both) these tests in debug mode?

As you may have guessed from the headline, the tricky part is ClassCleanup. Before I began to think about it, I just assumed that the test run would spin up TestClass1, execute the tests in that class and clean it up, then proceed with TestClass2 in the same manner.

Obviously, I was forgetting that unit tests are not ordered. Although they are executed sequentially, the order in which they are executed is not guaranteed. This is, in fact, a good thing, because it helps you remember that test cases should always be independent.

In any case, here's the result from my Output Window:

AssemblyInitialize
TestClass1: ClassInitialize
TestClass1: TestInitialize
TestClass1: MyTestCase1
TestClass1: TestCleanup
TestClass2: ClassInitialize
TestClass2: TestInitialize
TestClass2: MyTestCase2
TestClass2: TestCleanup
TestClass1: ClassCleanup
TestClass2: ClassCleanup
AssemblyCleanup

It's hopefully no surprise that AssemblyInitialize runs first, and AssemblyCleanup runs last. Notice, however, that although TestClass2's ClassInitialize execution is deferred until needed, this doesn't mean that TestClass1's ClassCleanup executes immediately after the last test case in the class! In fact, it waits until all test cases are executed, and the executes together with TestClass2's ClassCleanup.

This surprised me at first, but that was obviously only because I hadn't really thought it through: Since tests are, in principle, unordered, there's not guarantee that all tests in TestClass1 are executed in immediate succession. Theoretically, the execution engine may pick a test case from TestClass1, then one from TestClass2, then another from TestClass1, etc. Since that is the case, there's no guarantee that all tests from one test class have been executed before a new test class is initialized, and thusly, all ClassCleanup methods may as well be deferred until all test cases have been executed.

When this first surprised me, I had written some code in one class' ClassCleanup and expected it to run before another class' ClassInitialize. What I got instead was a strange interdependence bug, where all tests in both classes succeeded when executed together with other tests from their own class, but as soon as I ran all tests from both classes, the tests in the second class failed. The lesson here is that not only should you treat your test cases as independent, but the same goes for initialization and clean-up code.

Comments

  • Anonymous
    December 10, 2008
    Nice post, but I think you do not reach the correct conclusion. Your assumptions were correct, it's the VSTS implementation that is wrong. Fast simplistic Unit Tests do not need parallel processing, and complex slow ones (Database, external Process, ..) cannot be independent if thery use some common ressources, so VSTS should be predictive, not randomistic.

  • Anonymous
    December 11, 2008
    Hi Guy Thank you for your comment. The main point of this post was to share a finding on how MSTest works, and I believe I arrive at the correct conclusion. MSTest isn't random. As far as I can tell, the algorith is completely deterministic - it's just different that you might first expect. My point about MSTest giving no guarantees about test ordering is, AFAIK, true for all the xUnit testing frameworks. It's by design, since you should never design your test to depend on executing in a specific order.

  • Anonymous
    August 27, 2009
    Interesting, I really learned how MSTest works! Keep up the good work

  • Anonymous
    March 11, 2011
    The comment has been removed

  • Anonymous
    June 17, 2012
    The comment has been removed

  • Anonymous
    December 27, 2014
    I totally agree with the above conclusions by Jostein Kjønigsen: I case you run a batch of tests and at some point a single test fails, there is no way to release its resources (as the CleanUp routine will be deferred). So all test that follow on will fail as well as they don't have access to those resources. It is actually a bug - just like allocating memory and not releasing it on exit. Just a shame that I'm writing this post almost three years after the previous one and nothing has changed...