Share via

Good tools may hide symptoms but not cure the disease

  • Article

A few weeks ago a question was asked on an internal mailing list about how the unit test framework part of VS2010 works. Basically some client had decided to move away from that framework in favor of NUnit. Even though I think they did the right thing I think they did it for all the wrong reasons. The problem they encountered was that tests from multiple test classes were run at the same time and also the class initiators for one class ran before the class cleanup for another one was completed. If I understood the problem correctly the problem was not with their unit tests but with a number of bigger integration tests that were implemented using a unit test framework. Still that is not their problem I think. the problem is that they had tests that used some common resource (probably a database) and that it was used so that fixtures overlapped. Switching to a framework that happened to not run tests concurrently (like VS2010) nor in random order (like xUnit.net) did not fix their problem. It just suppressed the symptoms. It's like talking pain killers for a broken leg. It may feel good for the moment but it will not help you in the long run.

What they should have done was to fix their test fixtures. There are three things they could have done;

  • Do not used shared fixtures. The obvious advice but it may make tests run slower and sometimes you have some resource that just needs to be shared.
  • Protect shared resources. Sometimes you may have some resource you just have to share. A better approach would have been to protect it in the test code by using a lock or something. Forcing the tests to not run in parallel.
  • Share fixtures but not data. Some fixtures takes a long time to setup and it is OK to share those fixtures amon lots of tests. But you should not share the data used by each test. For example consider a fixture that needs to create a database and add data to it. If you have 20 tests that needs a customer record in the database; have each test use different customer records. That way you share fixture but not data.