MSTest does not call all tests in a class before moving on the the next test class

I recently heard of somebody who moved away from using MSTest and started to use NUnit because NUnit called the class setup method, then all tests and last the class tear-down before moving on to the next test class while MSTest more or less interleaved everything. I'm not a big fan of MSTest so I'm absolutely not defending MSTest but I think the team that made this decision moved to NUnit for the wrong reason. First of all I want to make it clear that the problem was not with a set of unit tests, it was with an integration test suite that was implemented using a unit test framework. And being an integration test it neaded to setup a lot of things so it wanted to do the setup only once for each test class (i.e. suite). The problem was that several suites worked on the same data and hence could not run concurrently. I think that was the real problem. Each test fixture, regardless of if it is setup for a single test or a test suite must be isolated in it's context. This means that if you setup a database you should have different suites working on different parts of the data so that they can coexist. This is a good idea to avoid weird test failures because a cleanup method is not correctly implemented. If you're testing a network service you need to start it using different entry points (i.e. listening to different network ports) so that they don't interfere with other services being tested at the same time. Because if you design your tests in that way you will not be limited by the framework, it will just be a tool for you and you can change the tool more easily if you need for another more valid reason.

Comments

• Anonymous
  February 23, 2011
  I think that team moving to NUnit is completely right: I would expect exactly the same behavior. Otherwise it is just defeating the purpose of [ClassInitialize] vs [TestInitialize]. IMHO unit test must be designed to be called in any order and without making assumption of the current system state, however it is just stupid to get rid of the ability to setup complex environment once for all in [ClassInitialize] (aka. [TestFixtureSetUp] in NUnit). Regards.