Are all SysTest tests unit tests?

In my last post, I categorized automated regression tests into two broad categories – UI and headless. I placed SysTest tests in the headless category since the framework can’t test UI by design. In this post, I’m going to discuss different types of headless tests – particularly in relation to unit testing.

In Inside Microsoft Dynamics AX 2009, the SysTest framework is introduced in a section in Chapter 3 titled “Unit Test Tool”. The section refers to unit testing throughout. It stands to reason that any test that you write with SysTest must be a unit test…

Before I continue, I do realize that I’m entering into a debate that can become religious for some developers. Nevertheless, I will continue…

One of my favorite definitions of unit tests comes from Michael Feathers, author of Working Effectively with Legacy Code, offers these Unit Test Rulz:

A test is not a unit test if:

1. It talks to the database

2. It communicates across the network

3. It touches the file system

4. It can't run correctly at the same time as any of your other unit tests

5. You have to do special things to your environment (such as editing config files) to run it.

Another very good source for unit testing is a blog post written by Steve Sanderson title “Writing Great Unit Tests: Best and Worst Practices”. From a unit test definition point, I like the continuum between two Sweet Spots: True Unit Tests and Integration Tests. In between these Sweet Spots is what Steve calls Dirty Hybrids.

Hmm… a developer would need to do a whole lot of mocking to meet either of these definitions of “true unit tests” in AX. It seems that most SysTest test cases will not meet these definitions.

So if most test cases written with SysTest in AX don’t meet the rigorous definition of a unit test, what good is the framework? Both sources actually explain that to us. Feathers immediately follows his ‘rulz’ with the following comment:

“Tests that do these things aren't bad. Often they are worth writing, and they can be written in a unit test harness.”

Sanderson explains that Sweet Spot B, Integration Tests, are the most valuable tests for detecting regressions in the product. When written correctly, these are black box tests that require limited maintenance.

To answer the question in the post title – no, not all SysTests are unit tests. If you use a rigorous definition of unit tests, the vast majority would not be considered true unit tests.

But that doesn’t mean tests created with SysTests won’t be valuable to your organization. As I noted in my last post, headless tests are more reliable, higher performing, and more maintainable than UI tests. Here’s the guidance that I would suggest:

1. Creating tests near the True Unit Tests Sweet Spot as suggested by Sanderson is a valuable practice for developers. Realize that it will take some effort to build up the necessary mocking tools to be True Unit Tests.

2. Creating tests near the Integration Tests Sweet Spot as suggested by Sanderson can be very valuable for finding regressions and is likely the best starting point for automated testing. Make these tests as black box as possible and focus on the business logic that you are developing. A key challenge is to keep the business logic contained in Classes and Tables, away from Forms since SysTest can’t access Form logic.