Data rollback

A major challenge in testing, particularly data driven applications like ERP systems, is managing data during the test execution. For a test to execute reliably, it needs to start from a known state… every single time. How can we make that happen reliably? Fortunately in AX7, a significant testability feature was added to the SysTest framework that enables this behavior.

Before describing the new feature, let's dive into the problem a bit further. A book that I highly recommend, XUnit Test Patterns: Refactoring Test Code, uses the term fixture to describe the setup for a test. There are several different fixture patterns, but I will focus on only a couple.

The fresh fixture pattern creates a new environment for each test method. This is the most common approach for tests at the bottom of the test pyramid, primarily unit tests (see this post on the test pyramid). This typically produces the most reliable tests as there is no possible carryover from previous tests.

The shared fixture pattern creates an environment that is reused by many tests. As tests get broader in scope as in integration or business cycle tests, more and more data needs to exist for the test to execute in a reasonable amount of time. This is where a shared fixture is frequently used, but it has its downsides. A significant challenge can be inadvertent state carryover that impacts downstream tests. This situation impacts test reliability, can be very frustrating to debug, and results in lost confidence in the test suite.

An ERP system, with its mass of back end, interconnected, data, lends itself to a shared fixture for tests broader than unit tests. Internally, we've come up with different approaches to address the data sharing and fixture tear down challenges over past releases - things like test suites and various isolation approaches. Most of these were "bolt-ons" to the SysTest framework, and all ultimately had some drawbacks.

For AX'7', we focused on reliably rolling back data at different layers directly within the SysTest framework. To accomplish this, the transaction savepoint feature in SQL Server was leveraged. Savepoints are created at the start of each class and each method, enabling nested rollback to these known states as part of the teardown process. Because it leverages native SQL capabilities, it is both reliable and fast.

Data rollback using SQL transaction savepoints is the default behavior for all SysTest classes. You can choose to disable it at the class level using the SysTestTransaction attribute with the 'NoOp' value. This might be helpful while debugging tests, but you should almost always take the default behavior.

Now that we have reliable and fast data rollback, we've discouraged the use of legacy test isolation mechanisms. This has resulted in big reduction of 'flaky' tests… and that has been a huge benefit to our engineering efforts!