Automated Regression Test Qualities

 

In my last post, I discussed the need for a balanced approach to testing efforts. The goal, informed by the context of the project, is to consider all four quadrants of the agile testing matrix.

In this post, I'm going to turn my focus to a testing activity that is typically found on the Support Engineering half of the matrix - automated regression testing. First, let me describe what I mean by 'automated regression' testing as these are terms that can be overloaded in the software test world.

Automated - an 'automated' test is one where the system under test is exercised by a computer and not by a human. It does not mean "have the computer use the system under test like a human would use it". It is this second inference that drives some software test industry leaders to never use the term ‘automated’ or ‘automation’ and prefer something more like 'computer assisted testing'.

Regression - a regression happens when something that used to work no longer works. In the software world, this typically happens when a change is introduced - either directly by an engineer or by an change in a dependency.

Putting these together, automated regression testing involves a computer exercising the system under test to identify and/or prevent changes in the software that cause it to no longer work as it did before.

When creating the set of automated regression tests, the goal is find the most effective test design that will support the engineering effort. Software test teams sometimes will try to serve too many needs with their 'automation' efforts by using it as their primary bug finding activity. While bugs will be found while developing automated tests, there are far more efficient ways to find important bugs that customers care about. I expect to cover exploratory testing in a future post.

Defining desirable qualities in an automated regression test helps us to achieve an effective test design. The following graphic is one that we're referred to frequently during the Dynamics AX'7' project:

While these qualities seem fairly obvious, they are deceptively difficult to achieve in practice. In my next post, I will address different test granularities that will lead to an automated regression suite with these qualities.