Share via

The Minefield Myth (Part 2) – The value of regression testing

  • Article

Last week I discussed the fallacy of the minefield analogy misrepresented by some people to suggest regression testing as uninteresting or unlikely to reveal new or important information.  Their premise is that executing the same test is similar to walking in someone’s footsteps through a minefield. While this argument seems logical on the surface, this interpretation of the minefield analogy trivializes the importance of regression testing. However, there may be specific instances where regression testing (especially the execution of redundant, poorly designed, basic scripted test cases that simply reuse hard-coded data or mindlessly follow a prescriptive steps to retest unchanged or unaffected areas of code) may not provide great value to the organization. Some of those situations include:

  • retesting scripted, or simple procedural programs
  • retesting programs that are relatively static (unchanging),
  • retesting programs with no internal or external dependencies
  • rapid testing approaches where the test strategy is to simply sample the program behavior in order to provide a quick assessment and find some bugs

So, everything being equal, I would tend to agree that walking through someone else’s footsteps in a minefield may be somewhat redundant. But, there is a huge difference between redundant testing and effective regression testing. Redundant testing has zero probability of revealing new or valuable information. But, well-designed tests are not all equal, and in iterative and agile software development lifecycles some portion of effective regression test suites have some reasonable probability in exposing new information. For example, regression testing is typically useful where:

  • the underlying code base is changing (new features, refactoring, or bug fixes)
  • poor or incorrectly used revision control processes
  • a change in one module affects other modules in software developed using object oriented and procedural programming paradigms
  • the complex system has other internal and external dependencies
  • the software is regulated by some governing authority or law (Sarbanes-Oxley, FDA, FAA, etc)
  • the system is highly critical
  • an established baseline is required or important
  • legacy code bases where new functionality can unintentionally destabilize older code
  • code bases contain lots of ‘spaghetti’ code that is simply patched together
  • the results of a well-designed regression test suite helps instill a sense of confidence in the decision makers

In these contexts, an effective regression testing strategy can not only help identify important and potentially destabilizing changes in expected functionality quicker, but can also increase overall confidence in the product; especially in key areas.

Overcoming some common misconceptions of regression testing

One common misconception of regression testing strategies is that all tests are equal and all documented (manual or automated) tests go into the regression test suite. But, the simple fact is that not all tests are equal, and not all tests need to be re-executed repeatedly (every build) throughout the development lifecycle. For example, a test to check for proper tab order on a dialog or property sheet doesn’t need to be re-executed on each new build unless the UI elements have changed on that dialog or property sheet. So, in most situations this type of test is probably a poor candidate for inclusion in a regression test suite. (As a side note, if the UI elements are changing every build, I wouldn’t even test tab order until there was at least some semblance of UI stabilization.) When redundant test cases or tests that don’t provide great value to the overall testing effort are slovenly added to the regression test suite to simply increase code coverage or to artificially inflate the number of tests for ‘feel-good’ generally only results in an overloaded test suite that rapidly grows out of control and becomes a management nightmare.

Another common misconception of regression testing strategies is the suggestion that unit tests are sufficient for retesting code churn. Of course, this is simply a fool-hearty assumption and idealistic. Unit tests are a type of smoke tests performed by developers. The primary purpose of unit testing is to verify a method or function does what it is supposed to do, and may include ancillary negative tests as well. However, unit tests generally do not include comprehensive data coverage, data permutations, combinatorial analysis of parameters, etc. (which is why API testing is important in a well-rounded test strategy). Also, unit tests are usually executed in a “clean-room” environment using stubs or mock-objects, although in some cases unit tests are reran on private builds before the code is checked into the main build tree as a form of low level regression tests. While there have been significant advances in both static and dynamic analysis tools to increase the robustness of unit tests, if unit testing were the answer for effectively evaluating code churn I suspect a lot of testers would be out of a job and testing would simply be a rudimentary process of behavioral-type acceptance testing performed by non-technical ‘end-user-like’ individuals poking about the UI looking for errant anomalies and subjectively evaluating their perception of ‘usefulness’ of a product.

Another common misconception of regression testing strategies is that regression tests are less valuable because of their ineffectiveness in identifying or exposing ‘new' bugs. This is a myopic view of testing that focuses on the current version of the product, and on ‘bug-finding’ as the main purpose of testing. Of course, value is very subjective, and industry reports suggest that less than 15% of the bugs reported during a product development lifecycle are detected via regression testing. But, if even half that number are critical issues that would cause the company to pay upwards of $100K to release a patch (or even worse, the issue leads to a major calamity) then regression testing (especially highly automated regression testing) is most probably worth the time and effort. Of course, the cost of designing effective regression test cases is quite high and executing those tests requires valuable resources. So, we should not just consider the legal or financial liability issues to the company. We should also consider the product shelf-life as a factor. An effective suite of well-designed regression test cases will not only help provide a baseline assessment of the current version, but a significant portion of the test suite will/should be reused during maintenance or sustained engineering of that product for upwards of 10 years, and some subset of those tests in that suite will also be reused on the next release or version of the product. There are additional benefits to well-designed regression test cases (or any test case for that matter) in that it preserves knowledge and helps eliminate or at least reduce tribal knowledge and hero worship mentalities.

Some effective regression testing strategies

The regression test suite is essentially a set of test cases that provide a baseline measure of expected, or important functionality, and to verify previously fixed bugs do not reoccur. Based on those assertions, test cases included in the regression test suite must be carefully selected to prevent overloading or rerunning unimportant tests during a regression test pass. I generally recommend 4 types of test cases for inclusion in a regression test suite:

  1. Test cases designed to verify critical functional attributes and capabilities of the software program
  2. Test cases designed to validate baseline functionality/behavior specified in project requirements or user acceptance criteria
  3. Test cases designed to verify functional anomalies that are found and fixed during the software development lifecycle
  4. Test cases designed to evaluate collateral or dependent areas associated with a code fix

Even with these targeted tests the regression test suite can grow quite large. Within the regression test suite categorize the test cases by the feature or component that test directly targets, and sub-categorized to identify dependent modules or features. Associating test cases in the regression test suite with the primary features or components they are designed to evaluate allow the tester to prioritize the regression test pass to focus initially on the modules in which code churn occurred,  followed by dependent or interrelated modules, and finally the other remaining tests in the suite. Also, prioritize test cases within each category based on criticality. Prioritization of tests is another mechanism we can use to help us better identify an appropriate set of tests based on the constraints of available time and resources.

Finally, automate, automate, automate! In order to gain the full benefit of an effective regression test suite (similar to the build verification/acceptance (BVT/BVA) test suite ) automate as many regression test cases as is reasonable possible. But, I am referring to well-designed automated test cases; not just a bunch of simple, elementary UI script monkeys. While some regression test cases will be very targeted prescriptive tests designed to verify specific functionality, most well-designed test cases include variability in the execution while still evaluating the hypothesis, or purpose of the test case. Also, some test cases in the regression test suite will be executed through the UI layer, but in applications where the functional code is separated from the UI layer some regression test cases are more effectively executed below the UI or through an abstraction layer.

Similar to other testing approaches, regression testing is an organizational investment, and its value to the team must be considered as an investment and not done simply because it seems like the right thing to do. In some cases it may not make much sense to invest heavily in regression testing, but in cases where a missed regression in functionality results in fines, legal costs, or potential loss of hundreds of thousands of dollars in post-production costs I suspect regression testing is a critical component of the overall testing strategy to validate a baseline and instill confidence.

Comments

  • Anonymous
    January 29, 2009
    PingBack from http://www.anith.com/?p=3745

  • Anonymous
    February 10, 2009
    The comment has been removed

  • Anonymous
    February 13, 2009
    The comment has been removed