I Have a Dream…No Regressions

OK, so “I have a dream” isn’t original, and perhaps neither is my dream. However, the path to my dream is the road less traveled, and I believe we should have a super-highway instead of a dirt road.

Imagine you could build software without fear of breaking existing behavior. That’s my dream. As a developer, I want to be able to constantly improve our architecture to make it easy to add new functionality. But I don’t want to break anything. I don’t want to break the build. I don’t want to break someone else’s code. I don’t want to break the product. And I don’t want to break a customer’s scenario after shipment.

You may be wondering if this is even possible. I believe it is through a suite of automated regression tests. In our case, it’s mapping Mike Cohn’s test pyramid, which I first encountered while reading the book Agile Testing, into the Visual Studio/Team Foundation Server stack.

Traditionally I’ve heard people say it’s the job of testers to ensure quality. Rubbish. It’s the team’s responsibility. Without developer involvement, it would be very expensive to get to the goal of no regressions. As developers, it’s our job ensure our code is testable. I’ll go into more detail on this later.

Path to No Regressions

In my experience working on teams, there are several things that lead to regressions and loss of productivity for the team. On one team, we were constantly fighting with build breaks. Before checking in I would get the latest code from source, only to find it wouldn’t build. How frustrating is that. Then once I got it to build, behavior we depended on would be broken, so we’d have to figure out who’s code it was and work with them to get a fix. Double grump. I really don’t want to go through that again.

Here is a short list of things you can do to prevent this type of problem, which I’ll cover in more detail below:

• Gated check-ins
• Suite of automated regression tests
• Failing tests for new bugs

At the core of all of this is discipline. Our team has to believe in “No Regressions” and work together to achieve that dream. My job is to help inspire the team to achieve this goal.

Automating Regression Testing

Automated regression tests should run automatically. No, that’s not redundant. In one team I worked on, it was a huge effort to get the regression tests to run. Testers needed to manually start a test run. And even then, the run often required manual intervention in order to get the tests to run. Ideally a regression suite should run automatically, without any intervention. And the results should be easy to access by anyone on the team.

It’s also important to have the right mix of tests. On the left is a version of Mike Cohn’s test automation pyramid found on p. 277 of the book Agile Testing. I’ve modified this slightly. The idea is that each level of the pyramid provides a general idea of how many tests should be at that level. Each unit test should exercise just one public method of a class in isolation, should run in less than 20ms, and should run on any computer. Developers should be writing the tests in the bottom section of the pyramid.

The upper two levels of the pyramid are generally where testers will create automated regression tests, but we as developers still have a responsibility in these two levels—we need to design and write our code so testers can write tests below the UI, where they can be less fragile and run faster.

Stage 1: Unit Tests & Gated Check-Ins

Here is where we enter the dream. All of the tests in the pyramid should be run automatically and frequently on servers. Today our team’s unit and component tests run automatically whenever we check code into Team Foundation Server. We’ve set this up with gated check-in turned on. If the build fails, or if any of the unit tests fail, the check-in is rejected. In other words, we don’t have build breaks anymore. Woo hooo!

Of course, we can still break the product, and we have. This is where the next two layers come into play.

Stage 2: Acceptance Tests

The next level up is where things get trickier, especially when we’re writing software that works with server software, such as System Center Configuration Manager in our case. Here we want/need to run the tests in a real environment. Of course, this means the tests can have side effects that destroy the integrity of the environment. I once wrote a simple test that ended up corrupting a SharePoint database so you couldn’t even run the administration tools. I had to wipe out the databases and create new ones.

This is where Visual Studio Lab Management comes into play. You can configure a set of virtual machines in an isolated lab. Once you have this all setup, you can spin off a new isolated lab for each test run. This means you can have several developers/testers running tests or debugging software with “real” environments without stepping on each other (this is big!). You also have a well-known environment each time you spool up a test run. Finally, and this is one that sounds pretty exciting to me as a developer, if a test fails, you can go back to the snapshot of the lab at the point where the test failed, which means you can actually look at the repro environment. Very nice! I want it now!

Stage 3: UI Tests

The last layer is the smallest because as many of the acceptance/integration tests as possible should be below the UI. UI tests tend to be fragile, slow to run, and hard to maintain. As a result, you want just enough UI tests to ensure that the UI still works. Without support from us developers, testers are usually forced into writing far more UI tests than headless tests below the UI.

If we take it as a given that we have good testing in the layers below, testers can use Test Manager to create recorded tests that are used as smoke tests. Now imagine that these tests are also run in Lab Management automatically. Wouldn’t it be great if every night the build server would build the latest source and then kick off all of the tests in the pyramid?

Stage 4: One Button Delivery

I decided to throw in one more goal because I’ve just started to read the book Continuous Delivery. The dream here is that you can deliver/deploy quickly and easily with a single button click. Then if anything goes wrong, which is detected automatically, the system automatically rolls back to the last known-good state.

Where Next?

I don’t know if our small team will realize this dream, but I certainly hope so. I will blog more about our progress towards this dream as we make that progress.

Comments

• Anonymous
  October 20, 2011
  Great Article, John. Love the way you think.