Kohl on "Reckless" test automation and test debt
Sometimes, I think Jonathan Kohl is living inside my head (you know, as in the film "Being John Malkovich").
His recent posts on Reckless Test Automation and Testing Debt (from October 2005) are dead on. IMHO, you just can't automate everything. It doesn't make sense, and there's just no replacement for simple human interaction with a product. A little manual testing can go a long way. The RTA article, in particular, illustrates how "fully automated" testing can actually give your team a false sense of security and miss obvious bugs that manual testing could quickly find.
That said, it doesn't make sense to abandon automation either. The trick is knowing what to automate and when to automate it. I'm of the opinion that end-to-end automation early in the development cycle is basically a waste of time. At early stages, unit testing is much more useful in testing the fundamentals of your product. As the user scenarios take shape and final UI layers are completed, only then should you consider implementing end-to-end automation. But always keep in mind that the spec really isn't stable until the product is out the door - and even then sometimes it isn't. If you think your product is different, you're probably just kidding yourself.
Things people tend to forget about test automation:
- Test automation is software development (read Kohl's 10/30/2005 post on the topic) Automation is a "product" in the sense that it requires design, development, testing (yes - you have to test your tests!), documentation, and maintenance. Usually though, you only have testers available to do all this, while the "real" product likely has dedicated resources for each of those roles. So obviously corners will be cut - either in flexibility, documentation, engineering practices, or (gasp) actual test coverage. Don't forget to code review that automation! There's nothing worse than missing a bug in the product because of a bug in the automation when you thought you had the scenario covered.
- If your product spec isn't stable, don't expect automation to help much. While you might be making what feels like a minor spec change, the impact to a whole suite of test automation might be prohibitive. A good example here is when we added Team Projects to Team Foundation Server. Until that point, much of our test automation for version control created files/folders at the root level of the repository for simplicity of testing server logic (i.e. prove you can add a file or folder, check it in, delete it, check it in, etc.). When Team Projects were introduced, we had to go update all our automation to "bump up" a level and do all its work in a team project folder. It might sound like a minor change, but it actually took a significant amount of time to fix up existing tests and re-verify they were all doing the right thing. You can try as much as you want to anticipate upcoming spec changes like this and build flexibility into your automation, but don't create a battleship kit when what you really need is a jet ski.
- Myth - "it's just test code". Anyone who believes this and believes you should automate everything just hasn't thought through the problem deeply. Test code must be designed, efficient, maintainable, documented, agile, etc. On top of all that, it will likely get gutted and re-written several times throughout an agile product cycle. If you're going to rely on test automation (or "computer assisted testing" as Kohl puts it), you're probably going to want skilled software developers to get the job done. Don't forget about the maintenance phase! Having detailed docuementation for your automation strategies, harnesses/tools, and test cases thoroughly will be invaluable a couple of years from now when your developement team has moved on to new product versions and someone new is in charge of regression testing in maintenance mode.
Jonathan Kohl really has his head on straight as far as I'm concerned. I have thoroughly enjoyed all his posts, and I'm looking forward to more.