Get good returns on your test automation dollar.

Get good returns on your test automation dollar.

 

 

Building test automation is like building a house.

Test automation seems like such a good idea. Convert something time consuming and boring into an automated process. It really is a good idea but you have to watch out for some pitfalls. Just as you wouldn’t start building a house with the paint, you shouldn’t start your test automation effort without thinking it through. Don’t fall in love with scenario tests just because they mimic the customer experience so closely.

Your customers really want you to build on a solid foundation. As professional software testers our role is to channel the customer. We are the ones they rely on to ensure their point of view is heard in the software creation process. We have to think hard about the best ways we can fulfill this vital mission. We have to make sure the software they will be using is built well and that means putting our automation efforts on a solid foundation.

There never seems to be enough time for all the automation you want. If we spent all the time it would take to automate every possible test, the product would never ship. If we don’t do any automation or have a very small set of automated tests we may not ship the product we were hoping for.

A strong foundation of good test automation has a big impact. Setting a house on a good foundation can mean the house stands the test of time. You are passionate about software. Plan your automation strategically to use your passion for the customer experience the best way possible.

Automate the lowest level of tests first; just like you would start with a foundation for a house. Product designs flow down from user scenarios. It’s tempting to approach test automation the same way. Experience shows this isn’t the best approach. Start from the bottom up if you want a fast, maintainable and balanced test automation set.

Start with unit tests and work your way up.

The levels of testing from low to high are unit, component, API and then scenario tests. Scenario tests are also known as end to end tests (E2E) and often use the exact same UI the user will. When you are writing your test automation there are a lot of reasons to make sure you have a good set of unit tests first.

Low level automation has a high return on investment.

Low level tests don't require broad product knowledge. Therefore you can get started automating much faster. Your ramp up time is reduced as compared to scenario testing. You may not even have to understand what the product does. You just need to understand what functions, components and APIs are supposed to do at a granular level.

You can write a lot of it early. Because of the relative simplicity of low level tests you can create a lot of tests early in the product cycle. You can have a few component tests done the same day a developer drops you a working build. Because you can deliver automation so quickly you can close the feedback loop for developers in a timely fashion. They can get a good gauge of how their work is progressing and what the problem areas are. This is vital to a healthy product cycle.

You can run it lots. Because it’s so well suited to running in a fully automated environment you can run the low level automation with a very low “tax”. Developers can run it whenever they make code changes. You can run it every time you build. You can run it as part of every test pass. This is a big advantage when trying to gauge the quality of your product.

It can enable you to create scenario tests robustly and more cheaply. The final advantage of a good set of low level tests is that they will inform your creation of the scenario tests. You can sand the rough edges off the product before you even try to test scenarios. You can make sure that people on the test team have an in depth understanding of the way the product is put together and works internally. You will discover if your product has API level problems or hazy specifications that need to be cleared up much earlier which will in turn make your scenario tests more focused.

API testing can often be completed automatically. If you have a well defined API that’s spelled out in technical language you can write lots of tests very quickly. You can use several approaches to generate your tests and test data. You can often create thousands or tens of thousands of API test cases in a very short period of time using tools.

Unit, component and scenario tests are a good match for early automation goals so do them early.

Scenario test have one set of advantages. They are highly customer focused. There are a nearly infinite number of interesting tests. The bugs they find are important to customers. Low level testing has another set of advantages. These advantages make them better suited to automation runs than scenario tests. There are several reasons for this.

Low level tests are more deterministic than scenario tests. Have you ever run a scenario test only to find out that it doesn’t run the same every time? You can end up endlessly tweaking and fiddling with them. You do hacky things like adding hard sleeps and resets into the code. Low level tests almost never have these kinds of problems. They run the same way every time. You can run those tests thousands of times in a row and get the same result. When you are looking at reports of the health of your product, you want to have the most reliable tests be the most numerous in the automation runs.

Low level tests are simple enough to be trusted. Nothing is worse than having the automation fail and having to carry out lengthy manual tests to figure out where the problem is. Developers will often point the finger at your test automation. Lower level tests are better in this regard. It’s easy to prove that the test is correct. It’s also easy to realize when the test is wrong and you can spend your time making the test better rather than doing lots of manual work to diagnose the problem.

Low level tests can be fast enough to run many pre-check in. Ideally your developers can run your automation before they check in changes to component code. If all you have are complex scenario tests that require a lot of setup you won’t get many people to run them. If your tests are more or less self contained and can run in a couple of minutes you are in much better shape. Developers love running tests that find bugs quickly and save them making bad check-ins. They can focus on the subset of tests that really matter to them. Since the developer will have a personal relationship with their testers they will be more willing to trust the automation and they know who to go to for help.

Low level tests are highly maintainable. When you have to hand off your test suite to someone else, low level tests are superior to scenario tests. They are usually made up of fairly elemental code. They can be put in the debugger and understood easily. Scenario tests are seldom this maintainable. This is also a big advantage when product code changes. Picking which tests to keep, which tests to fix and what new tests need to be created is much easier when considering the low level tests. When you go back to them after weeks or months you won’t be struggling to understand what you were thinking when you wrote them.

Low level tests are highly diagnostic. Sooner or later your tests will fail. It’s important to understand why the failure occurred. Product bugs need to be understood by developers quickly and completely. Test automation bugs needs to be fixed correctly by testers. When the time comes to diagnose failures lower level tests are much easier to deal with.

Scenario tests should be automated last (but not least).

After all these points you may think that I don’t like scenario tests. This isn’t true at all. But the reasons they are important shouldn’t be used as an excuse to skip the lower level automation. If you have to make cuts in automation, cut scenario testing and do it by hand. The scenario tests deserve to be run but they aren’t the highest automation priority.

Scenario tests don’t require expertise to run manually. This is not true of the unit, component and API tests. Anyone with a scenario script can run theses tests. If you have to cut automation, it’s a lot less risky to cut tests you can run by hand. These tests can be run by testers who don’t have strong automation skills. The testers won’t need in depth black box knowledge of the product. They can often be outsourced with reliable results.

Scenario tests are not diagnostic. Even if you manage to automate a large number of scenario tests they aren’t great at isolating bugs. It’s true your product should contain logging and diagnostics to make this process better. In practice these self analysis systems often have holes and blind spots that are hard to predict. Analyzing scenario results can be a very complex process. In a product with no or poor self analysis tools you will find yourself very frustrated when the scenario tests fail. You want your highly diagnostic tests to be created as early as possible. Save the less diagnostic tests for later.

Scenario tests are not easy to maintain. This is because of the complexity you must build into them. Even the simplest scenario touches on wide swaths of the product and may rely on external dependencies that are hard to deal with. Because of this complexity these tests are expensive to maintain. You can mitigate a lot of this problem with a solid library set. However the library will require expertise to create and a budget to maintain as well. You can make the problem better and move it around a little but in the end you can’t escape the complex nature of these tests. Get the maintainable tests running first, they won’t consume your life later while you are working on the harder stuff.

Scenario tests are best built on a solid framework of lower level tests. The best scenario tests are ones that build on a library of knowledge and library of well tested working code. If you can encapsulate your features into lower level tests you can set yourself up for success with the scenario tests. Your understanding of the product will be more complete and this will lead to better tests. You can use more reliable automation to setup the parts of the test not core to the particular test case. By keeping the UI driving code to a minimum you make your test suite higher quality and easier to maintain. You need to have this foundation built up before you can leverage it.

Think ahead and prioritize high return investments to keep your automation house in order.

By concentrating on the "top level" customer scenarios you often end up with a small set of tests that are expensive to run and keep. This means you should prioritize automating the lowest level of tests first.

Learn to do this well so you can become an effective advocate for quality and ultimately for the customer.

 Edit: Removed "Do Unit Testing First" sidebar. I think it's the culprit for messed up formatting some people are seeing.

Comments

• Anonymous
  November 09, 2007
  Side bar: Do unit tests first. (Note, this text was removed from the article becuase it broke the page formatting. I wanted to keep it, but not spend hours tweaking HTML.) There are a lot of reasons to make sure your product has a good library of unit tests that are highly visible to the team. Unit tests are dead simple. Automation is simplest for us when developers write it. You didn’t spend a nickel of your budget on it and you already have automation. I have seen products where there is little if any unit testing. Make sure your product has some. Integrate them into your BVT automation effort if you can. Testers can easily expand unit tests. Once the developer has created a few unit tests the barrier to entry is broken. Testers can then go in and expand the unit tests with negative and boundary tests. Another good feature of unit tests is that they are finite. For a given function there aren’t a lot of unit tests that are really worth writing. You can easily know when you are hitting the point of diminishing returns and move on. By showing the expanded unit tests to the developers they can gain a better understanding of how testing works. They also have a solid base of tests to rely on pre-check in. You can rely on unit tests the entire life of the product. Unit tests are short, fast and reliable. They are usually quick to write and explain. Being able to run lots of them quickly is a really big advantage. Since unit tests typically run in less than a second, you can run a ton of them every time you change the code.

• Anonymous
  January 31, 2015
  I agree with this. Nice article you got here.