Escape the BVT automation trap.
Escape the BVT automation trap.
Create outstanding BVTs on time and on budget.
The build verification tests (BVTs) for your product are important to get right. Automated BVTs are a very powerful tool in your test arsenal. They allow you to know if a build is worth testing in a short amount of time with little or no user intervention. If you tie them to your official build system you can usually have results waiting for you when you arrive to work in the morning.
You need a clear plan for moving from manual BVTs to automated ones. The methods that make good manual BVTs aren’t a good fit for automated BVTs. In fact they are a trap! You need another set of principles to work from.
Don’t fall into the clunky, slow BVT automation trap. Bad automation doesn’t justify the cost to create it. Testers will idled for hours every day while someone tracks down the root causes of failures. If you are in the trap you can lose days of productivity when it matters the most, crunch time. The bait in the trap is complex, customer focused scenarios. It’s yummy and can be irresistible to testers and managers. A good idea at the wrong time turns bad.
Imagine BVT’s that are fast, reliable, trustworthy and useful. This is the ideal that you should be shooting for. This gets you out of the trap where BVT’s try to be all things to all people.
Create your BVTs with the simplicity (also known as K.I.S.S.) principle. Simpler is better when it comes to BVTs. Resist the urge to use complex tools to solve simple problems. Instead use simplicity itself to create outstanding automated BVTs. Simplicity means optimizing for automation goals and leaving other testing goals for other tests. BVT automation goals include things like being fast, maintainable and diagnostic. BVT automation non-goals include core customer scenarios and ensuring features are correct.
BVT automation is a tool.
Just as a hammer is a great tool for nail, BVT automation is a great tool for a specific purpose. It’s not a catch all place to dump all the automation you have ever created for your product. Be vigilant and keep things out that don’t belong. Good BVTs systems have certain qualities in common. Keep your tests simple and keep your BVTs focused on their mission.
A good chance that testing can proceed is enough what the tool is for.
The purpose of good BVTs is a quick way to tell if a particular build is worth testing further. The “worth testing” threshold will be different for every product and change as the cycle progresses. If the BVT system can wave you off a bad build even half the time (50%) it’s probably paid for itself. You reach the point of diminishing returns quickly. In order to have a 80% chance of testability you need to do a ton more work just to get 30% better. Don’t waste time. Predicting possible build breaks is a trap. Observe your product over time and add tests where they will do the most good. Adding a BVT for each detected build break may be like closing the barn door after the cows are gone. Ask yourself if this test will catch future breaks. If the answer isn’t “probably” then don’t add it to the BVT suite.
Fast results are money in the bank.
When you build a new version of your product you want the green light as soon as possible. BVT systems that take hours to run are wasting valuable cycles. When a developer has a potential check in they should be able to run BVTs in a matter of minutes. If it running the tests requires a full build and human intervention in a customized environment you won’t be getting very good leverage on your BVTs. I have seen BVT systems that were so top heavy that a check in late in the game could cost an entire day just to build, get on the test bunch and run a “sanity” pass. Don’t fall into that trap.
An automated test in the hand is worth two in the bush.
You want to deliver BVT automation as early in the process as possible. You can solve a lot of problems with enough time. But when you are delivering BVTs time is the first problem you need to solve for. If you have a great idea for an object model for the product but it will take weeks to code, don’t wait on it to create your BVTs. Go for a few cheap (and yes, possibly dirty) tests you can deliver today.
BVT automation needs to trusted
If your team has no faith in the results they get from the BVTs you have wasted your time writing them. When a BVT fails everyone should believe that the results are valid. You won’t be right 100% of the time of course, but you can come close. If your BVTs are failing on a regular basis because of setup issues or other reasons outside actual product flaws they lose a lot of their punch. If you find yourself in this situation you may need to rethink your BVT design. Get away from complex brittle scenarios and move to something simpler. If BVT failures are catching more problems with the environment, the network of other “noise” you either need new BVTs or a more solid environment or both. Make this a priority or manual BVTs will be the only thing you can trust.
Create fast diagnostic BVTs
Fast BVTs make your automation more trustworthy. You can use the short feedback loop to iterate them until they are rock solid. Make sure BVTs can be run quickly. Then run lots of iterations and verify they are good. Create BVTs with diagnostic results. A BVT that points clearly to a method in code is superior to one that can’t isolate failures. When you detect a break you want to be able to zero in on the offending part of the build and fix it very quickly. Couple highly diagnostic with a quick execution time and you cut the time you need to fix a break to practically nothing.
Create adaptable BVTs
Adaptable BVTs are more trustworthy. They will spend less time being irrelevant. Your product will change out from under the tests as the cycle progresses. Adapt the BVT test suite to these changes quickly. It’s ok and even desirable for an individual test to be brittle. By making the BVT tests as simple as possible they are less likely to go wrong. If the product changes you can easily decide what tests are obsolete and need to go and what new tests need to be created. Complex tests that rely on intricate libraries to interact will idle testers while fixes come in. Fixing a giant test library that was made with assumptions about the product that are no longer true can be time very consuming. If the library owner is on vacation or has left the project you are in real trouble. Keep individualt tests simple to keep the overall suite adaptable.
Create maintainable BVTs
Simple code leads to maintainable code. Make your tests as maintainable as possible. The bugs are clear. The repro scenarios are clear. You can easily prove the automation is working in the debugger. Developers can easily suggest fixes for faulty tests that are easy to understand. When you are on vacation and one of your tests fail you won’t be getting a panicky call or email. If the time comes to hand the product off to another team or the customer to maintain, they can easily run and update your tests.
BVT automation needs to be manageable
An automated system of tests has to have a lot of moving parts. This can make the systems difficult to manage. Help to keep the complexity down by leveraging things you will be doing anyway. Running BVTs isn’t free. There is a cost in hardware, support and time. Make wise use of these resources.
Leverage unit tests
Unit tests are really good BVT tests. Typically all you need to do is get them to run in your BVT system and output reports. If you work with developers up front and support them they will often be happy to write the tests to in whatever harness the test team is using. If this isn’t possible for some reason, it’s still easy to adapt or port the tests. Another key way to leverage the unit tests is to simply create more of them. Once a tester sees one unit test for the code it’s easy to use it as a template to add other tests. Developers often only create a functional test and maybe a negative test. You can improve these by adding boundary tests and tests that randomize valid inputs. This is a simple and fast way to crank out a lot of simple, fast and dependable tests that will last the life of the product.
Leverage simple component tests
Subsets of component tests are really good unit tests. My favorite is a test that checks to see if the component installed correctly. That is a dead simple test to make and a fantastic BVT. Don’t worry about all the core scenarios for the component. Save that for other automation. Just look for quick high level tests that are simple. Checking for performance counters, log entries and configuration files are all good examples. Checking the correctness of those same things is more complexity than the BVT system should have.
Leverage core API functions
If your product has simple to use APIs they make good BVTs. Say the product has a web services XML interface listening for HTTPS connections. A good test would be to check if it’s actually listening on the port. Listening on the port is really core to the way the API works. Enumerating users in the user table is a compelling scenario but save it for your other automation suites.
Will you find the right balance for your BVTs?
Creating a set of BVTs that prevents wasted time is harder than it seems. Prevent customer scenarios from luring you into a bloated BVT system. Customer scenarios are a Good Thing™. You must do them. Just don’t do them at the wrong time when they will cost you more than they should. Create your BVTs with the simplicity principle. Simple BVT tests are the path to having a great BVT system. A great BVT system focuses your team on shipping a superior product.
Keep asking if your BVTs are simple enough. Then make your product awesome with the help of simple, streamlined BVTs.