Building Services: The Death of Big Up-Front Testing (BUFT)?

 Let's start with some definitions:

Death

• Demise, end, termination, the state of no longer existing

BUFT

• "Big Up-Front Testing" an activity or "Big Up-Front Test" a group that carries out that activity (the latter sounds cool, like "Big Pharma")
• It's an acronym of my own invention. At least I believe this to be the case since my Bing search only found British Upland Footpath Trust, and even they have abandoned it for the catchier UPT (Upland Path Trust). I was however inspired by BDUF, which Bing knows all about.
• The activity in question are the validation and most of the verification traditionally applied to software prior to release so as to reduce and remove defects and meet the end customers’ needs. (In honesty I just made that definition up on the fly, and I can never keep verification and validation straight…I know what they are and how to use them, it's the labels that get swapped…so best to just follow along and the meaning of BUFT will become apparent in context).

Services

• Software where the engineering team has access to monitor and deploy the service for use by the end customer.
• I know this is not the definition, and I am sure someone can point out entities called services that do not meet the criterion, but it is the definition I am using here.

Who Killed BUFT?

Why do services not require BUFT to achieve quality deployments that meet our customers’ needs?

Because we can do it all now with Testing in Production (TiP).

TiP at it’s most basic is the ability to leverage real world usage in a controlled way to find and fix defects quickly. For more details I refer you to Ken Johnston’s blog entry TIP-ING SERVICES TESTING BLOG #1: THE EXECUTIVE SUMMARY.

For the “we can do it all” statement to be true that means that the cost of TiP must be zero or near-zero. This is not the case and therefore it follows that the “we can do it all” statement is a lie.

I am a huge advocate of TiP, and I believe it is of immense value in software services testing. I even believe that TiP can reduce up-front test costs and when TiP is combined with this smaller up-front testing, the total cost will be lower and the quality will be higher.

However, TiP is not free.

 

The Cost of TiP

Deployment and Monitoring

To effectively use TiP you need to know immediately when a defect has manifested and you need to be able to efficiently deploy a fix. If you find out about your defect via your weekly conference call with customer service then you have insufficient monitoring. If your deployment is any more complicated then pushing the button and waiting for the green light [to steal Ken’s line], then you likely feel pain every time you need to deploy a fix and pain is not part of a good process. Building sufficient monitoring and automating deployment and maintaining those systems are real costs.

Cost per Bug

TiP is a great way to find bugs that might otherwise elude a traditional test environment and plan. The maxim still stands though that the cost of finding that bug after deployment is still higher then finding it before and much higher than never having created it in the first place [1].

• Testing is not Quality Assurance: A skilled QA team (even if they are called the “Test” team) can help find issues in design, code, and even requirements before the code is compiled.  For the bugs that make it into the build, a sensible up-front test regimen can help prevent many defects from getting to production.

 

 Fixing the Bug

Unless you are planning to roll back your newly deployed code with the first defect found, you need someone to rapidly ascertain the issue and deploy a fix. Companies like Amazon.com have their developers carry pagers to be on-call for exactly such duties. The cost in lost productivity for these developers and the impact on morale (read retention) for those tethered to the pager are real costs.

It’s already been mentioned you’ll need some good deployment automation, but even before that your developer needs to get the fix built and ready for deployment. Investment in automated build tools is a must for TiP.

Also, without a regression suite, or even a test framework, how does the investigator determine what the issue is and reproduce it? Your production logging better be the best – developing this is a cost. And if you truly wish to be world class you should have that test framework in place (possibly designed and built by the QA Team).

Preventing the Bug….Again

Automated regression suites are one of the most powerful tools in the toolbox. The ability to run through a suite of tests before every deployment (even better, run them continuously) gives us the assurance that we know these use cases were exercised and they work. A key practice with such suites is to add a test to exercise the logic of any known and fixed bugs (because if it happened once, it can happen again). Even if you find your bugs in production this is still a good practice, and you want a skilled QA team to build, maintain, and run this suite.

Are You the Gambling Sort?

TiP is great because by using real world data you are exercising a huge variety of use cases, including many in the long tail (i.e. edge cases…really edge cases). Still, it is statistics. With an infinite user base (or time) you will exercise all use cases, but if you know up front your key use cases, why not test them ahead of time rather than relying on the statistical probability that you will hit them? Testing with synthetic data is a great adjunct for the production data used in TiP. This is another reason to have and maintain those automated regression suites and framework.

 

TiP-ing is a QA Role

The actual procedure of Testing in Production involves planning the deployment, controlling the exposure of the new code to users, and knowing what to monitor and look for. These all sound like excellent applications for the skills of QA professionals -- and did I mention they’re not free?

Not TiP-ing is a QA Role

If you are going to apply TiP then I hope I have made it clear you’ll want to do this in some combination with BUFT (probably without the “B”). Who makes the decision on how much and what sort of up-front testing to do? Again your skilled QA team is going to make a leading contribution here.

The report of BUFT’s death was an exaggeration

Maybe it’s only had its “Big” removed. Maybe in a services world we need a QA process and a QA team that knows how to balance up-front testing and Testing in Production to achieve optimal results. “Big” is a 4 to 5 Tester to Developer ratio. This is what Microsoft uses for “shrink-wrap” software like Windows and Office. By comparison Google and Amazon use 1 to 7 (although at Amazon this varied widely from team to team, and excludes teams without any QA all). I can tell you from experience that Amazon.com pays a price for that ratio. The role of Developers changes, and not all developers think it is for the better. Amazon’s investment in monitoring and deployment is impressive, but even for all that Amazon still does up-front testing, and on some critical teams (think where the money is handled) they do a good deal of up-front testing.

Testing in Production (TiP) is a great tool that will increase quality and reduce costs but it isn’t free.

 

[Addendum Jan 21, 2010]

Credit where credit is due.

I transposed the T/D and the U when I derived BUFT from BDUF….oops. If I actually Bing for BTUF then I see someone has beat me to it. Credit goes to Thiago Arrais….. good idea Thiago.

Comments

• Anonymous
  August 28, 2012
  The comment has been removed
• Anonymous
  August 28, 2012
  Thanks for the input Ram.  You may have noticed this blog post is quite old, and although I still strongly support TiP, I have refined my thinking since this post.  Some principles that may help.

• Test up-front what you can test up-front, but no more.  Do not chase diminishing returns of up-front testing.
• Any bug that can be found easily up-front, should be found up-front (and not in production)
• Synthetic transactions, applied against production, are valuable part of TiP and are an adjunct to using real (organic) data.
• Your TiP strategy will strongly depend on your risk profile, and how you can mitigate the risks of TiP