the future of software testing (part 4)

Moving Testing Forward

There is a gap that exists in testing that is eating away at quality, productivity, and the general manageability of the entire development lifecycle. It is the gap between when a bug is created and when that same bug is detected. The larger the gap, the more time a bug stays in the system. Clearly that’s bad, but pointing out that the longer bugs stay in the system the more expensive they are to remove is what we’ve done in the past.

What we’re going to do in the future is close the gap.

But closing the gap means a fundamental change in the way we do testing. In 2008 a developer can introduce a bug, quite by accident mind you – our development environments do little to discourage that, and few concerted attempts are made to find the bug until the binary is built. We insert bugs and then simply allow them free reign until far too late in the process where we depend on late cycle bug finding heroics to bail us out.

As software testers we provide a valuable set of bug finding and analysis techniques; what we have to do in the future is apply these techniques earlier in the process, far sooner than we do now. There are two main things I foresee that will help us accomplish this. One is simply not waiting for the binary and applying our tests on early development artifacts. The second is building the binary earlier so we can test it earlier.

Let’s take these in order beginning with ‘testing on early development artifacts.’ During late-cycle heroics we apply any number of bug finding strategies on the binary through its published interfaces. We take the compiled binary or collection of assemblies, byte code etc, hook them to our test harnesses and pummel them with inputs and data until we ferret out enough bugs to have some confidence that quality is good enough (perhaps I’ll cover measurement and release criteria in a future blog entry). But why wait until the binary is ready? Why can’t we apply these test techniques on architecture artifacts? … On requirements and user stories? … On specifications and designs? Can it be possible that all the technology, techniques and testing wisdom collected over the past half century applies only to an artifact that executes? Why aren’t architectures testable in the same way? Why can’t we apply what we know to designs and storyboards? Well the answer is that there is no good reason we don’t. I actually think that many progressive groups at Microsoft do apply testing techniques early and that in the future we’ll figure out how to do this collectively. Testing will begin, not when something becomes testable as is the case now, but instead testing will begin the moment there exists something that needs testing. It’s a subtle but important distinction.

‘Building the binary earlier’ is the second part of this but doing so represents a technological hurdle that needs jumping. In 2008 we write software component by component and we can’t build the whole without each of the parts being ready. This means that testing must wait until all the components achieve some level of completion. Bugs are allowed to sit for days and weeks before testing can be brought to bear on their discovery. Can we substitute partially completed components with virtual ones? Or with stubs that mimic external behavior? Can we build general purpose chameleon components that change their behavior to match the system into which they are (temporarily) inserted? I predict we will … because we must. Virtual and chameleon components will allow testers to apply their detection craft soon after a bug is created. Bugs will have little chance to survive beyond their first breath.

Testing is too important to wait until the end of the development cycle to start it. Yes iterative development and agile create testable code earlier (albeit smaller, incomplete functionality) but we still have far too many bugs appearing after release. What we are doing now is not enough. The future must bring the power of testing to bear on early development artifacts and allow us to scaffold together a workable, testable environment long before the code is entirely build-able.

Comments

• Anonymous
  September 12, 2008
  Jason Barile on What I Look for in Test Plans Dave McKinstry on A New VSTS Users Group - In Your Neighborhood!!!...

• Anonymous
  September 18, 2008
  Regarding: ‘testing on early development artefacts’: I couldn't agree with you more. I’ve heard some tester call this ‘static testing’ because of the static nature of documents. Anywhere I've worked I have managed to inject degree some tester feedback into the earlier phases of the SDLC. The biggest problem I experience is that some business analysts and developers resent this kind of tester feedback. Developer more so than BA's. Some of it is delivery of feedback but I suspect most of it is related to where testers sit on the perceived pecking order. I'm glad to know that there are other testers out there that care about testing from end-to-end, not just ten minutes before it goes out the door.

• Anonymous
  February 03, 2009
  [Nacsa Sándor, 2009. január 13. – február 3.]  A minőségbiztosítás kérdésköre szinte alig ismert

• Anonymous
  February 05, 2009
  [ Nacsa Sándor , 2009. február 6.] Ez a Team System változat a webalkalmazások és –szolgáltatások teszteléséhez