Share via

Myth Busting Testing #2

  • Article

Myth #2: Testing does not, by itself, improve software quality.  

I love this one! I must have heard it like a gazillion times. And taken at its most literal, it is entirely correct. Huh? Yep, it's entirely literally correct. So why am I calling it a myth? Because it implies that testing and quality are not related, and that is completely false.

Reality: Without testing, you’ll never be able to improve your software quality. In order to develop better, you must test constantly.

Here’s a great quote from one of the best books on developing software, Code Complete by Steve McConnell:

“Testing by itself does not improve software quality. Test results are an indicator of quality, but in and of themselves, they don't improve it. Trying to improve software quality by increasing the amount of testing is like trying to lose weight by weighing yourself more often. What you eat before you step onto the scale determines how much you will weigh, and the software development techniques you use determine how many errors testing will find. If you want to lose weight, don't buy a new scale; change your diet. If you want to improve your software, don't test more; develop better.” - Steve C McConnell, Code Complete: A Practical Handbook of Software Construction

McConnell’s quote above is a fantastic one; it makes great copy. However, I completely disagree with the conclusion that implies that testing and developing are at odds (at the best the tone probably implies that they are orthogonal), I believe that the quote actually shows just how important it is to test more, not less. Because in order to achieve improvements in your software by 'developing better', you simply must test more. Test has to become inextricably entwined in the development process. Here’s what I mean…

You are sitting in a restaurant somewhere, perhaps it’s a Saturday, and you’ve decided to take a ride to explore highway 2, and you found a little greasy spoon diner, and since it’s lunch time, you stop in to grab a bite. As you’ve mostly polished off the blue plate special, the spatial relationship between what’s left of the open faced sandwich and the side of fries gives you a moment of elucidation: you’ve hit upon an idea to solve a problem you were stuck on for most of the week. As you design, before it’s even going down on the napkin with the crayons that are supposed to be for the kids, you are testing your ideas. You form them up, write them down, and then somewhere in version 27, after you’ve done this iteration of design and test, design and test, you eventually translate those ideas to the napkin. It’s even mostly readable, although the crayon tore the napkin a couple times. You pocket the napkin, pay the bill, and you are happily on your way again, seeing the sights on highway 2, and they seem rosier than before lunch, and they probably are, because now your mind is free from the problem it had been background processing on.

When you return home that evening, you bust out the laptop, and carefully unfold the napkin, and you start to translate from napkin to code.  Hey, you are a geek and love writing code, what else are you going to do on a Saturday night? And somewhere in the iterative process of translating from crayon wax to electrons, you’ve designed something that you believe will solve the problem. At which point, what do you do? Why, you hook this thing up to the rest of your system, you create some test data, and then you test it. You build even more test data, and try it another way. You find a corner case you missed, you fix it, and you test it again. You throw garbage data at it, and you fix a couple bugs that were there that weren’t validating some inputs. And because you are such a great developer that your ego never gets in the way of creating great code, you run through all those same tests again to make sure that the changes you made didn’t break anything.

When you get back to the office, you schedule a design review. Yep, you test it again. Once you’ve made changes to the design from that review, you implement the code, and then you have a peer do a code review. You see, you’ve measured (tested) your coding proficiency, and you know that for every 100 lines of code you write, there are probably 3 bugs. And sure enough, you find 4 during the code review (yep, you tested it again). You are ready to check it in to the source tree, so of course you run the standard set of check in tests on it. And then, because you are a great developer, you write unit tests for it so that when someone else checks some code in, or makes a change somewhere else in the code base, there's an early warning system that can signal that there's a problem.

As part of the check-in, you write a test release document on the code you’ve checked in, and you fire it off to your test partner. You know that the software design engineer in test (SDET) who will read it, will probably think of how this integrates with the larger system under test, and write scenario based tests for it. If he’s really good (and he probably is, because great developers always want to work with great testers, they have no ego problems) he probably has already had the scenarios coded up, and now he can go back in and fill in the gaps that your document has pointed out as potential areas of concern. Wow, you tested it yet again. You review the unit tests with your SDET partner, and he finds a corner case you missed, and he says he'll go ahead and add it to the test suite, and then he adds them to the check in tests and daily automation run. He can do this because the development and test teams use the same test harness for unit tests, checkin tests, functional tests, so any test that dev writes can be leveraged by test in the test lab, and any test that an SDET writes can become part of the dev unit or check in tests.

So if you want to lose weight, you do weigh yourself all the time. Why? Because you want to know where you stand. You want to make immediate course corrections, not wait a month and find out that you’ve put back on 3 pounds. You want to do it far more frequently. And you don't use just one scale. You measure yourself with a tape measure, and when you put on your jeans. You measure yourself when you look in the mirror. And while some measurements are more qualitative and/or subjective than others, all of them are useful and help you towards your goal of losing weight.

Think and Test, Design and Test, Develop and Test. Fix and Test. Fix more and Test. Then you hand it to someone else and help them test it. It’s what great developers, and great development teams, do.

Better development is better testing. If you never measure your code’s quality by testing, you'll never know if the quality is sufficient.

I will say it again (because I can, it’s my blog, and because it’s important)

Without testing, you’ll never be able to improve your software quality. In order to develop better, you must test constantly.

Do you have that kind of relationship (communication and tools) with your test partners? Shouldn't you? At Microsoft's engineering excellence days during the last 3 years, we've talked alot about 'pushing quality upstream.' This is half of what I think it means. There's more on what test and pm can do as well. What do you think 'pushing quality upstream' means?

Comments