Power Toys and code coverage

One of the Aftermarket Developer Solutions team goals is to create communities around shared source tools for VS developers. Of course, we are a small team and can't provide full featured tools and support them for long periods of time.That's where the community comes in. Feature enhancements and even long term support can come from the community. But, what we do produce should be useable and not prone to crashes.

The best way to insure that our tools are useable and don't cause too many problems is to execute all or "nearly all" of the code and test the Power Toy against all the functional requirements identified in its specification.

Testing all the requirements is critical to insure that the “average user” has a good experience with our Power Toy. These tests are usually designed solely on information that is in the specification for the Power Toy. The test plan will define user level scenario tests which verify all functional requirements of the Power Toy.  They are called “black box” tests because the person designing them does not read the code while designing the tests.

But, these tests alone are usually not enough to test "nearly all" the code. So, we decided to use code coverage tools to tell us what else needs to be tested to insure that we execute nearly all the code.

In his books and debugging classes, John Robbins claims that 85% to 90% code coverage is required to produce production level software quality. For mission critical software such as medical devices and space missions, he says that nothing less than 100% should be accptable.

My own experience suggests that “average” unit tests hit no more than 50% of the code. However, it is fairly easy to improve that to 65% - 75% code coverage by testing requirements using a “black box” approach. Getting to 85% or above often requires that additional tests be designed by looking at the code and figuring out how to make the code execute an untested path (i.e. “white box” testing).

Our goal is to execute between 80% and 90% of all code blocks. I selected the 80% level as a minimum. This should achieve a “nearly production” level of quality without being excessive for a team our size. It should work well for us.

So, under a code coverage tool, we will execute the user level scenario tests and any unit tests provided with the software. The results will tell us which paths through the code have not been tested and suggest additional tests. The new tests will be added to the test plan and the test suite prior to the initial shared source release of the Power Toy. 

Until we reach at least 80% code coverage, the Power Toy just hasn't been tested enough to be ready for release.

Comments

• Anonymous
  August 21, 2006
   John is the lone SDET on the developer solutions team. He's fairly outnumbered by his developer...
• Anonymous
  August 23, 2006
  You talked about low unit test coverage metrics in your software.  Have
  you folks discussed moving to a test-driven design/development approach?
  If so, I'd be interested in hearing the decision points that moved you
  away from TDD adoption.
• Anonymous
  August 23, 2006
  Perhaps, I could have been clearer in my original post.
  
  As I clearly said in my biog, most of my experience has not been at Microsoft. I have only been here 3 and a half years. And, most of my career has been in development so I know unit tests from both sides of the dev/test fence.
  
  When I said that my experience shows that typical unit tests cover no more than 50% of the code, I meant throughout the industry. In my opinion, Microsoft does a better job of testing its products than most places I have worked. But, the tendency here is to rely on the test team to do the testing. Companies doing military contracts or medical devices often do exceptionally good testing with most of it being done by the developers.
  
  On our team, I am the only tester so we work a little differently. Typically, the developers write the unit tests they think are important, I design the rest of the test suite and the devleopers do most of the work. Then I do a code coverage analysis run of the tests to see what we missed and we fill in the gaps.
  
  We haven’t decided against using TDD! In fact, I have been encouraging it on our team.
  
  One of the code review criteria that I recently added is that developers must include unit tests for any new code (e.g. classes, methods, or bug fixes) that is to be reviewed. Developing unit tests at the same time you write the code is certainly the easiest time to do them. The code is fresh in your mind and you won’t have to spend a week writing unit tests at the end of a month long development cycle!
  
  In the criteria, I specifically recommended that the developers try test driven development. Since I’m not the manager of the development team, I can only make suggestions and recommendations about how they do their work. I can and do put requirements on what they must produce to meet the team goals but I can’t dictate how they get it done. In other words, I can require that they produce unit tests for their code but I can’t require that they write the unit tests first. It’s as simple as that.
  
• Anonymous
  February 09, 2007
  The comment has been removed
• Anonymous
  February 12, 2007
  Your idea of adding  failure injection and logging is a great start! The only thing I would suggest is that you also consider your code "mission critical" and aim for a much higher level of code coverage than we do with our power toys.  If I were in your place, I'd be aiming for at least 95% coverage and trying for 100%. Since you're adding failure conditions, you should be able to get over 90% pretty easily.  Once you have your failure conditions tested, the tricky parts of getting to 100% code coverage are often the if statements that have multilpe conditions, switch statements, and your code that handles error conditions from system calls (etc). If your tests are automated, you can add cobminations of options or parameters necessary to get the corner cases. If that's not enough or you don't have automated tests, run your code in a debugger. Then, you can set breakpoints in methods where you need better coverage. When you hit the breakpoints, change variables that will force your code to take the uncovered path(s). You can also use the debugger for executing code that handles error returns from system calls. Just set a breakpoint on the call; step the debugger over the call and then force the debugger to continue in your error handling code. The bad part about doing the hardest-to-test  testing manually is that you may have to spend time doing it manually every time you release an updated version.