Dela via

97%

  • Artikel

Metrics are very important to the success of an engineering team.  They help with data-driven decisions.  Today let's focus on QA metrics, although having metrics for all the important indicators we use to ship products is essential to any engineering team.  For QA metrics, some of the most important ones are around # of test cases, % automated, % run, % passing, and % code coverage.  But recently there’s been a big push for traceability.  In Visual Studio’s TFS, we have work items for many different artifacts in the product lifecycle like requirements, test cases, and bugs.  So my team started using metrics to track items like how many test cases do we have per requirement, how many bugs have we found per test case, etc.  Now obviously you just can't have these individual items in TFS, you actually need to make sure to link them together.  Sometimes making sure everyone in the team is consistent in how they enter their data (like links between test cases and bugs) is the hardest part of getting an accurate metric.  So when the first report rolled out and one team showed that they had 97% of their test cases associated with bugs, we commended them for their efforts.  That means they went through all their test cases, even legacy ones, and associated them to bugs found.  This was a big accomplishment, or was it?  Do you see a problem with this?

At first, we honestly didn't.  But it only took a few days and some different perspectives by some incredibly smart people on my team to ask the question, is having 97% of our test cases associated to bugs a good thing really?  What does that metric really mean?  It means we find a ton of bugs while doing our test passes.  Again, is that a good thing?  Well, not really.  Our job as quality assurance engineers is not to find all the bugs but to help stop them from occurring to begin with.  Our test cases shouldn't be written to prove there are bugs in our code.  They should be written to prove there are no bugs in our code.  So having bugs associated with 97% of our test cases in one of our projects means there are way too many bugs in our code and we need to push quality upstream. 

Granted, if we were to say that 97% of our test cases were reviewed to see if there were bugs associated with them, then we would be measuring amount of work and progress towards overall traceability.  This is what we wanted from this metric, but that's not what it tells us.  It tells us that we don't do enough testing upfront in the product cycle and that we are using test cases to catch bugs and not to verify that none exist.  Guess what?  We have a lot more work to do here.  And fortunately, by having metrics and critiquing them appropriately, we have exposed places where we need to improve as a QA organization and as a product team.

What metrics does your team collect?  Are they the right ones?  Watch out!...and think twice about what behavior you are trying to drive!

Comments

  • Anonymous
    April 03, 2012
    I thought it should be: all our bugs are linked to test cases, not the other way...
  • Anonymous
    April 05, 2012
    Yes, getting mearurements of how many bugs are linked to test cases would be a better metric that how many test cases are linked to bugs.  That's what makes testing so interesting - subtle differences like this are very significant.
  • Anonymous
    April 09, 2012
    "Our test cases shouldn't be written to prove there are bugs in our code.  They should be written to prove there are no bugs in our code."Some questions.Can we really ever prove that there are no bugs in our code? Is the purpose of a test case really only to prove that there are no bugs in our code? Doesn't the purpose of a test case change depending on in what context it is executed?
  • Anonymous
    April 11, 2012
    @Johan - yes definitely.  Some test cases exist to find bugs and some exist to prove no bugs exist.  My statement was more intended to imply that if you are pushing quality upstream and bugs are found early in the design and development phase, your test cases really should focus on proving no bugs exist.  By approaching testing with the philosophy of proving no bugs exist, you help drive more of a culture of quality throughout the product lifecycle instead of product quality after the test/QA team has a chance to look at the features.