Reduce number of invalid defects -> Improve Test Productivity and Efficiency & Let your Developers be happy

Just the other day we were discussing that how can we (testing team) reduce number of invalid defects. I was in deep thinking that why is it really important to reduce  the number of invalid defects.

• • Isn't it a tester's fundamental job to log every potential defect and let it go through normal defect life cycle and let business and management take call if it is a valid defect or not?
  • Isn't it right that tester shouldn't assume that it is not a bug and then regret later because of  a false assumption made?
  • Isn't a tester taught to think negative and always be suspicious and uncover what is not seen by someone like dev?

 

The point here is what is a big fuss if testing team raises 'invalid' defects unknowingly. At least they don't leave anything to assumptions which is far more dangerous. There primary job is to find defects, whether that is a 'valid' defect or not is a secondary question.

One of the tester I am mentoring complained that his testing team had found 108 defects out of which 104  were valid and only 4 were invalid but his management didn't seem to appreciate them for the the number of valid defects found as they were expecting number of invalid defects to be zero.

My take on this:

Yes, it is important to reduce number of invalid defects.

Why?

-> Test Metrics gets screwed (Test Effectiveness or lets say Test Productivity goes down with no. of invalid defects)

            Test Effectiveness = No .of Valid defect  / (No. of Valid Defects + No of invalid Defects) X 100 %

            Example for above:  104  / (104 + 4) X 100 % = ~96.6 %

-> Time lost in tracking and logging invalid bug

When you raise a bug in your reporting tool like Test Director, it has to go through complete bug lifecycle. Say you raised a bug spending effort in recording it and then it turned out to be invalid, you developer rejects it.  Finally you have to close it.

-> Management don't like invalid bugs

You bet me that "invalid" bugs doesn't please any manager. It is a human behavior to criticize something that is not right.  It sets them off.

-> Developers stop taking you seriously

When they observe that you raise many invalid bugs, they start expecting that every time. They stop paying due attention to valid bugs considering them to be invalid. Quality over Quantity concept.

-> Time lost in triage meeting to discuss invalid bugs

When you log invalid bug, it not only your time getting lost, developers waste their time reading them, then testers and developers waste time arguing on that as it has been officially logged, and most importantly business waste their time in triage meeting to take call.

-> Spoiling terms with development team

Developers are under pressure to reduce number of defect found in their code by the test team. Now if you log it, they go defensive and try their best to prove your bug an invalid bug so that it doesn't spoil their commitments.

 

Now that we have a problem. Let me propose something which we successfully implemented

 

 

Now with this process, every bugs gets verified online by the Development team even before we officially log it. They update the sheet saying that they are okay with so and so bugs and we log only those bugs in the Bug Management tool  and hence all "VALID" bugs.

Bugs which they update as "INVALID" or "REJECTED", we update the SHARED SHEET with more information like repro steps etc and they change the status in the sheet accordingly. Now if it was our fault and it was actually INVALID defect, we update the SHEET and close it there itself rather than logging it in bug management tool and going through the entire process.

Now our metrics always say  100 % valid bugs. We don't miss any bugs because we record it anyway in the Shared Sheet and triage it with development team online. Development team feels good as they get a chance to to repro bug and confirm before it is actually logged against them.  We don't have to waste time in the triage meetings discussing whether its a bug or not. Now business only take call on functional bugs which are more important to end user.