Code Inspection for SDETs, part I

(Note for those unfamiliar with the acronym, SDET stands for Software Design Engineer in Test.  In simple terms an SDET is an engineer who develops code to test software)

Code inspection has historically been a “high-yield for low cost” bug-prevention or bug-detection approach.  This is especially true when used as part of formal code review process and integrated into the software development lifecycle.  In the multiple teams and companies I have worked at it has been very common for the developers to practice a peer code review system before every check-in, and periodically review entire features post code-complete.  However, it has not been common for the test discipline to be fully plugged into the code inspection pipeline, which is less than ideal.

Here are some examples of the benefits realized with the test discipline participating in the code inspection of the code that they test.  Let’s walk through a few of them. 

Motivation

1)      Knowledge Growth:  Code inspection is an excellent way for testers to become more familiar with the code base that they own from a quality perspective.  As we move into a world which requires more technical horsepower and understanding from SDETs, there is really no substitute for having deep code knowledge of the components that they own.  In addition, it is a great way for  SDETs to uncover test holes and lesser-known functionality in their components.  A strategy to grow SDETs into senior roles has to include some level of in-depth code knowledge improvement, and code inspection is a great way to attain that knowledge 

One common concern about testers using code inspection techniques is that the tests that they author will somehow become extremely reliant on implementation details and become fragile and brittle when confronted with product code implementation changes.  This is a valid concern and is something to consider.  Testers should be careful that they do not rely too much on underlying implementation when designing automation for regression testing for example.  On the other hand, knowing current interdependencies and contracts between portions of the product can teach SDETs about what portions of the code may warrant additional testing attention

2)      Efficient Defect Prevention:   Root Cause Analysis of various defects has shown that many high priority bugs could have been effectively found and/or prevented through code review or static analysis.  In some cases, the best test tool for the job is going through the code itself.  Certain classes of bugs can be found quite easily through inspection, for example: 

- Cleanup and logic errors in failure conditions

Misuse of certain APIs

• Broken API contracts
• Some types of security issues
• Stylistic coding issues which have long term time drain in terms
  of maintainability
• Etc.

3)      Time Savings:   A wonderful side effect of having a low cost bug finding approach is that it frees SDETs up to do other more time-consuming testing.  After all, we are not being paid to be test execution machines.  When we find certain classes of bugs more quickly and with less effort, we can move on to tracking down the really difficult bugs through other sophisticated techniques 

4)      Constant Feedback Loop:   After the code inspection process has been followed for some time on a codebase, certain types of patterns emerge from the various reviews.  Many of these defect patterns could be avoided by simply evangelizing how particular defects could be avoided at the code level.   As part of the process, these defect patterns can be communicated to the development team as part of coding guidelines which will also prevent future occurrences.  Participating in this feedback loop is empowering and provides great value 

5)      Eliminating Classes of Defects via Tools:   Concentrated effort at code inspection pushes quality upstream and can help multiple teams, division, or company-wide.  Working with static analysis tool teams to help identify certain classes of defects and being able to feed that logic into the tools themselves can have a huge impact downstream by preventing the bugs from ever seeing the light of day in the first place 

6)      Beginner Impact:   Even new code reviewers can make statistically significant impact.  Some engineers shy away from doing code inspection because they feel that they need to be a seasoned expert in order to get anything out of it.  This couldn’t be further from the truth – studies have shown that although experts at code review do find more bugs than novices, the delta between these two groups is not large and should not discourage folks from participating 

Goals 

While building our SDET Code Inspection virtual team in my group, we wanted to avoid having a single-minded emphasis on finding defects (although we do encounter numbers of defects).  We wanted to also think about the education aspect and the defect prevention aspect as well.  Along those lines we concentrate on the following goals: 

1)      Find bugs earlier in the development cycle and push quality upstream 

2)      Increase SDET awareness of the code that is being tested so that more educated decisions can be made with respect to test coverage, test investments, test prioritization, etc. 

3)      Lower the bar of entry into code inspection.  Learn techniques and approaches from others 

4)      Provide general code quality feedback per area and feature in order to raise the quality bar

After every milestone I review our progress towards these goals and tweak the process if needed.

 

Upcoming blog entries will cover the exact process that we follow in my code inspection team and some examples of the types of defects that can be found through code inspection techniques.  Thanks for reading! 

-Liam Price, Microsoft

Comments

• Anonymous
  October 10, 2011
  The comment has been removed

• Anonymous
  October 10, 2011
  @Jeff, we use both styles of code review.  For pre-check-in reviews, we tend to use the collaboration tools.  In a future post I will lay out what has worked for us in the "meeting" style of review (for post-check-in reviews), and hopefully it can be of some use to you.  Please note that one of the goals of our reviews is to provide an educational platform which can be used downstream as well as uncovering defects.  Thanks for the question! Regards, Liam

• Anonymous
  July 13, 2012
  At <a href="www.inspection.com/.../a>, our vision is to become the leading market for un-biased home inspection leads in the world. To this end, we need to quickly and promptly help consumers make decisions about choosing an inspector, and ensuring that all inspectors on our network put their best foot forward. This includes helping inspectors with their web and marketing presence, while ensuring that those inspectors that are qualified and provide good service, rise to the top.