A Bug's Life

No, not the Pixar movie. This is about the lifetime of a bug in our product. How they go from discovery, to fix, to maintainance.

Stage 1: Discovery. Perhaps what's most interesting are the various ways a bug can be discovered. We often find them through test suites, where a build of the compiler is run on a test suite by means of automation. The front-end compiler test suites are mostly a mix of specific feature tests, regression tests (more on that later), and third-party conformance tests. (For a lot more on how we test, you might check out some posts from FiveTesters.) Sometimes, a bug is found by a customer - these usually come to us via PSS, through newsgroup postings, or as beta feedback. Generally, simply the fact that a bug was discovered by a customer affects its impact.

Stage 2: Determination. Is it a bug? Just because you see odd behavior doesn't always indicate a bug. Determining if something is a bug usually means digging through the ISO C++ or C standards, or through other standards documentation. (Except for the obvious bugs, like an ICE or a rare compiler crash.) I like to get the confirmation of at least one other person, too, when it isn't completely obvious.

Stage 3: Raiding. I've mentioned before why we call it “Raiding” a bug. When you file a bug in our bug system, a lot of other pertinent information is included. Usually, you include the source code that reproduces the bug, along with a note to the developer (or other people looking at the bug) that justifies the bug, and explains why it is a bug. We also try to use very descriptive titles, so that when other people are searching for bugs, they have a high probability of finding the right one. When bugs are raided, their status is “Active.“

You give the bug a Severity and Priority - usually these two are quite similar. They range from 1 (most severe) to 4 (typo in the diagnostic?). There's a zero priority, in Microsoft lingo, we call that “Pri Zero.” Pri 0 bugs have an expected turnaround time of 24 hours. Like most Microsoft-isms, I hear people using them in other contexts. My former officemate referred to his son as “Pri 0.“

Stage 4: Triage. Representatives from the dev, QA, and product management team meet about twice a week, and go through Triage, where they inspect all newly-raided bugs and give them a quick once-over. They try to verify that it is a bug, that it isn't already raided, and that the priority and severity are correct. They then assign it over to a developer to investigate and fix. (Occasionally, the bug is kicked back to the QA person, either because it is duplicate of another bug or because it isn't a bug at all.)

Stage 5: Fixing. The developer performs his or her magic, and fixes the bug, linking in the pertinent information, and a link to the source diffs that represent their fix. They then “resolve“ the bug. A bug can have a number of resolutions, some of the most common are Fixed (the developer fixed the bug), Not Repro (the developer couldn't reproduce the behavior), and By Design (that behavior is not a bug). The Resolved bug usually ends up assigned back to the QA who opened the bug.

Stage 6: Verification. The QA who opened the bug gets a build of the compiler containing the fix, and verifies that the bug is fixed properly. If they don't feel the bug is fixed, or don't agree with the resolution of the developer, they “push back.” This is another Microsoft-ism (perhaps I should create a glossary...) - but at least this one is pretty self evident. Usually, this just means they re-assign the bug back to the developer with justification. Sometimes, they just go down the hall, knock on the developer's door, and have a discussion.

The QA person is also responsible for ensuring that this scenario is covered by our test suites. If there's no test to cover it, they check one in, and mark the location of the test case in the bug. We typically call these “regress tests.“ They're just there to make sure the bug gets fixed, and, if it somehow gets un-fixed, we catch it. When this is complete, the QA closes the bug.

Stage 7: Maintainance. Closed bugs don't disappear from our bug database. If a bug ever regresses, we can usually find the bug in our database, and reactivate it.

So, there's my quick explanation of a bug's life cycle.