Share via

Bug Reporting Best Practices

  • Article

Since the MSDN Product Feedback Center was launched publicly at the end of June with the Visual Studio 2005 Beta 1 release, we’ve received a lot of great input from customers. So far we’ve fixed 24% of the bugs filed. Here’s some insight into how you can make the most impact through the bugs you report.

Which Bugs Get fixed?

To make the most impact on the product you want to spend the most time on the issues that are most likely to be fixed. How does Microsoft decide which bugs get fixed and which don’t?

Bugs that get fixed:

  • Affect many customers
  • Are embarrassing or do not convey professionalism. Ex: fit and finish bugs
  • Are easy to fix

Bugs that less likely to be fixed:

It would be great to fix every bug in the product, but it’s also great to ship J. In prioritizing which issues to fix, here are the some factors that cause certain bugs to miss the triage bar. Bugs that are least likely to be addressed:

  • Are not reproducible or very hard to reproduce. This may be because the problem occurs intermittently or there are not enough details in the bug report
  • Have strange and complex steps to introduce failure
  • Have no perceived customer impact
  • Are edge cases
  • Risk introducing greater instability through the bug fix then the bug itself causes. Especially late in the product cycle, the bar for these types of bugs is very high because the bugs that may be introduced by the fix we will likely not have time to fix.

Found a bug? Dig deeper…

To create the highest quality bug report which will save developers time and increase the likelihood the bug is fixed, a little extra work goes a long way:

  • Try to find a more general repro
  • Are all of the steps required?
  • Is the bug tied to a single setting or is there a small range of cases? Identify the range.
  • Is the bug more serious than it appears?
  • What other user actions cause the error to occur?
  • Does the error occur with different settings or options selected?
  • Look for follow-up or related errors

How to file a good bug

Here are some tips on how to create a great bug report that will enable Microsoft to action on it quickly. Often bug reports submitted via the PFC lack key peices of information so triage teams send bugs back to customers to request more information. We don’t want to mistakenly resolve and issue as By Design, Won’t Fixed or Postponed, so providing a complete bug report helps ensure the best response from Microsoft. We want to spend as much time as possible fixing customer bugs – to that end, high quality bug reports help us spend less time deciphering the issue and more time resolving it.

To file a great bug report:

  • Use clear and minimal repro steps
  • Include only one issue per report. If you have multiple issues together, file separate bugs, even if they are related. This allows us to deal with them separate. Even if you encountered them in the same action, they need to be treated seperately internally.
  • If possible, try variations to see if you can better nail down the issue. Do any of the Tools Options settings affect what happens with the bug? Example: internal vs. external help.
  • Example: In the Help system, help can be internal or external. Does it repro with both?
  • Avoid jargon or terms that may be difficult to understand.
  • Make sure other customers will easily understand the bug

Bug Titles

Bug titles should completely and succinctly describe the issue. Internally, we spend a lot of time querying the bug database and looking at lists of bug titles. It saves time when we don’t need to open the whole bug report to know what the problem is. On the PFC site, when other users are searching for bugs, they too don’t want to open every link.

Here are some tips on writing bug titles:

  • Fully capture the essence of the bug. We see a lot of partial titles that require more information. Example: “Autosave function” is not a good enough title. What about autosave? When the bug comes in, we put it in a bucket. The title should provide enough information to differentiate it from other bugs in the same bucket.
  • Think about it from the perspective of other customers trying to find your bug.
  • You can start by considering the keywords that you used to search for dupes before filing your bug.
  • Include the specific name of the components involved in your bug in the title. Toolwindow, dialog, designerExamples of Bug Titles
  • Most of the improved titles are longer, but not overly long. Don’t include unnecessary commentary either.

Original Title

Improved Title

“Search selected text”

“Find and Replace dialog should default to last-searched term”

“Control Tab Files”

“Had to press Ctrl-Tab twice for toolwindow selection dialog to raise”

“Change filename in Solution Explorer”

“Changing case of a filename in Solution Explorer doesn’t take effect”

Repro Steps

Steps to reproduct the bug are the most import part of a bug. Investing a little extra time in this area helps developers immensely. When creating repro steps:

  • Always try to recreate the bug before writing the report
  • Walk through the steps of your bug and make sure you didn’t leave out a step.
  • Could someone reproduce the issue without familiarity with the feature?
  • Start your repro steps from empty IDE. Don’t assume that a project is open, or a dialog is open. Don’t assume that the title has already been read when you write the repro steps. Often times, if a bug occurs on a dialog or an options pane, the reader of the bug won’t know how to get there.

Repro Steps Example

  1. Choose Help->Search
  2. Search for "integer"
  3. Right click on a result and uncheck "Show Abstract"
  4. Scroll down to the bottom of the list

Result: White space appears, because the scroll region didn't recalculate after hiding abstracts

Expected: After hiding abstracts, scroll region adjusts to match the search result display region

Descriptions

Descriptions should provide all the other useful information about and issue that is not captured in the repro steps some of which may include:

  • Does the bug repro on other hardware or software configurations
  • Are there configuration dependencies? Install options, Tools Options settings, etc.
  • Is the bug new in this version?
  • For Visual Studio specific issues:
    • Does the bug still repro even after changing to a different vssettings file through import export settings wizard?
    • What project type and language are you using?

Attachments

Attachments are extremely helpful, especially for fit and finish UI bugs.

  • A picture is worth 1000 words. Screenshots with comments are priceless.
  • A picture doesn’t replace repro steps. The bug should still be captured in text even if a screenshot is attached.
  • The actual files, project, or solution you are working with helps developers repro the issue. Keep in mind any code you attach is made public via Ladybug.

Workarounds

Sharing workarounds is one of the best features in the PFC. It’s great to see the developer community members help each other be more productive with Visual Studio!

Marie Hagman

Visual Studio

Program Manager

Comments