My Thoughts on the MSDN Feedback Center

Overall I know it’s a great thing and there are some issues that need to be improved over time for it to be truly successful. I may be a little close to the project since my co-workers are responsible for the Feedback Center, but I think I’m still allowed to have opinions and air them in this space. :-) Recently I was asked some specific questions that do a good job framing my opinions on this project.

Do you like this kind of transparency?
I love this transparency and think it’s only the first step towards a more transparent future of developer products at Microsoft. Why do I like it so much?

In case it’s not clear… when you enter a bug through the feedback center it is also opened directly in our internal bug tracking solution alongside of bugs being found by our own testers. Internally we can’t escape these bugs since they are in the database we scrutinize every day on our way to shipping the product. You get to see when, how, and who fixes your reported issues and you get notified of its progress along the way. We also require that team members provide quality responses to every reported issue to further encourage accountability and close the feedback loop.

Historically we’ve had crash reports and statistics that helped us know which crashes we should focus on based on the number of reports. Triaging non-crashing issues has always led to debates over how we think customers might feel about an issue versus what rating and how many customers might actually be affected by an issue. This will eventually help us triage bugs much more efficiently and gain an even better customer understanding of the impact of bugs we find internally.

We will only get more transparent. Soon you may be able to see our actual internal bug counts and even see internal bugs alongside customer reported issues. When we are in a world where you might have access to any build of visual studio this will be even more important.

How does this compare with the closed, semi-private, bug reporting on beta.microsoft.com ?
I’ve used both and the MSDN feedback center is much better simply because you can search, see, and vote on bugs reported by other users. You also get to add your comments and workarounds to issues as well. With betaplace (beta.microsoft.com) you only get to see issues that you report.

Microsoft products will still have private beta’s for a long time so there will always be a need for a non-public site, but there are teams talking about creating closed group instances of the Ladybug application moving forward because it is seen as a step up from existing solutions. So you can imagine that beta.microsoft.com could eventually be leveraging the Ladybug solution.

What do you think about the quality of submitted bugs?
It could be improved. I’ve spent a lot of time in my career here triaging bugs as a Test Lead and I’ve also been involved in triaging the bugs from the Feedback Center. In general feedback center bugs require more “translation” time to understand what customers are trying to report and what they may have been trying to do when they encountered the bug. Some teams have suggested that the triage time per bug is up to 10 times as much when compared to bugs reported internally. I’m not sure I buy that number (given the statistics you’ll see below), but it does take more time. The extra time required is probably because of the following reasons:

• Lack of Additional Background Info: There is a set of information that each team feels should be collected when a bug is logged on their feature. When this information is missing the issue requires an extra round trip with the customer to attain it. We would rather not burden every customer with these specific requests, since we want the bug reporting to remain easy, but we may eventually run an active X control that automatically fills out portions of a longer form for users when they attempt to file a specific bug. This information would include what profile settings you chose at first launch, what other versions of VS you might have installed, what type of project you were working on, etc.
• Lack of Clear Repro Steps: We should provide you guys with better bug report examples. I’ve seen a bugs that look like “Sometimes I seem to get an error message that says Foo” without explaining what they were doing that led to the error message. The feedback center is a community in its infancy and the unspoken rules/expectations of a community have not been fully explored yet. Over time, with better examples and additional practice submitting bugs it will improve. The quality here is not unlike the quality of bugs you might get from a new-hire testing the product and, like the new-hire, I’m sure it will improve over time with a little guidance.
• Lack of Expected Behavior Reports: Good bugs should not only contain the description of the behavior experienced, but also the expected behavior from your perspective. Instead of just saying “When I click X, FOO occurs” it can be helpful to say “When I click X, FOO occurs, but I was expecting BAR because of Y”. I also believe this will get better over time.
• Lack of Good Microsoft Responses: Yes, this something we can improve on that will help quality. A running joke on our test team was that we should have all been filing our bugs through beta-place or Ladybug as customers because we’d get better responses about why issues are “Won’t Fix” or “By Design”. A tester internally will often just see these two word responses and it would be up to the tester to push for a better explanation if they felt the bug was important. Strangely enough we felt we should encourage people to treat the community members with a little more dignity than our own employees.
  • Providing clear explanations about bug resolutions is something that is new for a lot of people. Speaking as a former tester I’ll say that it is about time. Currently we’ve had some rough spots and bits of information that are Lost in Translation. Eventually we will adapt by improving our accountability, learning better “customer-speak”, and finding more common ground within this community. This is new for us too. We are investigating providing a direct “Contact the Developer” form for MSDN Feedback center bugs so you don’t get the rather blind “Resolved by Microsoft” text. You would then see who resolved it any be able to contact them directly.

What do the results look like so far?
So we’re listening and guaranteeing a response. IMO this is only part of the solution. In the end we’ll also be judged by what we do with what we hear. It’s unfortunate that this feedback mechanism was opened so late in the cycle of Visual Studio 2005 because a lot of this great feedback will only start to get leveraged for the next version. This is especially true for suggestions that would require more than a trivial amount of new development work. In the mean time I’ll share some raw statistics since late June when the feedback center became public that I took a snapshot of today.

For comparison I thought it might be interesting to look how these percentages compare to internal bugs and suggestions opened during the same time period. For these, unfortunately, I don’t have a good measure of time to response so the raw % will have to do. Also, just assume the total numbers for internally reported issues are larger. :-)

MSDN Feedback Center Bug Stats

• Total # Opened So Far 1679
• Responded to 75%
• Avg # of Days to First Response 6 Days
• Still Active 50%
• Resolved Fixed 24%
• Resolved No Repro 9%
• Resolved By Design 7%
• Resolved Postponed 6%
• Resolved Won’t Fix 4%

Internal Bug %’s for the same time period

• Still Active 55%
• Resolved Fixed 20%
• Resolved By Design 8%
• Resolved Won’t Fix 5%
• Resolved No Repro 5%
• Resolved Duplicate 6%*
• Resolved Postponed 1%

MSDN Feedback Center Suggestion Stats

• Total # Opened So Far 1174
• Responded to 79%
• Avg # of Days to First Response 6 Days
• Still Active 34%
• Resolved Postponed 27%
• Resolved Fixed 16%
• Resolved Won’t Fix 12%
• Resolved By Design 9%
• Resolved No Repro 2%

Internal Suggestion %’s for the same time period

• Still Active 52%
• Resolved Won’t Fix 17%
• Resolved Fixed 11%
• Resolved Postponed 10%
• Resolved By Design 5%
• Resolved Duplicate 5%*
• Resolved No Repro <1%

*If an MSDN Feedback bug is considered a duplicate it is associated with the primary internal bug so customer bugs can only be duplicates of other customer bugs. That number has been very small since customers can “+1” an existing bug rather than opening new entries. So the % duplicate of MSDN Feedback bugs was not worth reporting.

What’s important to note is that the jury is still out on around half of these issues that are still marked as active. Because of this the resolution percentages are very likely to change over time.

What I learned through this is that, so far, we’ve actually fixed a higher percentage of customer reported issues than internally reported ones! It will be interesting to see if this keeps up over time.

Now that you’ve read this far I’d like to know some things:

1. Are these numbers interesting to you guys?
2. Would you like to see more?
3. If so, what other information would you like to see?
4. How would you like to see top contributors to the Feedback Center rewarded by Microsoft?

Comments

• Anonymous
  July 28, 2004
  The comment has been removed
• Anonymous
  July 28, 2004
  The numbers themselves were ok; I think your analysis was more important. I'd like seeing more of this sort of thinking aloud. Ladybug is a good move, and it's interesting to hear how it affects the way you work.
  
  I don't think you should start systematically rewarding people at least yet; that leads to unhealthly competition on a field where the "top" is not very easily measurable (quality vs. quantity etc.). If somebody is an excellent reporter, invite him to a job interview - rumors will spread and people will consider that rewarding, but not in a way that would lead to competition ;-)
• Anonymous
  July 28, 2004
  The comment has been removed
• Anonymous
  July 28, 2004
  >when you enter a bug through the feedback center it is also opened directly in our internal bug tracking solution
  
  This is not as good as it sounds. This way more pressure will come to close these bugs as soon as possible to get them off the internal bug tracking solution (which is in most companies primary benchmark of the state the product is in).
  A bit relaxed attitude could be better, and then devs would not rush to close so many items without proper explanation.
  But I see you have good ideas about how to cure this - make sure every tester/dev goes through at least one bug using ONLY ladybug. Then hopefully we will get more meaningful explanations on closed bugs.
  
  That said, you are going in the right direction - please, keep up the good work.
• Anonymous
  July 28, 2004
  You could start publishing the email address of the employee who resolved a bug or suggestion (at least to the person who submitted it). That way, the customer could continue the conversation, as internal testers can.
  
  I'd be interested to know the browsers people are using to access the feedback center (keeping in mind that you can't submit a bug in Firefox).
  
  Also, things like number of bugs in each category, average and distribution of importance of bugs.
• Anonymous
  July 28, 2004
  I don't know what the reward for top contributors should be but I have a pretty good idea how it should be given: without fanfare. Quietly mail them some thank-you gift and be done with it. Keep it unofficial.
  This would prevent useless competition and attempts at "gaming" the system.
  
• Anonymous
  July 29, 2004
  Yes, this information is very interesting. As for what information, I say just keep posting whatever thoughts you have... don't make it too formal.
  
  How about a free version of the MS software package for the most helpful contributors, especially the next version with the bugs fixed. This would inspire better bug reporting, especially since the reward would be directly related to the effort. ("Fake" rewards would only cheapen the feedback.)
• Anonymous
  July 29, 2004
  The comment has been removed
• Anonymous
  July 29, 2004
  Jouni: WRT job interviews. I’ve actually been wanting to see this happen for a while. In a world where so much of our process is being opened up, why not hire the people who work with it the best. I’m not sure that the hiring managers are all thinking the same way, but it won’t be long I’m sure.
  
  I hear what you are saying about rewards. It’s a common debate internally as well about how to reward testers for their bug numbers. We do, however, want to do something to acknowledge that the community is doing great work here (even if we don’t hire them all).
  
  
• Anonymous
  July 29, 2004
  The comment has been removed
• Anonymous
  July 29, 2004
  The comment has been removed
• Anonymous
  July 29, 2004
  David: E-mail publishing is under consideration. Thanks for your feedback on things you’d like to see.
  
  Stephane: I also think some “offline” thank-you system is a good idea. Everyone like being rewarded differently.
  
  Cade: Thanks for your feedback. Software products are interesting, but a good number of customers already have MSDN Universal so the benefit of getting another copy of VS would go down for those people. It could be an option for those that don’t though.
  
• Anonymous
  July 29, 2004
  I have been quite impressed by this new bug tracking system (I would like to see some of the products I work on go down a similar route). I added my feature request (http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedbackid=78fbe39f-954d-494f-b35c-9a7cad5dd245) which was looked at soon after I reported it and although it was marked as 'Won't Fix' I am glad someone has taken the time to respond (even if I don’t entirely agree with their reply I am sure they are right).
• Anonymous
  July 29, 2004
  Jonathan: Thanks for your additional comments and good example.
• Anonymous
  July 29, 2004
  josh:
  Duplicates vs. resolving all bugs with Fixed. Ohh .. I see.. I've assumed this in my comments.
  
  Can we get real numbers ?
  What is average number of reports per unique master bug in old semi-private beta.microsoft.com reporting site ?
  
  BTW, On WindowsBeta there was additional information for bug reports resolved as duplicate. Each bug report additionaly listed master bug resolution. Thus instead of 100 duplicates resolved as Fixed - you will have 1 master bug report as Fixes and 99 reports resolved as Duplicate with ability to see master report status.
  
  Just a FYI, I've participated in initial test-drive. I've found ProductFeedback not usable and proposed a few improvements. But unfortunaly they was not all implemented before June public anouncement.
  
  I believe that ProductFeedback is on long way to collecting productive feedback. I've not expected such a little and feature-less site created by Microsoft for world-wide use.
• Anonymous
  July 29, 2004
  Not sure what you want mean by "real" numbers. A customer bug is a "valid/real" bug regardless of whether it already existed in our database because of the build latency. If you mean, how many of these we actually end up re-directing to other internal bugs... it is probably around 12-15%.
  
  Regarding the windowsBeta implementation. I think this is an additional level of complexity we don't want to add in the public space. It's better to just re-associate the bug for the customer so they don't have to worry about anything else.
  
  Regarding your feelings of the site being unusable... Without specifics i can't explain the trade-offs, but I know there are a LOT of people who think this is a HUGE improvement compared to never hearing back from your feedback report.
  
• Anonymous
  June 16, 2009
  PingBack from http://fixmycrediteasily.info/story.php?id=16443