Why we want Watson reports

One of the features we added to the mid-tier web services was the ability to create and file reports when error and/or unexpected conditions occur. When these happen, a information about the condition is gathered and then a report is filed with the Watson reporting mechanism.

The next time an administrator logs in to the mid-tier machine, the Watson reporting UI should appear asking if you want to send the queued reports. Please consider doing so -- we really need this information to see what kind of real-world problems are occurring. We include information about the error condition and stack trace of the thread active when the condition occurred in the report.

You might wonder what happens to the Watson reports that are sent to Microsoft. Each week, the Team Foundation Watson Review team (a representative from the test organization and myself) review the Watson report that arrived in the last week. We focus on the newly arriving Watson reports and the reports that have been reported the most. We used to investigate only the top 10 reports (as measured by our internal Watson tool) but have now extended our focus to the top 60 or so (we look at any report that has been reported more than a few times).

We look at reports from Beta 2 deployments and some other key deployments or test beds that are internal to Microsoft.

After we investigate the set of Watson reports, we send a summarization that includes the Watson report, status, the bug (if any -- many problems have already been fixed) details, and the developer responsible for fixing it. This summarization gets wide visibility by design -- we even send it to our PUM's boss. I'm happy  to report that we've been making great progress in fixing issues illustrated by the Watson process.

One more thing -- and I can't stress this enough -- we really do look at the Watson reports that we receive and make every attempt to correct the issue(s) that are contained in the report. Please continue to allow the reports to be sent.