Share via

Customer Experience Program in Whidbey

  • Article

Hello World!

This is my first blog on MSDN - so I am going to take this opportunity to talk about the new Customer Experience Program in Whidbey.

Internally, the Customer Experience Program is known as SQM (Software Quality Metrics). It is basically an infrastructure that we use to log, transfer and report on how customers use Visual Studio. SQM is not proprietary to Visual Studio; in fact, many other product groups are also using SQM: Office, Media Player, MSN Messenger and Money, just to name a few.

Starting with the Beta of Whidbey, we will ask customers to participate in the Customer Experience Program. Participation, of course, is completely voluntary. There is absolutely no disadvantage to participating - it will not degrade performance, and the process is completely transparent to the users. Participating will allow Microsoft to understand how customers use our product and allow us to focus our resources on the things that matter the most to you! Here are some of things that we are trying to answer by using SQM:

  • What are the most frequently executed commands in Visual Studio? Are they executed through shortcut keys, menu, context menu, or the automation object?
    Why we want to know:     we want to create better default menu structures and shortcut keys for our customers. More commonly used commands should be placed higher up in the menu, and they deserve better shortcut keys.
  • Which tool windows get used the most?
    Why we want to know: we want to create the right set of tool windows that are visible by default, depending on your main development activity. You wouldn't believe the extent we go, just to create the default window layout.
  • How many projects are in most customers' solutions? How prevalent are cross-language solutions?
    Why we want to know:     we have no idea how big a real-world solution is (in terms of # of files) or whether cross-language solutions are prevalent or not. If we knew what the average solution size is, we run our perf tests using similar numbers. If cross-language solutions are common, we could perhaps focus on that scenario more and make the experience more seamless.

One very important point - all data logged is COMPLETELY ANONYMOUS!!

Some customers feel very paranoid about participating in this sort of programs. I will have to admit that before I joined Microsoft, I belonged to this camp as well. But after having worked on SQM hands on, I can assure you with 110% confidence that our entire company takes every possible measure to ensure that all information collected is not personally identifiable.

  1. We never log any strings with SQM - every log is just an ID and DWORD pair. Although this seems like a severe limitation, the SQM team has deliberately omitted this feature so that it is more difficult for product teams to accidentally collect non-anonymous information.
  2. Reporting is only done in an aggregate fashion - this means I cannot even look at user #43432 and see trends about how the user behavior has changed through time.

If you have studied statistics, the power is in the numbers. We do not care what a specific customer does - we never know whether that customer is a representative sample. What we want to know is the aggregate of customer usage data, so there is really not much benefit to looking at a single user.

If you can rid yourself of the paranoia, I would strongly encourage you to participate in this program. The more data we get from customers, the more representative the data is, and it will ultimately translate into a better product for you. Just think about how cool it would be to be able to help make Visual Studio better without even completing a form, a survey or anything at all!

When you get Whidbey, remember to participate in this program by going to Help menu and choosing “Customer Feedback Options...“!!

James Lau
Program Manager, VS Core

Comments