Usability Redux

Answering some of the questions asked about
our recent usability efforts...

Q: Can you tell us how many usability engineers are involved in all those
activities?

A: Yes.

Q: How do you like the eye-tracking stuff?

A: It's certainly interesting, and no doubt the "heat maps" look impressive. 
What we're going to be focusing on more than anything in the next two eye
tracking studies are the effects of the new visual look of the product.  As
I've
mentioned before, the current visual look that you're seeing is just a
temporary skin.  Yet, it's in the product and it's what has formed the
baseline of our usability results thus far.

Now, we're to the point where something close to the final visual skin is in our
internal builds and, being quite different from the temporary skin, we expect it
to have some effect on how people parse, find, and use the new UI.  The
next few eye-tracking studies are to learn about the differences between the two
skins and figure out if and how we might need to tweak the new look.

(No, the new visual look will not be in Beta 1.)

Q: Will results be made available publicly?
A: Generally, we don't make verbatim Office usability reports available because
there are certain internal techniques we use and data we gather that present a
competitive advantage to Microsoft.  Our usability team has worked hard to develop
innovative methodologies and a rich base of data about how people use Office
that helps us compete in the marketplace.

At the same time, you're hearing directly from the product teams of Office for
the first time through our blogs, and I'm definitely able to share general
results ( good and bad .) 
I don't have any marketing person telling me what to say or not say, and if I
did, you wouldn't read.  I get that.

The other reason you can count on getting a lot of information about the Office
12 UI is that our customers are demanding it.  We realize that no matter
how positive the change is, it's still change--and with that comes risk. 
My team is committed to providing clear data that shows the benefits and
highlights any
potential downsides as well (so that we can help people mitigate them during
deployment.)

So, I'll be very open about what we learn in general terms, without publishing
full 30-page reports online.

Q: What is "Send a Smile?"

A: There's a general philosophy Microsoft has been embracing more and more in
all of our beta products, which is that people should be able to send one-off
comments as easily as possible, while they're "in the moment."  Windows XP
had a "Comments?" link in every dialog box that let you tell us if the dialog
was stupid.  Previous versions of Office had the same thing.

Send-a-Smile is a related tool that goes a bit further.  Anywhere, anytime,
someone can click a "smiley face" to tell us they like something or a "frowny
face" to tell us they don't like something.  We get a lot of context (with
the user's permission of course), including a screenshot, sometimes a short
movie of the last 30 seconds, related documents, etc.  There's another tool
called the Office Feedback Tool (also known as "Ebert") which does a similar
thing but with Thumbs Up and Thumbs Down.

All of these tools work on the principal that if someone has to open a
newsreader, log onto a
newsgroup, type a long message, and send it, we'll lose a lot of valuable
feedback just due to complacency.  The idea is to reduce the barrier to entry for sending comments
so that we get more data from the "heat of the moment."

And of course, we have all sorts of tools that help us sort an analyze the
feedback on the back-end.

Q: OK, really, you better tell us how many people are involved in evaluating
the usability of the new Office UI. Because you blew me off back on
question 1.
A: A handful of people are involved full-time, and a lot of people are
involved part-time.  Every piece of usability feedback we get from running
individual feature tests (in Word, Excel, etc.) is really data about the user
interface as well, so it's hard to say exactly.  I will say this: at least
as many people have helped with the research and validation around the UI as the
core people working on the design of it.  It's a big effort.

Q: But Mr. Wizard, why isn't the volcano doing anything?

A: "I haven't put the vinegar in it yet, Billy.  Hush!"

Comments

• Anonymous
  October 25, 2005
  >>we'll lose a lot of valuable feedback just due to complacency.
  
  Ahh, Jensen, this is Marketing. Lets not call our customers complacent. A bit of a negative connotation there. How about going with "we'll lose a lot of valuable feedback because of the hassle." Shift the blame to us, you know?
  

• Anonymous
  October 25, 2005
  Excellent stuff, Jensen.
  
  I like the idea of the smiley/frowny buttons. How do you separate wheat from the chaff? YOu must get thousands of malicious/immature/stupid responses to that kind of thing.
  
  And could you be accused that by doing this, you actually are trying to keep errors and bugs secret, and prevent them being made public?
  
  i personally don't believe that that's the case, but you must expect that criticism to be levelled at you - would you ever make the feedback public?

• Anonymous
  October 25, 2005
  The comment has been removed

• Anonymous
  October 25, 2005
  Andy:
  
  I think you'll as much of this data from Microsoft than any company in the world.
  
  I'll continue to share general results and what we learned from them as I've done since I've started writing.
  
  You're right, some immature responses do come in, but again beta testers are evaluated on the quality of feedback they give, so its generally worth it to them to give useful feedback.
  
  On the back-end, it's a combination of manual tagging, search, and other technologies to make sure the right people see the feedback.

• Anonymous
  October 25, 2005
  thanks for the volcano reference. my 7 year old daughter and i have been having alot of fun with vinegar volcanoes recently.
  
  dan

• Anonymous
  October 25, 2005
  I keep thinking, should we all be doing similar things in our own software products? For example, I have a vertical market app where the users are opinionated, but difficult to get much detailed, usable feedback from. I can see the smile/frown thing working really well for those users. It could pop up a textbox for optional comment, then email all the details to me, behind the scenes (but with their knowledge). This would greatly increase my ability to find out which parts of my software they like, & which they do not.

• Anonymous
  October 25, 2005
  TC:
  
  You should give it a try maybe. I wrote the tool I mentioned in the article ("Ebert") in a weekend with a simple UI and capability to automatically capture a screenshot and some text and send it to us via SMTP.
  
  People are more willing to give their opinion when it costs them less time. (And the opinions are better, Schrödinger's Cat and all that...)

• Anonymous
  October 26, 2005
  Jensen,
  First, your blog is phantastic!
  Can't wait to get Office 12.
  
  One question:
  Do you use personas as defined by Alan Cooper during the design of Office 12? I know that MSN and the Windows group do use them now, but how about the Office group?
  
  Thank you!
  Roland
  
  PS: For those not familiar with the Personas concept, see: http://www.cooper.com/content/insights/newsletters/2003_08/Origin_of_Personas.asp
  
  

• Anonymous
  October 26, 2005
  Roland:
  
  I'm meaning to write about that in a future post. Thanks for the reminder, it kind of slipped off my radar.

• Anonymous
  October 26, 2005
  Please, PLEASE tell me the final "skin" for Office 12 will fit in with the OS that it is running on.
  
  Office has always done its own thing, UI-wise. But it's at least looked somewhat like the OS it's running on. What we've seen so far is that someone spent a lot of time developing a skin for Office 12 that looks nothing like XP and nothing like Vista.
  
  There is no reason for the most-used, most-prominent Windows app to use skinning. It should use Windows APIs and follow Windows guidelines. Leave skins for Media Players, IMs, and other taky consumer products.
  
  Please take a look at the comments in this blog entry of Raymond Chen's: http://blogs.msdn.com/oldnewthing/archive/2004/07/28/199589.aspx
  
  There's a lot of feedback in there about consistency between Office and Windows. There is ONE plausible argument given for Office having its own UI: Michael Grier wrote "Office ships out of band with the operating system and the support and user education costs associated with Office having a different UI on different operating systems is excessive. It's the whole economy-of-scale thing."
  
  I think there are a lot less (at least an order of magnitude) Office users using the same version of Office on two versions of Windows, than there are users using Office along with Windows-standards-compliant apps on a single version of Windows. What about the cost of using an office suite that looks nothing like the rest of the apps on your system?
  
  This is an issue which many people (including myself) feel quite strongly about. Personally, I've left comments on several blogs (including this one) and several Channel9 video forums, where I bring up this subject, and every time it gets silently ignored.
  
  Can we please get some kind of comment for why Office always has to fly in the face of Windows UI guidelines, and whether Office 12 will follow XP and Vista guidelines?
  
  (Quick example: from the page "Top Rules for the Windows Vista User Experience", which is part of the Vista User Experience Guidelines available at http://msdn.microsoft.com/windowsvista/uxguide/: "Use the standard window frame." Why oh why does Office 12 NOT?!)
  

• Anonymous
  October 30, 2005
  How about linking into the MSDN product feedback system?
  
  http://lab.msdn.microsoft.com/productfeedback/
  
  (Or something equivalent.)
  
  It is always useful to know you have a repeat issue and can just add to the existing issue, rather than creating one of your own.

• Anonymous
  November 01, 2005
  The comment has been removed

• Anonymous
  October 27, 2008
  PingBack from http://mstechnews.info/2008/10/the-office-2007-ui-bible/

• Anonymous
  June 02, 2009
  PingBack from http://woodtvstand.info/story.php?id=88682

• Anonymous
  June 07, 2009
  PingBack from http://besteyecreamsite.info/story.php?id=2609

• Anonymous
  June 09, 2009
  PingBack from http://cellulitecreamsite.info/story.php?id=10256

• Anonymous
  June 15, 2009
  PingBack from http://unemploymentofficeresource.info/story.php?id=7906