How We Evaluate the Experiences We Engineer

How do you know when an experience is ready for consumers? This is something we ask ourselves all the time. In this post, we’ll cover how we set our experience goals for IE9 and how we measured (and continue to measure) our progress toward these goals throughout the development cycle.

We set experience goals for all of the products we ship. These goals are at the product and “experience” (i.e., a meaningful unit of experience for people, not at the feature level) levels of analysis. We think about these goals in what we call a “Confidence Model.” This is a model that evaluates the product experiences across four dimensions – Useful, Usable, Desirable, and Principled. For each experience across each dimension, we have a “confidence rating” of how close we are to meeting (or exceeding) the experience goals. The dimensions of useful, usable, and desirable are common measures of experience across industries. In addition to those standards, we also evaluate our experiences in light of the Windows Experience Principles (more on these later). This is all an evolution of our process that we described in the Windows 7 Engineering Blog.

What we mean by “Useful, Usable, Desirable, and Principled”

When we think of experiences on these four dimensions, we have specific definitions in mind:

“Useful” is the notion of value to people. We want the experiences in IE9 to help people do something they couldn’t do before, save them time, energy, and/or effort, and have things get noticeably better in a way that matters to people.

“Usable” is what people typically think about when considering the “usability” of a product, site, device, etc. We measure how Usable an experience is on the standard factors – task success, time to completion, ease of use/setup, comprehension, confidence, security, etc.

“Desirable” is how an experience evokes the intended emotional response and perception. We have general aspirations for any experience (e.g., love) but then choose specific emotions each experience should elicit (e.g., feelings of control, efficiency, connectedness, clean).

“Principled” is how an experience expresses the Windows Experience Principles. We review each experience at regular points in the development cycle to make sure it is expressing our Principles. These principles are inspired by data, informed by our values, and tweaked by experience. They are not strict rules, but rather values we aspire to and shape all the experiences we design in our products. We first talked about our “design principles” for Windows 7 back at PDC 2008. Since then, we have evolved these into our current Experience Principles. Later in this post, we’ll cover how we used one of the principles to inform a design decision in IE9.

How we set and track goals

Across those four dimensions, we set goals for each of a set of experiences. These goals have multiple inputs including previous experience research, design explorations, and technical investigations. The goals are generated, iterated on, and agreed upon by the entire engineering team. They become the team-wide agreement of “what success looks like” for an experience and the bar we all hold ourselves to. We then establish specific metrics for tracking progress toward each goal.

For example, we had specific Useful goals for some of the general browsing experiences: People articulate the value of having a fast and fluid browsing experience as:

• They can get to their sites in fewer steps than in previous versions of IE
• They see that pages load faster than previous versions of IE
• They save time on their common and frequent tasks compared to with previous versions of IE
• They are aware of what is slowing down their experience

Progress toward these goals is measured primarily through qualitative feedback from participants in our research labs and also through community feedback.

After goals for each experience are set, we track them throughout the development cycle. For IE9, as with all our products, we used a large set of methods in our research, but there are three we rely on the most – lab studies, field research (e.g., visits to people’s homes) and usage instrumentation. Each of these methods has their own strengths and they give us different lenses to look at and understand the experiences we are building. During IE9 development, we conducted six different lab studies with about 60 different participants. These lab studies are great for understanding experiences as we moved from prototypes to working code and allowed us to control for whatever influences may bias our results. In addition to the lab studies, we also went on many site visits to people’s homes to give us a look at their experiences over time, in everyday settings (e.g., their living rooms), and produce specific examples of how people actually use our products in their lives. The insights we gained on our initial site visits in the summer of 2009 around how people were using web sites like their other applications was highly influential on our plans to build what eventually became Pinned Sites. Lastly, usage instrumentation of how people are currently using IE8 and IE7 gave us the huge datasets (tens of millions of people, hundreds of millions of sessions as Dean mentioned in his post) required to know what the common and frequent behaviors and patterns of usage are with our products. This data informed our decisions about which behaviors we optimized for in IE9.

We are constantly evaluating our experiences with these methods and the team always has a pulse on if we are on the right trajectory to meet our goals. We make adjustments to our products and experiences based on our research, something we call “data-informed decision making.”

Goals for Internet Explorer 9

As Jane described in her post, we started with three overarching goals for the experiences in Internet Explorer 9 – Sites Shine, A Natural Extension of Windows 7, and Fast, Safe, and Reliable. Throughout development, we evaluated our progress against these goals in general and specific goals for different experiences.

For example, here are some of the specific Usable goals for our navigation experiences:

People can get to where the way to go fast. This means people:

• Successfully get to sites they want to visit
• Successfully get to sites they have previously visited within a timeframe that meets or exceeds expectations
• Accurately articulate when a page is finished loading
• Successfully pin sites

It’s important to note that these goals are not for a specific feature. They span many features including Address bar, New Tab Page, Pinned Sites, and progress indication, because all of these contribute to the experience of navigation to sites with the browser.

For the last goal of “Successfully pin sites,” we originally had problems with the design. You could pin by dragging the icon from the Address bar to the Taskbar and dragging the site from the New Tab Page, but not by dragging the tab the site was in to the Taskbar. Through our research, it became clear that this was one of the top ways people attempted to pin sites when trying the feature. This video clip shows one representative participant whose first instinct was to drag tabs to the Taskbar to pin them (you’ll also see the red dot of some eye-tracking research we were doing during that study as well):

Based on this research, we knew that enabling pinning sites by dragging the tab to the Taskbar would meet people’s expectations and remove one more hurdle to people having the sites they love, need, and want at their fingertips. We then verified this decision with further research. Here is a participant pinning a site by dragging a tab to the Taskbar in a later build:

Over the course of these studies, our confidence that we were on track increased as we made changes to the design and verified we were improving the experience. This occurred across most of the features of the product. This is just one example.

For another example, here are some of the specific Usable goals for our fluid browsing experiences (different from the ones for the navigation experiences above):

People can fluidly move among sites . This means people:

• Successfully see sites side-by-side (with or without tabs)
• Successfully find the functionality they use most often
• Successfully get to their homepage after navigating to different sites
• Successfully recognize whether a webpage is secure or not without prompting
• Successfully queue multiple tabs

For the first goal of “Successfully see sites side-by-side (with or without tabs),” we evaluated many different designs to accomplish this goal. One of the tools we used to choose the design we built is the Experience Principle of “Reduce concepts to increase confidence.” This Principle is about taking advantage of what people already know and introducing new concepts only when necessary. We try to make only meaningful distinctions among concepts that people will understand and get value from.

As described in Jane’s earlier post, you can drag tabs out of Internet Explorer windows and directly Snap them to one side of the screen or the other on Windows 7. We deliberately built on what people were already using in Windows 7 and extend that experience to Internet Explorer 9. No relearning, no new concepts. People who use Windows 7 Snap already know how to use Snap with Internet Explorer 9.

These are just two examples of how we set goals for experiences and then evaluate their progress on the dimensions of Useful, Usable, Desirable, and Principled. We do this for all of the experiences and products across Internet Explorer, Windows, and Windows Live.

Setting goals for experiences across the four dimensions – Useful, Usable, Desirable, and Principled – and tracking our progress toward these goals across the development cycle is an important part of shipping the experiences we believe in. We hope you enjoy using IE9 beta. Please send your feedback in the comments and on Connect.

Jess Holbrook
User Experience Research Lead
Internet Explorer

Comments

• Anonymous
  September 28, 2010
  "Accurately articulate when a page is finished loading" - I don't get that from IE (or any browser for that matter). Short of the "Done" bottom right (which I had to re-enable), there isn't really any indication of that. Also, I've been using exclusively IE9 since the launch of the beta and I still haven't gotten used to the notification bar at the bottom. A more prominent system is necessary for actually immediately actionable things (e.g. starting a download, displaying insecure material on a webpage, etc). Otherwise, it looks like either a) the browser is ignoring my actions (in the case of the download) or b) there is something wrong with the rendering (CSS is being served over an insecure connection and I miss the prompt). The previous prompting system was annoying, but parts of this are a whole lot worse. I agree that I don't need a giant whopping POPUP BLOCKED that forces a reflow of the elements, but I also need something that'll catch my attention for something I need to see.

• Anonymous
  September 28, 2010
  I am quite happy with the IE9 beta. I am a long-time user of Firefox, but it is quite likely that I will move back to IE when it reaches final. However, my IE satisfaction seems to have very little to do with your user experience goals and objectives. It is quite simple: IE9 shines because of the vastly expanded standards support. It is not perfect, but it is a huge improvement over IE8. I really could not care less about interface, Win7 integration etc. I would be just as happy if you used the old IE6 interface and still had the same standards support. While I am extremely happy with IE9 beta now (September), my only concern is that by the time you reach final, your competition will have already reached and surpassed you in standards support. When that happens, I will probably go back to another browser. Here is hoping that IE10 is not too far away. Hopefully it will be a lot shorter of a development cycle. Again, congratulations on an excellent job in IE9.

• Anonymous
  September 28, 2010
  although i'm rather satisfied with the beta, you should add one more criteria: consistency

1. download a file with the same name and it automatically renames it, instead of prompting. this is not how any other program in windows or windows itself works.
2. open the rss feeds. it's on the right. click to pin it, it moves to the left. doesn't make any sense. you moved it to the right, keep it there or move it back to the left in its unpinned state

• Anonymous
  September 28, 2010
  The comment has been removed

• Anonymous
  September 28, 2010
  The comments before this one just kind of sum it up for IE9! I generally agree with all of them.

• Anonymous
  September 28, 2010
  It Appears that the biggest problem is getting Web Masters and Developers to get up to spped for IE9 Beta Users. The Windows Geniune Advantage site site has the biggest Bug when it comes to handling IE9 Beta. The validation page just hangs. Most of the other MS Sites have been keeping up on breaking changes with each of the browsers.

• Anonymous
  September 28, 2010
  The comment has been removed

• Anonymous
  September 28, 2010
  The problem about IE' features discoverability is that you need to know the option exists to find it. You should really consider adding some start config dialog to help people change common parameters. This would lead to customer satisfaction because "I can customize my browser the way I want". In previous versions of IE, the "first setup" dialog was annoying because it prevented you to navigate. Using the new notification bar, you could add a notification like "You didn't customize Internet Explorer yet. You should take a few minutes to make your browser at your fingertips!" and then show a wizard giving information about most commonly used IE options (mainly UI-related & add-on options). Regards, François REMY

• Anonymous
  September 28, 2010
  Can you add the goal of spell checker?  Just a simple spell checker like the one that Chrome, Firefox, Opera and all of your competition has.

• Anonymous
  September 28, 2010
  Just want to let you know that this IEBlog is not compatible with IE 9 beta. Check it out yourself.

• Anonymous
  September 28, 2010
  The comment has been removed

• Anonymous
  September 28, 2010
  The comment has been removed

• Anonymous
  September 28, 2010
  i noticed the "page loading" spinner isn't working most of the time. I couldn't tell if I've already clicked on a link. Then the new page just loads.

• Anonymous
  September 29, 2010
  @FremyCompany: Ugh. The first-time dialog in IE8 was really bad - a lot of people didn't ever finish it and simply clicked "Remind me later" each time. I think there should be a notification in the 2-3 first IE9 sessions opened that says "Would you like to take a look at what IE9 has to offer ?" "Yes/No" and that redirects users to some website with all the new things, as well as the old-but-good ones, like BeautyOfTheWeb.com. @ParrotLover77: You know, the more options they offer, the more code they have to maintain, the greater the attack surface, et al. People who actually dislike new features (the Ribbon, the Start Menu search bar, etc.) are a minority. If you want something that can be tweaked to look like a 15-year-old program, go open-source - they love to keep compatibility with very old things and to keep old UIs.

• Anonymous
  September 29, 2010
  @mocax: The beta build has some known problems wherein the spinner stops spinning too early. thanks! @Jeffrey: Would you mind elaborating? All of us on the IE team use IE9 with the IEBlog. thanks!

• Anonymous
  September 29, 2010
  "Accurately articulate when a page is finished loading" - yeah not so much. This is currently a total failure in IE9 beta. In addition you'll even notice in the eye tracking that when a user has loaded a page, they do not look at the bottom for notifications.  They should be moved back up to the top of the screen where they relate better to the content being loaded - esp. if they are actually prompting the user to select a choice.

• Anonymous
  September 29, 2010
  @pmbAustin: If you've hidden a site from the New Tab page, the only way  (using IE's UI) to get it listed again is to clear your browser history. Doing so will clear the hashed entries under HKEY_CURRENT_USERSoftwareMicrosoftInternet ExplorerTabbedBrowsingNewTabPageExclude.

• Anonymous
  September 29, 2010
  @Aethec : Yeah, it's exactly what I've said: It should appear in the notification bar, not as a "modal dialog" popup. I really dislike anything modal in the browser (I even reported bug on Connect about that). The idea is to make it appear only the 3 first times you open IE (or until you clicked on the 'Lauch the wizard' button). Using a website is a great idea you've got, since it's easier to maintain a website "up-to-date" and reactive to user needs than a frozen setup dialog. It also keeps the user in a natural browser environment. (Please note that every time FireFox introduce something new (update...)) they open a page saying what's new. The concept is not so stupid as it may seem at first glance, I must say.)

• Anonymous
  September 29, 2010
  Something I don't understand about IE9: I understand the problem of the 2px border around the IE8- chrome. But I suppose you can agree with me when I say the chrome can have the size you want it to have. Then, Why don't you make the chrome 4px wider and taller than the actual window size when running in IE8- compat, to have the awful border shown outside the window display area (and thus not shown, in fact) ?

• Anonymous
  September 29, 2010
  Is there a Virtual PC image for Vista or Win 7 with IE9 beta on it (or installable) that us developers on XP can use? I realize I won't see the speed differences since the virtual PC likely won't be able to use my GPU the same way but I'd just like to test basic rendering and JavaScript. I have a scheduled IT upgrade from XP to 7 in April but I need to test our site and application in IE9 before then. Can I also suggest that links to the Virtual PC images be added to the list of links on the right side of this blog? It seems like a fairly important resource for developers that come to this blog. thanks taylor

• Anonymous
  September 29, 2010
  @taylor : Go to ietestdrive.com and download the "Technical Preview 5" (not the beta). This will install a minimal version of IE9 Beta 1 which don't require you to uninstall IE8 (it is not a full browser but you can safely use it to test the rendering on some websites).

• Anonymous
  September 29, 2010
  "•They save time on their common and frequent tasks compared to with previous versions of IE" I use Send To OneNote, Send Link by Email and QuickTabs commonly and frequently; they're all that little bit harder in the new UI, which is beautiful and elegant and frustrating for power users. I know we're not the big number users - but I think we're the vocal users. And I can't believe a good designer can't create a design that's equally intuitive and elegant but with the option of adding functionality and information. Also the lack of any progress bar leaves me feeling like the browser has fallen off a cliff every time a new page is supposed to load or a link is supposed to open. blank page; will it go anywhere? have I clicked the link? should I click it again? what's happening? very unsettling...

• Anonymous
  September 29, 2010
  @FremyCompany : the problem with the "technical preview" (and IE9) is that microsoft has been stupid enough to not support their main user base : the XP's users! Because of that, I'll probably continue to code in xhtml for the next 10 or 15 years. The only good thing is that I'm not supporting IE6 and IE7 anymore. For now, my customers are ok with that.

• Anonymous
  September 29, 2010
  @FremyCompany - unfortunately I'm using Windows XP thus I can't use the preview tool - thus I was hoping for a VPC image.

• Anonymous
  September 29, 2010
  @EricLaw [MSFT],

1. I am using IE 9 Beta and when I come to this site (blogs.msdn.com/b/ie/), it takes sever seconds to load the whole page, and it seems to be freezing up to me. I have to wait for several seconds before I can move page up and down. But Google Chrome and FireFox loads the same page much faster.
2. I see the "Compatibility View" tab next to the address bar, does it not mean this site is not compatible with IE 9? If you go to Microsoft's main site (www.microsoft.com), you will not see the "Compatibiltiy View" tab.

• Anonymous
  September 29, 2010
  It's actually harder to tell whether or not a page has finished loading now, plus whenever a page fails to load (site is down, internet just went down, etc) half the time it just keeps me on the original page instead of going to a 404, while in IE8 I would just have to hit refresh every once in a while instead of wondering if the page has even started to load. As far as speed of use goes, IE9 has not really sped up anything I use.  Favorites menu takes longer, sometimes I don't even notice notifications and keep clicking, I had to disable the toolbar I use because it's way too tiny to justify an entire new row, I have trouble identifying tabs because there isn't much room, i have to select the address bar just to see the exact page i'm on, the home page and favorites buttons are way off to the side but my mouse is always on the other side, etc.  A lot of these could be easily added as options instead of forced changes, but right now IE9 is a lot less convenient for me and has made simple tasks a bit more annoying.  It's very frustrating because I really want to use IE9 for all the new HTML/CSS/ETC, but IE8 has such a nice GUI in comparison. At the very least could we have to option for the old style favorites menu?  The one where you mouse over a folder and it pops out the contents to the side?  I don't like having to click on each folder to open/close it, plus its a bad use of vertical space.

• Anonymous
  September 29, 2010
  The comment has been removed

• Anonymous
  September 29, 2010
  <<I see the "Compatibility View" tab next to the address bar, does it not mean this site is not compatible with IE 9?>> No, that is not what the presence of a Compatibility View icon means. The Compatibility View icon provides you the opportunity to render a site using the Compatibility View settings.

• Anonymous
  September 29, 2010
  I highly recommend the presentation of Don Norman about what makes software a great experience: blog.businessofsoftware.org/.../don-norman-at-business-of-software-2010.html His points about Simplicity start at 0:29:15 I think the IE team should consider his points and not neglect the power users among IE users (though they're just the minority)...

• Anonymous
  September 29, 2010
  The comment has been removed

• Anonymous
  September 29, 2010
  The comment has been removed

• Anonymous
  September 29, 2010
  The comment has been removed

• Anonymous
  September 29, 2010
  Iconsistent icon issue IE9 Beta 1 1 drag tab to task bar puts 32x32 icon into the taskbar 2 drag favicon from the address bar to the taskbar, puts 16x16 icon in it

• Anonymous
  September 29, 2010
  Check www.html5test.com. IE9 is the worst by far (96 out of 300) and I simply don't see how it will reach somewhere around 210 (as Chrome) to be acceptable when will come out of beta. Who cares about how fast it browse Web sites or icons, toolbars... HTML5 is the future if you want to use your browser as a client to business apps. Maybe someone who cares and can do something may read this.

• Anonymous
  September 30, 2010
  VC: not sure how you missed the memo, but that test is garbage. It doesn't just test HTML5 (which, incidentally, isn't even done being written) and it includes user-agent sniffing.   Just proof that you should not believe everything you read on the internet. Any fool with ten bucks can buy a domain name and tell all the lies they want.

• Anonymous
  September 30, 2010
  SS:  "Similarly, I don't think issue 5 is too important either.  Both IE8 and Firefox have the same capability, but no visual indication.  I guess I'm having trouble picturing what kind of symbol would be appropriate that both grabs enough attention and doesn't grab too much attention and makes things look weird." HEH!  I had no idea this ability was in IE8 (and I've been using IT since the beta every single day)... you mentioned it, I tried it, and low and behold, you're correct!  But that seems to underscore my point that this function/feature is NOT discoverable at all. Visual Studio has little 'grab bars' on each of its toolbars (little dotted vertical lines).  A sort of vertical bar, or hashed/textured area, or a dotted bar... anything that would say "GRAB HERE" would be very useful, IMHO... and woudln't be distracting.  It wouldn't be the full height of the boxes on either side of it, but would supply a drag-target.   You're right, it's not hugely important (I didn't mean for the numbered list to imply priority), but it's just such a perfect example of how a useful feature is completely undiscoverable right now. Otherwise, thanks for the comments!

• Anonymous
  September 30, 2010
  Useful, Usable, Desirable, and Principled...all characteristics of Chrome, not IE 9.

• Anonymous
  October 01, 2010
  I find it a bit ironic that Microsoft has been putting so much focus into making our experiences "consistent" yet they fail to consider consistency between versions. Also, IE9, and in fact all recent MS products, are leaving us "power users" at the wayside. This is great for the masses, but a nightmare for those of us that have to use, administer, and support their products every single day. Perhaps they should consider how many clicks it takes to perform common tasks in addition to how pretty the UI is. If you change something that used to take 2 clicks and now takes 5 I don't care how good/clean the UI is, you failed. Finally, for the love of Yoda, isn't it about time you just put in a spell checker? I mean REALLY.

• Anonymous
  October 02, 2010
  EricLaw, also, regarding the hidden pages on the "new tab page"...  I cleared out that key in the registry, most of my hidden pages came back (and I re-hid them)... but not Facebook. Even though I probably visit that site more than any other (okay, it might be second or even third), it's nowhere to be seen in the list.  Sites that I KNOW I visit maybe 1/10th as often are in the list, but not Facebook.  I even visited the site (hitting the favorites button as well as typing it into the OneBar) dozens of times trying to get it to show up, and it still won't. Clearly something isn't working right. Any other advice?

• Anonymous
  October 03, 2010
  Blah: While the html5test is not perfect now there is an opportunity for a dominant Web browser such as IE to build the next generation as a platform for business applications and not just to browse few pages better and faster. While I like the fact that MS is looking to add new features (such as hardware acceleration), the focus should be on ensuring that standards such as Web Sockets, File API, Storage, new Form fields, drag-and-drop, etc are primarily covered. Why IE9 team doasn't take the initiative to create a new implementation for a Web SQL database which is the main obstacle in being pushed further as a standard?  They definitely have the resources and the knowledge to do that! Indexed DB is very primitive when comes to database support, while SQL has been around for a while. Regarding the current status, most of the HTML5 specs are completed and very few changes are expected. If there is a standard IE9 team doesn't like it then they should speak up and add their contribution.

• Anonymous
  October 08, 2010
  On www.winsupersite.com/.../ie9_ff_onebox.asp (assuming a later build) we can see a few screenshots that show the refresh and stop button on the left of the OneBox. I ask for the Home button to be moved there, too.