Field Trial

We built OneNote in relative secrecy during 2001-2002. Well, it wasn’t cloak and dagger stuff - we just didn't make any effort to talk about it outside of Office - although rumours that Office was building some "TabletPC app" did spread to other parts of the company, a misconception which until recently was probably worse inside the company than outside, since we were able to communicate pretty clearly to the public the message we wanted to get across after we announced. The result of this lack of exposure was that we had become a little insular and no longer had a good perspective on the product as a whole - was it useful? Did it suck?

Although we kept the project secret, there was a select group of total outsiders whom we enlisted to burst our bubbles and give us raw real world feedback. Now, we had done usability tests on individual features, but we had not yet had the whole app up and functioning well enough to do real "dogfooding" - that is, living with the product and trying to do real work with it, rather than just test it or play with it. For that we wanted people in a variety of life situations - students, professionals - anyone who used a computer and tried to take notes or do research.

We recruited around 20 people in the Seattle area to become part of our "Field trial". We narrowed down a pool of about a hundred people to a set of around 30 whom we thought would make good candidates based on their answers to our screening questions. We then approached these people personally to verify if they would be a good fit, and if they were, we invited them to join. Since we had 6 program managers on the team and I had decided about 3 participants per PM was a good ratio given the time investment we were going to make, we wanted 18 participants, with a few as backup in case some had to drop out. We got 20, and prepared the NDAs (non-disclosure agreements, whereby the participants agreed not to blab to others or the press about what they were doing for cheap glory). We had 6 students at the University of Washington, and the rest were spread across quite a spectrum, from a dean at the university, to a health care professional, to a writer, to a belly dancing instructor.

Although OneNote could barely stay up for a few minutes at a time, we already had very good file robustness, so people were not likely to lose any data even if they crashed. Always saving helps in that regard, as does a new file structure designed explicitly to prevent corruption and allow super-fast incremental saves. Although I am a huge XML fan, sometimes binary file formats just ARE better at some things. (note: that doesn’t mean we won't do XML in the future - you just gotta admit that XML isn’t always the best at everything)

We collected a bunch of tablets (mostly beta hardware and prototypes, since there were so few to go around in the fall of 2002 before they had launched) and gave out the machines with an "alpha" build of OneNote on them. We agreed with our participants that we would meet them for 2 hrs a week - one visit was to watch them in action doing whatever they do with the tablet and OneNote, and the other hour was an interview where we asked them to describe their experiences, what they liked and didn’t like, what they had discovered that week, etc. We also committed to be on call to save them if a disaster struck with their notes. We kept a tally of what features were discovered when, and if people deviated too much off the norm, we stepped in and told them about some things so that they wouldn't miss out on the later parts of the trial. The trial ;lasted 8 weeks, getting progressively more advanced. We lost about 5 of the participants over that time – some just didn’t use the product, and a couple found they couldn’t handle the time commitment.

The trial went pretty well, although the instability of the alpha meant people were taking notes in class and had to switch to paper when the app came tumbling down. We also found unsurprisingly that the willingness to adopt new technology more or less correlated with age (can’t teach an old dog new tricks!). And on top of that the students were just animals in terms of adopting OneNote – it was clearly a good fit for them.

We learned a lot from the trial - it helped answer critical questions about our user model, our assumptions about how people would use the product and to what end, what features just didn’t work, what features would get used a lot, and importantly, what expectations people would have before they used the product. This latter was important because as we crafted our message to analysts, the press and the public, we had to anticipate what conclusions they would jump to when we announced, and tune our statements to make sure they did not come away with the wrong impression (like, it is a Tablet-only app!). One thing that the field trial helped us with was to make abundantly clear that the ink model we had was weighted too heavily on structure, as I discuss below. That prompted the ink-o-rama sessions to get ink to a shippable state.

BTW if any of our field trial participants read this and want to drop me a line, that would be great. We’ve lost track of several of you and we wanted to give you a free copy...

Comments

• Anonymous
  February 02, 2004
  your blogs are most interesting - some great insights into what's behind the covers at MS-Land.
• Anonymous
  February 03, 2004
  I agree. This is a great blog. My dev shop uses OneNote extensively for sharing client meeting notes using Sharepoint specifically. Do you have any comments on integration with sharepoint and how (if at all) it might change going forward? It seems to work well, though not necessarily what I would've expected. The inclusion of a shortcut in "My Notebook" to the sharepoint source file makes me apprehensive.
• Anonymous
  February 03, 2004
  OneNote lets you store one or more sections in remote locations. These can be file shares or WebDAV (such as many web sites and SharePoint). When you Do "Save As" and put a section in SharePoint, or upload a section to SharePoint and then open it into OneNote, OneNote puts a shortcut to that remote section in your notebook to make it easy to access that section when you want. I am not sure why it makes you apprehensive - can you explain?
  Going forward you'll see this work more smoothly and be a richer experience. I can't really say much about it at this point.
  
• Anonymous
  February 03, 2004
  Chris,
  
  Love your name :)
  As a fulltime developer/product manager/entrepreneur I love the insights you give. We might have different budgets but the process of developing software for MS or for my company looks surprisingly familiar. We also carefully evaluate features, vision, complexity and potential revenue. In addition, I love seeing customers using the products that I helped develop. It certainly makes my day if they actually push the app to its limits. Hope you will keep sharing more of your experiences, thoughts and insights. Regarding the MS blogging crowd >> John Porcaro has a nice topic regarding that >> http://johnporcaro.typepad.com/blog/2004/01/how_do_blogs_he.html
  
• Anonymous
  February 03, 2004
  I just did some tests to figure out what my apprehension actually was. It's not necessarily because of the linked file. That linked file seems to be smarter than I thought it was.
  
  
  The real oddness I find is that if I have a section called "New Section 2", and then do a Save As to sharepoint. As expected it puts it over in sharepoint and a link file to that copy. But wait..there's also a "New Section 2" OneNote file still in my Notebook folder. So when I open the SharePoint copy, the "New Section 2" still in my notebook folder also opens. I think, "but wait"...I put it in sharepoint? what's this?
  
  I guess it just has to do with the auto saving and I guess I'm not used to the software autosaving for me. When I do the SaveAs, I'm instinctivly thinking that it's the only copy of the file. What I end up doing is after I put the doc in sharepoint, I navigate to my notebook folder and delete that "original" to make sure I don't get confused.
• Anonymous
  February 09, 2004
  Iffy,
  Regarding save as, we had some debates about what to do here. The simplest would be to treat your note section like a file, and let you "save as" to in effect move it, leaving a closed local copy on your hard disk (as Word or any other app would do). But we worried that people often want to put things on a share so that others can see them, yet want to work locally with their own files. If we did a Save As like Word, you would "lose" your own copy (actually Word doesn't remove the original either). So instead of closing the local copy, we just keep it open. In the future we hope to solve this better by allowing you to move files or more explicitly make a copy in a different location. To answer your concern though - the section with the shortcut is the one on the server, and the other one is the local copy, which you can close or delete if you wanted to "move" your section to a Sharepoint (as you are doing).