Some A/B Testing for OneNote layout

Back in May, I wrote a little about A/B testing for OneNote and used napkin math as an example. We are about do some more of this and it will involve our entire team so I wanted to mention that here.

We are considering a small change to our layout code - the code that control where and how outlines, images and all other page content appears on the page. The change is small but we want to ensure it makes our layout even better. The complication is that even though our change is small, it may consist of several changes made to different areas. Since they all interact with each other - adding an image can cause an outline to scroll offscreen for one example - we need to understand how each change works on its own in addition to how they work when combined.

So what we will do is take our first proposed fix into our dogfood builds and use them for a while. Once we verify the behavior of the change, we can take the second proposed fix, again in isolation. Then we can combine the two once all validation is complete.

The advantage this gives us is that if we see some odd behavior at any given point, we can see if the first change was the likely culprit, or the second change (if that is what we are currently using for our day to day work), or if it is the combination of the two changes together. That makes tracing and debugging much easier for everyone involved. It is always easier to isolate a problem with fewer changes to a system than with many changes to that same system.

I hope this makes sense. It came up this week internally when someone was asking when we would see the combined proposed changes, and I used that opportunity to add some core testing concepts into the conversation. The more testing we can make part of our daily work by making little process changes like this the better we will all be.

Questions, comments, concerns and criticisms always welcome,

John

Comments

• Anonymous
  November 17, 2013
  The comment has been removed

• Anonymous
  November 17, 2013
  Let me follow up on that.  Are you using HTML or RTF mail format (or both)? And does the same thing happen when you copy/paste from Word?

• Anonymous
  November 19, 2013
  I've just tried both RTF and HTML; and in both cases pasting an outline from email to OneNote results in the odd looking embed.  Pasting the same thing from email into word works fine. Word into OneNote fails in just the same way as Email to OneNote. A.

• Anonymous
  December 09, 2013
  The comment has been removed