Getting ready for a setup pass

We have several recurring test passes that we do on a seemingly always recurring basis. One example is performance passes - you can never have too much performance - international passes, accessibility and now we are focused on setup.

Setup seems pretty easy, and in one respect it is. You "just click setup" and then make sure OneNote starts when you are done.  (It's very easy to think that after setup testing passes, all you have have really done is unblocked everyone else and they can now get the rest of the testing done.) But this hides a huge test matrix.

One of the most easily defined variables we take into account is the operating system. For OneNote 2010, we tested with Windows XP, Vista, Windows 7, Windows 2003 Server, Windows 2008 Server and Windows 2008 Server R2. So right off the bat, we had 6 versions of Windows to test. Then remember each version of Windows may have Service Packs available and the matrix begins to grow. And the servers have terminal server options, Windows 7 has Windows XP mode and there are 32 and 64 bit versions of each. This takes the 6 original operating systems and ramps it up to 30 at least.

After that, the next most obvious variable to define is whether a previous version of OneNote is installed. Here we had it a tiny bit easier than other applications which have more previous versions - we had only to consider to OneNote 2003 and OneNote 2007. Now we are at 90 test cases.

Once you start to look at that even this simple matrix and compare it to our features that install during setup, you can identify some holes in our coverage here. OneNote 2010 installs addins for Word, Outlook, Internet Explorer an PowerPoint. And the Outlook addin worked for Outlook 2003 and 2007 in addition to 2010, and the IE addin needed to work with IE6 through 10. Same for Word and PowerPoint.

So we had to create cases for each previous version of Office.

And then you can imagine users that have (and keep) PowerPoint 2007 but upgrade the rest of Office to 2010.

The test matrix now stands at over 1000 test cases. And we still haven't considered uninstalling (downgrading), repair cases, the Send to OneNote printer (and whether or not the needed XPS printer and .net framework was installed), third party addins, administrative vs. user installs, and so on.

And once you get a huge test matrix like this defined, you really start to consider automating all test setup cases so you don't have to do this manually….

A tester's work is never done.

Questions, comments, concerns and criticisms always welcome,

John

Comments

• Anonymous
  August 07, 2012
  John - as a OneNote evangelist, just want you to know that I apprecitae the work you guys do in making sure OneNote stays a solid product to use. I've seen in a previous blog that your most frequently asked question is about tagging, but the one I get asked about it image editing / cropping which seems like such an easy feature for you guys to add. OneNote looses some credability when I'm having to advise people to edit the image in Word the copy / paste it in to OneNote. I've even noticed, and given feedback on the fact that it's not in the 2013 preview version - much to mine and other colleagues dismay. Do you know if this is coming?  :-)

• Anonymous
  August 07, 2012
  Hi Steven - for image cropping, have you checked out Onetastic?  www.omeratay.com/onetastic  Omer added this support to his addin.

• Anonymous
  August 08, 2012
  John - thanks I'll give it a go. Would still be great to see the whole Picture Tools / Format tab in OneNote, one day maybe, one day.