It's automation focus week in OneNote Test

 

We started another automation push yesterday and are turning out new scripts, fixing old, clearing automation system bugs off our plate and generally trying to get some breadth of coverage in place using our new white box testing system.

 

The surprising thing some new testers don't expect is that we file bugs against ourselves to get our automation system working properly.  Broadly speaking, testers are responsible for 2 facets of the automation system: the scripts themselves which run to test the product, and the underlying framework which lets the system run.  New testers are expected to be able to write automation scripts as a minimum requirement.  Once you can get your mind around that task, we expect you to contribute to the framework.  The framework itself (named MOTIF in Office, for "Microsoft Office Test InFrastructure") is built and maintained by testers.  When he hit bugs in that product, we file bug reports that we are responsible for fixing.  Think of MOTIF as a product we use (the Office group) and we have to maintain, add features, fix bugs, respond to customer requests and the like. And since the test team owns the product, we own the bugs in it. 

 

One of our tasks is to work through the backlog of OneNote specific bugs to reduce the number of active bugs, increase the quality of the system and ensure that when automation runs complete we get a valid picture of what the state of OneNote is.

 

One of the bugs I had to fix was this (and this is a very trivial bug, but shows the process of fixing bugs in the automation system pretty well).  The "legacy" napkin math script has a test which computes interest rate payments using this formula:

pmt(0.05;36;30000)=1813.033713614259   

 

So a $30,000 loan at 5% interest for 36 months gives a monthly payment of ~$1813.

 

The previous method of verifying the math was computed correctly used a String.Contains() method to look for $1813.  It was something like this:

string answer = "1813";

String formula= Compute("pmt(0.05;36;30000)= ");

If(formula.Contains(answer)

Log.Pass("PMT function works!")

Else

Log.Fail("PMT did not work.  The answer was " + answer);

 

I did not like this verification when I rewrote other parts of the script.  It seemed too "loose" to me:  if I am testing the number of decimal digits expected, then I need to be more precise than expecting the answer to Contain the result.  In this case, I want to verify the result of the equation is exactly 1813.033713614259, and not simply "near" 1813.

 

So I rewrote the verification routine to be something like this:

String answer = "1813.033713614259";

String result= Compute("pmt(0.05;36;30000)= ");//change the Compute function to only return the answer

If(result.Equals(answer))

Log.Pass("PMT function works!")

Else

Log.Fail("PMT did not work.  The answer was " + result);

 

Next I had to rebuild the script to ensure I'm not breaking anything.  Then I run the test 10 times to make sure it passes (some tests we run hundreds of times to ensure reliability before we enable them).  It passed.

 

Once all that is done, I submit my changes for a code review.  Another tester on the team will look through both the syntax of my change and the logic of the verification. Once she is done with that, she sends me mail to let me know the code is approved, and I can check it in.  Future automation runs will include this more rigorous verification.  Once it actually is running in the lab, I can close the bug I have on my plate which says to update this test.

 

It's the same process for more thorough bugs as well.

 

Oh, and I also added an equation test for 850*77.1=65,535.0 :)

 

Questions, comments, concerns and criticisms?

John

Comments

• Anonymous
  November 01, 2007
  PingBack from http://msdnrss.thecoderblogs.com/2007/11/01/its-automation-focus-week-in-onenote-test/

• Anonymous
  November 02, 2007
  Since it is "napkin math" I think results should always appear with either 2 decimals, or the greatest number of decimals used in the calculation (so if I multiply .005 by 11 it shows a precision of 3 decimals) If you need to view precision in the hundred-billionths, as in 1813.033713614259, you probably aren't turning to OneNote. Especially a function like "pmt" which implies "money."  The extra numbers become visually noisy.   But I can see not wanting to automatically round numbers for later calculations. Maybe default to round to two decimals, and an autocorrect-like dropdown to "show all precision" if wanted?

• Anonymous
  November 02, 2007
  I agree with you on both counts, Seth.  It would be interesting to test the currency level of precision on non-US systems: do yen go past 2 decimals?  Hmm... But for the test I added for 850*77.1, notice the answer does go to the level of precision of the input. For undefined levels of precision (such as pmt, which can be repurposed to populations, etc...) ON goes as far as it can given the input constraints.  We default to too much data rather than too little.   Still, this feature was designed for quick calculations and for that purpose it is pretty solid.   John

• Anonymous
  November 21, 2007
  MOTIF - Microsoft Office/Open Test Integration Framework

• Anonymous
  December 20, 2007
  How much code does a SDET write at MS? Do you think it's a good role for a talented developer to take over? And what if I want to change to dev role? How difficult it will be? I must say, your blog tells me that SDET is not as bad as I had thought. You guys do write some code after all, maybe not as much as I would want to :sigh:.