Trying to narrow down a set of reproducible steps

We had a small test pass last week using the Windows Phone version of OneNote. We had four testers co-authoring in a shared notebook and we each were using a different phone. I'll just call the phones alpha, beta, gamma and delta since we found no problems unique to any specific manufacturer. I was using the alpha phone and in this case, this was our low end phone (think "low memory device").

As we worked through our tests, I noticed some odd behavior that I could not get consistently to occur. When I was scrolling, every so often the scrolling would just stop occurring. To trigger scrolling, I was making a long selection and just dragging my finger to the bottom of the screen, and three times out of 19 attempts the phone "got stuck". Further, once when this happened I decided to just keep my finger in place, and after about 10 seconds I started seeing the scrolling again. That told me that OneNote is (probably) fine and we were just waiting for the phone to devote some more resources to us.

But that is only one theorem I had. It could also be a faulty screen on the phone and maybe the (x,y) coordinates where I was dragging my finger are faulty. I could also be moving my finger off screen, or right at the edge of the screen, and never noticed.

I asked if anyone else in the room was seeing this. After all, this could still be behavior that only my fat fingers can trigger. No one else had. That doesn't tell me much - it doesn't eliminate OneNote, the device, or my own fat fingers from the matrix. So we each tried it a number of extra times, and finally one of the testers with the delta device got the behavior to occur on her phone.

That eliminated the hardware from the matrix of potential causes. Next I gave her my phone and she let me use hers and I was not able to reproduce the problem. After several attempts, she could hit the problem with the phone I loaned her.

Finally, after much close inspection, she noticed that in my case and in hers we were both dragging our fingers off the screen. That caused the scrolling to stop since Windows and OneNote both no longer got any input at all.

And this is a nice reminder that not every error condition testers create is a bug. It can simply be the case of "user error" and that is exactly what happened here.  We spent a lot of time trying to narrow down the exact repro here which is what test always needs to do when hitting bugs.  Just in this case there was no bug.

Now back to testing.

Questions, comments, concerns and criticisms always welcome,

John

Comments

• Anonymous
  January 31, 2014
  The question you have to ask here is: If two experienced testers have this issue, how many thousands of actual consumers will have it and down-rate, not recommend, or bad-mouth the product because of it? Shouldn't there be some indication that you're leaving the touch area?

• Anonymous
  January 31, 2014
  Yes - you have a valid point.  The balancing act here is also complicated by comparing "engineers on the OneNote team usage patterns" to "customer" usage patterns - this may not be the best example, but we always have to take into account that I, as a tester and with my knowledge of the product, may not always be able to mimic real world usage. So to help here we have frequent customer focus studies in which we invite customers to come in, sit in a room and we (program managers, typically, but we all can attend) watch them use OneNote.  We get much better results from real world usage this way. I get your point about this particular case though.  I'll see if we can do anything (how to detect the finger went off screen rather than being lifted right near the edge of the screen will be hard to detect...).