Manual Testing and Babysitting Your Tests

No matter how much test automation
you have nothing will ever substitute the value added by manual testing. I
could go on as to why but nothing beats a real story so here's what happened to
me a couple of weeks ago:

I had just finished implementing some
formatting tests for a new language service so I ran them to see if they were
doing what I wanted them to do (on a side note is testing the tests considered
recursion?). While one of the newer tests is running I hear some odd beeps from
my computer. I'm always listening to music while I work so at first I was
wondering if that was just part of whatever song was playing. Kill the music and
run the test again this time with my speakers way up and there were the beeps
again. I manually did what the test was doing to make sure it wasn't something weird
from the test library and confirmed that it was indeed a bug in VS.

At that point I freaked out a lil
bit since it was a Friday night and most everyone had left already plus we were
locking down on code changes but I did find someone still around to debug the
issue with and fix it. So what was the problem? The test was simply opening an
empty file and typing this:

 int foo()
{
     int a = 0;

And on the first letter of each "word"
IntelliSense was trying to kick in and failed (the failure was expected) so it
beeped. The bug was that on scenarios like the one I was testing it shouldn't
have even tried to kick in. Now you can imagine how annoying this would be on a
real production environment if your computer was beeping for the first letter
of each thing you typed...

Even though I noticed it while running a test there
would have been absolutely no way we could have caught this through our
automated tests since we don't test for sound. Lesson here never forgo doing
manual testing because you have good coverage on your automated tests. At the
very least sit down every once in a while and watch your tests run and make
sure no "bad" behavior that's irrelevant to the test purpose (but important to
your user) is going on.