Testing For Developers Illustrated
I have received numerous requests for examples of how to use my Testing For Developers checklist. So here y'all go!
First off, and I know I didn't put this on the checklist, but peruse your way through James Whittaker's How To Break Software. This is a great read even for expert testers; for newbies it's a must have.
Second off, just to be clear, this is a small set of examples for each point, not by any means an exhaustive hit-each-of-these-cases-and-you're-covered list.
Third off, the examples:
Customize This List. Do I really need to explain this further???
Use Your Tester. Basically that's what you're doing by reading this post.
Focus On Your Customer. First, figure out who your customers are. Note the plural - guaranteed you have more than one. Think about: whoever you first thought of is doing their work for, your CEO, your tech support group, journalists reviewing your product, .... Next think about how each of those customers will use your product. Then brainstorm ideas regarding what would be showstopper bugs for that use case. And finally bang on your app looking for such bugs.
Test Around Your Change. Say you're fixing a bug where undoing a font change didn't actually undo the change because the big switch statement in the undo handler was missing a case. Look through the switch statement to see if you can spot other missing cases. The old code path probably hit the default case - does that do anything the new code path doesn't? Try undoing lots of other types of changes to make sure they still work. (Ideally, run the entire undo test suite!) Try redoing font changes as well as other modifications.
Use Code Coverage. Look through the results of a code coverage run and find a block of code that was not hit. What does that code do? What kinds of tests does that mean you're missing? For example, if the undo function hasn't been hit, clearly you haven't run any undo tests! Or you have but they aren't hitting the undo function for some reason - like maybe you have two of them. Or maybe the undo tests are broken. Or maybe the undo tests aren't actually doing anything.
Consider Testability. How are you going to get visibility into what your code is doing? How are you going to control what your code does? To keep beating on the undo case, you're going to need some way to identify the application state before and after the undo. You're going to need a way to execute undo. You're going to need a way to set up the starting state in the first place. You may need to do all this not just for the outermost function but also for every helper method it calls. Can you execute that eighth-nested helper function on its own to verify it's working correctly? Do you really need to?
Reset to default values after testing other values. After every test you run, hit that reset button or start a new document or save-close-reopen the document and verify that whatever is supposed to be defaulted actually is.
Look for hard coded data. Install your application to something other than the C drive. Run your application under a limited user access account. Run your app from a read-only folder. Save your document to a network share. Run your application on a machine that doesn't have a temp folder or %TEMP% environment variable.
Run under different locales and language packs. Switch your locale to German. Or Russian. Or Turkish. Or Hebrew. Or Japanese. Change your system currency symbol to "abc". Change your date format to "hi <year> there <day> bugs <month number>!"
Run under different accessibility schemes. Turn on large fonts. Turn on high contrast mode. Change your system colors to be obscenely garish. Run a screen reader. Run a screen magnifier. Use a puff tube instead of a mouse.
Save/Close/Reopen after any edit; Undo, Redo after any edit. Do it. You'll find bugs.
*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great coding skills required.
Comments
- Anonymous
March 01, 2006
I'd like to throw a suggestion up for one more. It's one that I like to pay particular attention to, Testing your errors. No matter how much testing you do, your software will break at some point in time, and when it does your error handling should be graceful, cleanly exiting the application and leaving the underlying system in a usable state. Too often I found bugs in the error handling routines that would cascade a failure to the rest of the system causing a need for a complete system restart. - Anonymous
March 01, 2006
Honestly speaking, I find that you site is perfect the way it is. However, perhaps a navigational bar (side frame) would be beneficial for users that are interested in reading particular blogs, rather than having to read from the beginning to the end, as I have. Other than that, your blog is definitely the central source for any SDET potential for MS. You have covered everything.
I was offered two positions at MS. Honestly speaking, your site was a major contributor to how well I did. I am really interested in meeting you once I join the MS team. Hopefully, I'll get the chance one day. - Anonymous
March 02, 2006
I'll look into what I can do about adding a navigation bar.
I'm glad you find my posts useful! Definitely look me up the next time you're here! - Anonymous
March 03, 2006
Just in case you might be interested. This is what I used for my navigational bar. The entries within the tree can actually be dynamic and linked to a database (Oracle9i/Ms. Access); but a little modification in the source code is required.
http://www.treemenu.net/