Characteristics of great Beta Testers
Things have been particularly busy here for the Encarta Test team as we roll out the Beta for our next version, Encarta 2005. This seems like a good time to talk about characteristics of a good Beta tester, and is something that I would like to elaborate more on in the coming weeks.
I would categorize great Beta testers into one of two groups.
1. Power User Testers. These are the beta testers who could certainly excel if they were to choose a full-time career in Beta testing. They're naturally driven by curiosity about new technology, and enjoy pushing the latest software to it's limits. These are the same testers that tend to be very active in the Beta discussion newsgroups, and enjoy the problem solving aspects of a interesting software bug. Testers in this category are the “power users“ that we all know who have an aptitude for learning new technology quickly.
2. Less-Technical testers. These are testers who don't care much technology for the sake of technology. Instead, they're focus is on applying technology when it's needed and useful. They've got some great suggestions about how to improve products for common people. Picture our Moms and Dads that didn't grow up using computers, but still find interesting uses for them. Some of the most compelling suggestions for feature improvements come out of this group.
As you might guess, most Test teams at Microsoft are stacked with Power Users. We can script and code our way through our most frequent tasks, and know every keyboard shortcut of our heavily used applications so we don't need to waste time grabbing the mouse. We share these traits with our Power User beta testers. While the “core“ of our Beta tests rely on the feedback and bugs entered by our technically savvy Power Users, some of the greatest feedback comes through “suggestion“ bugs entered by our Less-Technical testers. These are the people who help us see the other side of the coin- that not all of our users find it clear when we intersperse phrases like “GB” and “MB” within the same document.
Perhaps the greatest tester of all is one that is a Power User, but still maintains the ability to see software as it's used by less-technical people. I often have problems “remembering“ what it was like to first begin using computers, but I think I share that problem with most of the software industry. In the mean time, we'll be using some of the non-technical feedback from our Beta testers to help us remember.
-Greg
Comments
- Anonymous
April 20, 2004
The comment has been removed - Anonymous
April 21, 2004
Hi Nicole- I enjoyed your analogy quite a bit. Every time a user doesn't "get" how to use the software, it's a failure for our design. Power Users all realize that a key goes into the ignition... but what if the car itself is locked? Many of the Mom/Dad/Aunt users become confused and frustrated with the software (or car). When events happen unexpectedly, it can really throw our less-technical users into a loop.
There was an interesting post on jobsblog about a Microsoft Recruiter getting hired into a testing role years ago. The post is worth reading- take a look here. (Unfortunately, I agree with Gretchen and Zoe's conclusion that this is increasingly difficult to do at modern Microsoft).
http://blogs.msdn.com/jobsblog/archive/2004/03/23/95037.aspx - Anonymous
April 21, 2004
The analogy is also a great test, to see if someone understands 'Mom and Dad'. If someone does not understand them, how is he able to understand any demands customers make?
Programming is always about 'what the customer wants' and not what the programmer thinks what is best for them.
And yes, I know the post, this is how I got to know your blog :o) The question about the salt shaker is a great one; it gives me as a reader the possibility to think about getting out of the usual routines / boxes.
Nicole - Anonymous
April 21, 2004
The comment has been removed