Hiring Great Testers - Tester Roles
In my mind, there are basically three roles on a test team. These three roles are: developers, scripters, and those who execute the test cases (runtime testers). In reality there is a spectrum of capabilities on any team but I think most roles will be closely related to one of these three roles.
Test Developers are the heart of a modern test team. There was a day when you could get away with hiring a few people to just use the product and call that a test team. This is no longer the case. Products are becoming more complex. The lifespan of products is increasing. More products are being created for developers instead of end users. These have no UI to interact with so simple exploratory testing is insufficient. To test complex products, especially over an extended lifespan, the only viable solution is test automation. When the product is an API instead of a user interface, testing it requires programming.
Test developers are programmers who happen to work on a test team. It is their job to write software which executes other software and verifies the results. The test developers need to be at least as skilled as the developers of the product. They need to know whatever language their product is written in. Their job is often as difficult or perhaps even more difficult than writing the product. Sometimes the job can be done in a simple fashion where each function is just called with various parameters and a simple method of pass/fail is monitored. This could be a success result or some notification from the program. A better test developer will not rely on a program to tell him if the right things happened. Instead, he will monitor the actual behavior. For a simple function this might just be the return result but more often than not, it requires monitoring the output of the program and making a complex decision based on it. Test developers are also responsible for creating the test harnesses and systems for their teams.
Scripters are those who have some programming ability but are not as skilled as a test developer. They will usually know a programming language but on that is less complex than the language the product is written in. Scripters may know Visual Basic, Perl, Javascript, or even C#. More important than which language they know, what distinguishes them from a test developer is that their understanding of programming is less advanced. They will understand the language syntax but their understanding of algorithms and data structures is likely less substantial.
Scripters will often play a role where they spend a lot of time living the product. They will use their programming skills to script the product (if it has such a feature) or to utilize tools to drive the product. The tools could be UI toolkits like Visual Test or something written by test developers. Because scripters have an understanding of programming they will be able to have a deep understanding of the product. They will be able to quickly determine the root cause of a failure and see relationships between seemingly unrelated bugs. Scripters play an important role on any team. They often handle the bulk of the test triage efforts and closely monitor the needs of the users of the product.
In some teams the responsibility for setting up machines, installing builds, and executing test cases falls to a group I'll call Runtime Testers. This role involves no programming. Runtime testers usually don't have any significant programming knowledge and if they do, they don't often get a chance to utilize it. They play a fundamental role, however. Because they are the ones running all of the tests, they often have the best knowledge of the state of the product. If you want to know what really does and does not work, ask one an execution specialist.
Sometimes this role is relegated to merely clicking buttons. They run an automated test and log some errors. This is a waste. They should be tasked with understanding the product and playing with it. As I've stated many times in the past, there is a critical need for manual testing on most project. These are the people best equipped to do that. To get maximum value out of runtime testers, they should be encouraged to live the product. To become experts in the functionality the way a customer might use it. In doing so they can become advocates for the customer and find the usability issues and corner cases that automation cannnot easily find.
Each of these three roles is critical on a test team. If you build a team with only one or two of the roles, you'll be missing something fundamental. Each project will require a different mix of people. Some need more test developers, others more runtime testers. The importance of these roles comes into play not only when you task your team but also when you hire. If someone is to be a runtime tester, hiring a CS grad may not be the best idea. They will likely be bored and quickly move on. Hiring a self-educated computer geek may be a better fit for the role. Likewise, if you need test developers, hiring someone who only knows perl is probably not a good fit. Instead, hire someone who is truly a developer. My rule of thumb is that if they couldn't cut it as a dev, don't hire them as a test dev.
Comments
Anonymous
January 16, 2007
I'm going to be doing a series not on testing but on the people that carry it out. This will be a postAnonymous
January 16, 2007
This is informative blog entry. I liked it. It describes different roles which are complementary to each other. ~NiteenAnonymous
January 16, 2007
The comment has been removedAnonymous
January 16, 2007
Surely, This blog information will help us to define the dedicated , long term Automation team strategy. It will help us to build/developed an Ideal Automation Testers Team.Anonymous
January 17, 2007
The comment has been removedAnonymous
January 17, 2007
Thanks Elisabeth. Are there any notes or papers or anything that were generated at this workshop that I might be able to access? The subject of what it takes to make a truly good test developer is one I think about a lot and I'm always looking for more input.Anonymous
January 17, 2007
Steve asked, "Are there any notes or papers or anything that were generated at this workshop that I might be able to access?" You'll find all the publicly available stuff here: http://awta.wikispaces.com/ See http://awta.wikispaces.com/Role+of+ToolSmith for some details on the tester-developer role...though not all the conversations were captured, so that's just a subset. Perhaps the most significant thing (for me) that may come out of AWTA is... ...another conference. Chris McMahon and I have started talking about what it would take to pull together a small conference for developer-tester-tester-developers focused specifically on programming interesting tests. Not on tools. Certainly not on commercial tools. But on good programming practices/tips/techniques for tests. (Our motto: "Show me the code!") We'll see what happens. We're just noodling around with the idea for now...though if we get a bunch of emails along the lines of "Yes! Do it!" we might be persuaded to stop noodling and start planning...Anonymous
January 17, 2007
As a programmer turned tester I enjoyed the 3 articles you linked and really looking forward to reading what else you have to say Sadly where I work we're not in a position to hire great testers - or any more testers full stop - so any advice on getting the testers we do have to up their game would be useful ( and Elisabeth - stop noodling and Yes, Do it !!!)Anonymous
January 17, 2007
Phil said, "any advice on getting the testers we do have to up their game" What do you wish they knew how to do really well? BTW, Brian Marick's book "Everyday Scripting with Ruby: For Teams, Testers, and You" is due out any day now. I read a pre-release copy. Great stuff. http://www.amazon.com/exec/obidos/ASIN/0977616614 ElisabethAnonymous
January 17, 2007
I forgot to say... good post. It's raised a subject close to my heart. More and more testers are finding that in order to compete, they need to develop their coding skills... Hopefully, your post will help spark even more discussion on the subject. Competing aside, the demand for tester-developers is growing simply due to necessity. Personally, I think it's a good thing since it is opening up new career paths for many. All the best AntonyAnonymous
January 18, 2007
Thanks for an interesting post Steve. I generally work between all three roles you identified depending on what my project requires at any given time. When working as a Test developer I find that I make much more effort to use an automated unit tests and automated build/test approach than many of the full time developers I work with. The majority of developers I work with do not use automated unit tests or automated build tests. GlennAnonymous
January 18, 2007
The comment has been removedAnonymous
January 24, 2007
I am unconvinced that all roles must be filled by a test team; the context of the project is important. I don't think a compiler back end test team has a lot of use for a runtime tester as you've described them.Anonymous
January 24, 2007
A compiler back end test team probably still has an automated suite of tests that have to be run every day. The person responsible for making sure that they all run, that the machines are in the right shape, etc. is the person I see in the runtime role on that team. You are right though. It's not a huge need and it might even be small enough that another person on the team could absorb it. This falls under "Each project will require a different mix of people."