Maturity model for test organization
One advantage of changing job as frequent as I did, is the privilege of getting hands-on experience of different team environment. I had the opportunity of being on 3 different test teams, each with very distinct characteristics and each team is in very different stage in terms of maturity. Just like a person growing up, there is different stage in life – from a child to adulthood. The same can be said for any organization, including a test organization.
Before going into detail of what levels there are, I would like to recall my past (and current) experience with each of the team. I’m going to try to describe each team as best as I could, and then we all can see what level each team is in. In no particular order, they are:
Team A
Under this test organization, there is a well-defined test manager with sub-teams, each with its own lead. There is formal test process established, and is well accepted by development teams. Test cases are organized in a systematic way. Standard test execution method is done manually with some test automation. Tester’s level of expertise is low to average, and they largely still depend on developer for help (meaning: tester still need a bit hand-holding from dev).
Team B
Similar to Team A in terms of organization structure – there’s a distinguished test manager, with test leads managing different sub-teams. 80-90% of test cases are automated, and development process is very test-driven. There are many senior testers on the team who establish concrete test framework. The framework itself is being utilized heavily by the rest of the team for test automation. Coding standard, code checkin, and code review standards have been established specific to the test teams and is currently in use. A number of test metrics is being tracked for each release.
Team C
Majority of the team are junior testers – most of whom are entry-level. Tools used for keeping track of test cases, bugs are very crude and homemade (e.g. Excel). Test team are heavily taxed from insufficient resource. Products are being released with known bugs. Dev and PM team frequently perform test estimates without test consultation. Test execution are done manually and almost in ad-hoc manner.
Looking at these 3 scenarios, it’s pretty clear that Team B is the most advanced, and Team C is still in its infant stage. Before diving into which team sits where, I would like to recap some well-known models which I'll be referencing as the base.
I have stumbled upon quite an excellent book “Just Enough Software Test Automation?” by Daniel J. Mosley and Bruce A. Posey. An article which abstract a few chapters from the book can be found online here. In one of the chapter, the author talks about the “Four-level maturity model for automated software testing" which is very interesting – Accidental automation, Beginning automation, Intentional automation, Advanced automation.
This is also very closely tied with the infamous Capability Mature Model (CMM) established by SEI at CMU. In case you forgot (I always did), here is the list again -- Initial, Repeatable, Defined, Managed, Optimizing
I find CMM tend to be overly generalized for a test organization since it needs to apply to any software organization, and the “four-level maturity model” is just geared towards test automation. So I basically dissect the two models and come up with my own version specific for testing organization. Here’s my shot at it:
Ad-hoc
Testing has just been introduced in an attempt to stabilize the development process. Test organization is still in its very early stage -- there may or may not be dedicated testers on the team. Developers mainly perform the bulk of testing. Formal test process is yet to be established. Test organization may still go through trial-and-error phases in order to find what work and don’t work under the incumbent system. Testcases and bugs are not consistently managed. Most tests are executed manually with very little test automation is implemented. If test automation is developed, it is done so in an attempt to save time and free up resources.
Predictable
Test process is now established and is repeatable from one release to the next. A solid test organization is fully established, with ranks of testers. Upper management understands the benefit of testing and ensures that test is involved with the whole development cycle. Test cases are clearly documented and regularly maintained. Products are released with high confidence and typically no major issues are found after released. Testing process as a whole becomes predictable. Bulk of test execution is still being done manually with some automation.
Standardized
Test organization is well equipped with well-established test process, tools, frameworks, and highly capable testers. Different levels of testing methodologies are routinely performed on upcoming products. There are senior management in test organization itself, with well-defined goals, test strategy, and measurement of success. Test teams actively play a crucial role in determining the whole software development, and routinely set standard criteria of software release. Test automation is the standard for test execution.
Progressive
The final level in the model. The focus of the test org is not simply to maintain a very high quality bar, but how to increase that level further. Another major focus shift is from finding defects to prevention of defects. Utilizing advanced testing techniques – model based testing, scenario based, fuzzy testing, relevance testing, etc. Both the dev team and test team are equally at the advanced level (i.e. dev extensively unit test their own code prior to dropping the code to test). Team repeatedly make full use of test metrics, and progress from past experience.
So, there you have it – my very rough version of maturity model for a test organization. I’m sure there are many holes in my theory, but I love to hear any opinion and learn from that.
Before we are done, let’s take a very quick recap of three teams which were described early on in the post and let’s rank them. Hopefully, it should be quite obvious now that which team fits where. Here’s what I believe is their model:
Team A – Predicatble
Team B – Standardized
Team C – Ad-hoc
Comments
Anonymous
March 20, 2007
One advantage of changing job as frequent as I did, is the privilege of getting hands-on experience ofAnonymous
November 22, 2008
please provide ur personal email id