Software Statistics Engineer?

I recently read an (internal, so I can't give a link, sorry) article by a developer pundit. In the article he opines that while the current practices of unit testing, etc. are having a definite impact on code quality, eventually we will reach a point where it becomes much harder to find bugs, at which point testing becomes a lot harder.

He sees Test as having three responsibilities:

• Finding bugs that devs miss.
• Running in the customer's environment.
• Analyzing quality metrics.

(Note that this is his definition, not mine, so don't yell at me if you think it's all wrong. I'm not entirely happy with it either.)

Clearly the first of these becomes a lot harder as the code gets cleaner. My team is looking hard for new metrics to judge the quality of our code and our testing because metrics such as test case count and number of bugs found just aren't interesting anymore. (Yes, I know they weren't really ever interesting. Humor me here.) If you test your specs rather than waiting for code to be delivered before you say "Hey -- what exactly should happen when I twist this widget", and if your developers thoroughly test the code that they write and so find most of their bugs before ever checking in, then naturally there will be a lot fewer bugs for Test to find. This is A Good Thing.

The second role isn't affected at all. Although I disagree this is just Test's responsibility. Just like the entire feature team is responsible for ensuring the quality of their feature, the entire team is responsible for knowing how the customer will use the feature, which includes understanding what the customer environment is and running within that environment. Dev is not doing enough testing if they aren't running in the customer environment at least some of the time. I do however agree that having more time to focus on interfeature behavior because Dev lets fewer bugs through is (again) A Good Thing.

Which brings us to what this pundit sees as Test's third responsibility: analyzing quality metrics. He believes that just following disciplined development practices will only take one so far, and that breaking through to the next level of development efficiency will require devs to collect data throughout the process and use that data to accurately determine when all the bugs have been eliminated. He seems to think that we testers are "salivating" at the prospect of having all this data to fold, spindle, and mutilate into (I guess) oodles of test cases.

At first I had a violent reaction to this. "I'm a tester, Jim, not a statistician!" Subsequent readings have mollified me somewhat; he's not suggesting that the *only* thing tomorrow's testers will do is crunch numbers, just that we will be doing more testing-once-removed (as it were), sifting and sorting all this information to spot trends, identify weak areas, and find ways to make life better for us and our team and our product. I don't have any ambitions around becoming a statistics professor, but I certainly wouldn't mind more data to help guide my testing.

But then he goes and says "[T]est becomes quality assurance". No no no no no! I'm a tester, Jim, not a QA engineer! Quality is the entire team's responsibility.

[Thinking calming thoughts....]

Oh, I get it: he thinks that quality assurance is going to become so important that we will have an entire team dedicated to pursuing it.

I agree, I guess. It's just that I think we already have that team: the entire product team. We have a lot to learn about en/assuring quality, certainly, but we are the ones who must learn it. Doing anything else is just passing the buck.

[Postscript: One of the cool things about Microsoft is that anyone can fire off an email to anyone else -- regardless of how far apart the two people are in the org chart -- and be likely to get a reply. Case in point: I sent this post to the abovementioned pundit asking for his comments on my comments on his comments. He replied that "the tester responsibilities I listed were never meant to be exclusively for testers" but rather that the entire feature team (including even y'all who sign up to be beta testers!) should be doing these things. He focused on testers because he believes somebody needs to be the point person for doing this and he thinks Test is the logical group to do that.

So maybe we aren't as far apart in our beliefs as I thought. I have an ally!  <g/>]

*** Comments, questions, feedback? Want a fun job on a great team? Send two coding samples and an explanation of why you chose them, and of course your resume, to me at michhu at microsoft dot com. I need a senior tester and my team needs program managers. Great coding skills required for all positions.

Comments

• Anonymous
  November 17, 2004
  Looks like the role of the future testers would be very limited. Current software development methodology emphasis alot on Dev to act as a tester. Will this means there will be vey few jobs for software testers in future. XP suggest that the customer people should write the acceptence test and dev should implement those as well as unit test. Agile process also emphasis on developer writing a test case. Does this means that testers would be eliminated. If not what their role would be?
• Anonymous
  November 17, 2004
  There is no shortage of people who think others should collect and calculate stats.
• Anonymous
  November 17, 2004
  The role of a tester on an agile project is a topic of great debate right now. I don't have any links handy but Googling for it should find bunches of opinions. Personally I'm glad to see the work Test is expected to do is finally approaching what I think our job should be: combining a great depth of understanding of the customer with a great depth of understanding of the application as a whole to do testing the app as a whole in a holistic way.
  
  Dev can do this to some extent, but they tend to be so deep into the innards of individual features that it's really hard for them to take a wider view. Test has been forced to live at this level as well because of the sheer number of bugs down there, but as Dev gets better at testing their own code we can widen our focus and spend more of our time taking that wider view.
  
  Testers will never be eliminated. Dev is never going to have the time to think about testing that we do. We can provide a huge service to Dev by helping them do better testing. We can provide a huge service to our customers by becoming an expert them and using that deep knowledge to drive our testing and to make our product truly useful to them. We can provide a huge service to ourselves by continually dreaming up better ways to do the testing we already know we need to do and figure out what testing we are missing.
• Anonymous
  November 17, 2004
  I've tested on a number of agile projects, and I don't see testers not being needed any time soon. It can be very different than testing on a more traditional waterfall kind of project, and I end up doing different tasks. It can be harder to find bugs, but I enjoy being able to try different things.
  
  One thing to note is that the testing described in XP and other agile practices can be very different from what conventional testers think of as "testing". When developers are using TDD for example, the tests they are writing are more like examples that guide their work. Later on when they refactor, the tests are a kind of safety net that allows them to improve code with confidence. This is not what conventional testers do when they test.
  
  Customers find bugs, but this isn't their area of expertise. I've often worked as a testing coach with customers helping them develop tests, or running through acceptance tests with them. There can be large gaps between acceptance tests and the automated unit tests developed with xUnit tools during development.
  
  This is where a tester can fit on an agile project (among other places) - helping to identify these gaps. Testers are good at it, and I've yet to see a project yet that didn't benefit from the right kind of brain-engaged testing that tester folks can add to a project.
  
  Some agile projects do very well without conventional testers, and I've seen value added by having them. Ultimately, the question of whether there should be testers on agile projects will be answered by the customer. If they want testers on the team, they will need testers who know how to operate in that environment.
• Anonymous
  November 18, 2004
  I agree with you that the role of a tester will be decided by the customer . Recently I worked on a XP project where manager prefer not to have the testers on the team. They have four customer support people who wrote acceptence test and also have the responsibility to interect with the customer. Devloper wrote the acceptance test using xUnit and test frameworks and one person is given the responsibility to run the regression suite. Since the customer people have clout and all the decision making power it basically tilt the balance in the team to their favour.Tech people (and test people) felt insecure as most of the decisions were routed through customer understanding people.
  I think if the customer people develop the ability to write the test cases ( using scripting technique, which now a days so easy to learn) why a team would need a dedicated tester and for what purpose. Further XP and other new methodology is shifting the balance more toward customer understanding people, which is good for the business but is it good for developing a good product (and to keep tester employed)
• Anonymous
  November 18, 2004
  As Jonathan says above, nobody is ever going to be as good at dreaming up good test case as a professional tester. (Sure, genius testers may show up occasionally on the customer team or other disciplines, but this will be few and far between.) Further, as we train our customers and developers to get good at one type of testing, there will always be a higher level for us to work to.
  
  Understanding the customer has always been part of being a good tester -- we just haven't consciously realized that in the past. Understanding the customer is the only way to build the product that will actually solve your customers' problems. If no one buys the product the product is going to die regardless how perfect it is.
  
  So yes, understanding the customer is good for developing the product and for keeping testers employed.