“if Microsoft is so good at testing, why does your software suck?”

What a question! I only wish I could convey the way that question is normally asked. The tone of voice is either partially apologetic (because many people remember that I was a major ask-er of that same question long before I became an ask-ee) or it’s condescending to the point that I find myself smiling as I fantasize about the ask-er’s computer blue-screening right before that crucial save. (Ok, so I took an extra hit of the kool-aid today. It was lime and I like lime.)

After 27 months on the inside I have a few insights. The first few are, I readily concede, downright defensive. But as I’ve come to experience firsthand, true nonetheless. The last one though is really at the heart of the matter: that, talent notwithstanding, testers at Microsoft do have some work to do.

I’m not going down the obvious path: that testing isn’t responsible for quality and to direct the question to a developer/designer/architect instead. (I hate the phrase ‘you can’t test quality in,’ it’s a deflection of blame and as a tester, I take quality directly as my responsibility.)

But I am getting ahead of myself. I’ll take up that baton at the end of this post. Let’s begin with the defensive points:

1. Microsoft builds applications that are among the world’s most complex. No one is going to argue that Windows, SQL Server, Exchange and so forth aren’t complex and the fact that they are in such widespread use means that our biggest competitors are often our own prior versions. We end up doing what we call “brown field” development (as opposed to ‘green field’ or version 1 development) in that we are building on top of existing functionality. That means that testers have to deal with existing features, formats, protocols along with all the new functionality and integration scenarios that make it very difficult to build a big picture test plan that is actually do-able. Testing real end-to-end scenarios must share the stage with integration and compatibility tests. Legacy sucks and functionality is only part of it…as testers, we all know what is really making that field brown! Be careful where you step. Dealing with yesterday’s bugs keeps part of our attention away from today’s bugs.

(Aside: Have you heard that old CS creationist joke: “why did it take god only seven days to create the universe?” The answer: “No installed base.” There’s nothing to screw up, no existing users to piss off or prior functionality and crappy design decisions to tiptoe around. God got lucky, us…not so much.)

2. Our user-to-tester ratio sucks, leaving us hopelessly outnumbered. How many testers does it take to run the same number of test cases that the user base of, say, Microsoft Word can run in the first hour after it is released? The answer: far more than we have or could hire even if we could find enough qualified applicants. There are enough users to virtually ensure that every feature gets used in every way imaginable within the first hour (day, week, fortnight, month, pick any timescale you want and it’s still scary) after release. This is a lot of stress to put our testers under. It’s one thing to know you are testing software that is important. It’s quite another to know that your failure to do so well will be mercilessly exposed soon after release. Testing our software is hard, only the brave need apply.

3. On a related point, our installed base makes us a target. Our bugs affect so many people that they are newsworthy. There are a lot of people watching for us to fail. If David Beckham wears plaid with stripes to fetch his morning paper, it’s scandalous; if I wore my underpants on the outside of my jeans for a week few people would even notice (in their defense though, my fashion sense is obtuse enough that they could be readily forgiven for overlooking it). Becks is a successful man, but when it comes to the ‘bad with the good’ I’m betting he’s liking the good a whole lot more. You’re in good company David.

But none of that matters. We’ll take our installed base and our market position any day. No trades offered. But still, we always ready to improve. I think testers should step up and do a better job of testing quality in. That’s my fourth point.

4. Our testers don’t play a strong enough role in the design of our apps. We have this “problem” at Microsoft that we have a whole lot of wicked smart people. We have these creatures called Technical Fellows and Distinguished Engineers who have really big brains and use them to dream really big dreams. Then they take these big dreams of theirs and convince General Managers and VPs (in addition to being smart they are also articulate and passionate) that they should build this thing they dreamt about. Then another group of wicked smart people called Program Managers start designing the hell out of these dreams and Developers start developing the hell out of them and a few dozen geniuses later this thing has a life of its own and then someone asks ‘how are we going to test this thing’ and of course it’s A LITTLE LATE TO BE ASKING THAT QUESTION NOW ISN’T IT?

Smart people who dream big inspire me. Smart people who don’t understand testing and dream big scare the hell out of me. We need to do a better job of getting the word out. There’s another group of wicked smart people at Microsoft and we’re getting involved a wee bit late in the process. We’ve got things to say and contributions to make, not to mention posteriors to save. There’s a part of our job we aren’t doing as well as we should: pushing testing forward into the design and development process and educating the rest of the company on what quality means and how it is attained.

We can test quality in; we just have to start testing a lot sooner. That means that everyone from TF/DE through the entire pipeline needs to have test as part of their job. We have to show them how to do that. We have to educate these smart people about what quality means and take what we know about testing and apply it not only to just binaries/assemblies, but to designs, user stories, specs and every other artifact we generate. How can it be the case that what we know about quality doesn’t apply to these early stage artifacts? It does apply. We need to lead the way in applying it.

I think that ask-ers of the good-tester/crappy-software question would be surprised to learn exactly how we are doing this right now. Fortunately, you’ll get a chance because Tara Roth, one of the Directors of Test for Office is speaking at STAR West in November. Office has led the way in pushing testing forward and she’s enjoyed a spot as a leader of that effort. I think you’ll enjoy hearing what she has to say.

Comments

• Anonymous
  August 11, 2008
  The comment has been removed

• Anonymous
  August 12, 2008
  There is a bit of a well known qoute that testers cannot ensure quality - they can find the bugs but they can't make people implement a fix. I don't entirely subscribe to this view:  there are ways and means of pushing for fixes. However, the question I'm slowely heading towards is this "Of the bugs identified within Microsoft, what sort of fix ratio does Microsoft have?"  Do they attempt to fix everything?  Is the majority of things found in the field that joe public whing about already on the Microsoft defect management system? I think it's marginally more forgivable for an organisation to know about bugs and not have fixed them, than for the bugs to come as a complete surprise to the organisation.  Of course, I'm a tester so I would aire more towards this viewpoint.

• Anonymous
  August 19, 2008
  Check out James's recent post with the eye-catching title “if Microsoft is so good at testing, why does

• Anonymous
  October 01, 2008
  At project wrap-up, a surefire conclusion is that "more people should've been involved earlier".  Kinda like "more documentation was needed". But the more people you involve, the smaller the chance your project goes anywhere. :)  If you require testing involvement in an early stage, you'd probably stifle innovation.

• Anonymous
  October 04, 2008
  Excellent post James (as always (and yes I like parentheticals too (really! :-P ))). One further point I'd make is that geting the smart SDET's into the process sooner isn't enough. Getting the support organization folks in sooner can pay dividends. Someone in the MS third tier of support has been fixing, deploying, and reverse engineering these products for years - often for much longer than the dev's, testers, or PM's and through far more versions. And yet they are usually added to the mix right before the public beta. Far too late to have said 'that new feature as designed is unsupportable by us or customers and here's why'. But that's an old beef - I definitely think that the closer testers get to the design/spec phase, the better. I've known some darned smart testers and seen them catch some ridiculously insidious problems before they got out the door. That's the worst part of your job - the better you are, the less anyone knows how great you are. It's like all the goofs who think that Y2K was some made up non-event; they didn't see a tens of thousands of people preemptively fix the issues for 3 years. :)

• nedpyle

• Anonymous
  October 04, 2008
  The comment has been removed

• Anonymous
  October 05, 2008
  I liked your posting but I have a slightly different view about the 'testing quality in' perspective. Testing is a practice of damage control for trying to find the bugs that have been already designed and developed into the product. As we are greatly outnumbered by the ratios you mentioned, all we can do is focus on "customer tolerance limits" by targeting those defects with the highest impact to the user base first and work our way down the list. Certainly the solution is not for test to become a quality "police" but like you well said it, to make sure testing and all stakeholders get involved in the design phase so that our combined skills and customer understanding are used  to prevent those defects from being introduced in the first place.

• Anonymous
  October 05, 2008
  Добрый день. C момента публикации знаменитой статьи...

• Anonymous
  January 13, 2009
  I was triggered to read a blog post by James Whittaker, software architect at Microsoft, through an article

• Anonymous
  February 03, 2009
  [Nacsa Sándor, 2009. január 13. – február 3.]  A minőségbiztosítás kérdésköre szinte alig ismert

• Anonymous
  December 09, 2013
  The comment has been removed

• Anonymous
  December 09, 2013
  part 2 The same approach can be applied to word processing and several open source desktop publishing programs use it to good effect.  I use these in my work I do in isolation, but I cannot use them to collaborate with everyone else that is using Word. So, please, don't be defensive.  Just admit that MS products are not nearly as good as they should be and that  they are enjoying their market position and don't need to be all that great to be rich. I read somewhere else, may not be true, that MS farms out code writing to private contractors all over the world.  A common practice is to get the code to function anyway possible, hit compile, and then celebrate "yeah, I'm done and get to cash a check".  I am not a trained computer scientist, but I've had to learn the basics of programming for my work and I produce very impressive ecosystem simulations for research purposes.  Even I know that this is not the correct way to write programs that other will be using.  I've also read where people have reversed engineered some MS products to look at the code and they were shocked at how poorly written and inefficient the code was.

• Anonymous
  January 07, 2014
  The fact that your software is overly complex and full of features and bloatware no one would ever want or use is no excuse for it to have the same bugs for years and years spanning version after version.