How do you ship software?

I surely don't need to tell you that we've been working on Whidbey for a while now. You probably wish we would hurry up & ship the thing. Me, too.

However, we still have some bugs left to fix. Most were found by our QA, but many came from Ladybug. Thanks for all of those.

We also got a whole lot of good suggestions via Ladybug. While we'd like to implement many of them, doing so is counter to the goal of shipping this product. So most of the ones I see, I resolve as Postponed. You get a comment from me saying "thanks, but no thanks". It'm bummed to write these comments, because I really want to get customers more involved with how we create our software. Your suggestions will be considered next time, though. 

One thing that always happens is that you look at some feature and realize it has some serious problems. You need to rewrite a peice of it to really fix it right. It's really hard to say no on these, but the time has come. We gotta shut down.

One of the techniques we use is "staged goals". Suppose a team has, say, 100 bugs per dev, and we intend to hit 0 bugs in 10 weeks. To do that, we need to drop by 10 bugs / dev / week. So say we'll hit 70/40/10 bugs per dev, every 3 weeks. It's a good way to make sure you're on track.

Trouble is, we're not on track. We still have too many bugs to fix in the time that's available. We don't want to slip the product, but we do want deliver high quality.

Over the last week I've been combing through the bugs on my team. I read each one of them. I verify that they still exist in the latest build. 

I also look for bugs that we can punt. (By "punt" I mean not fix in Whidbey.). Some get resolved "Postponed", which means we'll look at them again next time. One of the problems with Postponed is that there are some bugs we postponed last time (and the time before that). If we are postponing it repeatedly, we'll probably never fix it, and should resolve it "Won't Fix". That saves us the time spent reviewing it every product cycle.

There's another class of bugs I like to resolve as "Won't Fix". They're real bugs in the product, but it's not clear whether customers will ever run in to them, or care. So we Won't Fix, and assume that customers & dogfooders will tell us if they are annoyed. If it never comes up again, then we don't have to fix it.

We're also doing something usual. Sometimes we look at a bug and think "I'd like to fix this, but I wouldn't hold up the product for it." We're marking these bugs as "puntable". If the ZBB date slips, or we clear out our bugs faster than expected, or magic happens, then we'll fix these. But if things continue the way they're going now, we will resolve them before the ZBB date as Postponed.

As you can imagine, reading through every bug on my team is time consuming. There's some other stuff going on, too, hence not much time for blogging.

Comments

• Anonymous
  August 30, 2004
  Perhaps Microsoft should adopt an incremental build approach. You release your product at a certain point in time, but thereafter continue to release minor bug fixes and functionality improvements as patches (it would be nice if the products were auto-updating too). I could be wrong, but I recall few such updates been made available for Visual Studio 2003.
  
  Or institute an early access program such as that used by JetBrains for their 'Idea' Java IDE (http://www.intellij.net/), which lets early adopters get their hands on pre-alpha versions of new products ahead of time, with new iterations released weekly.
  
  In the Internet age, with unit tests and automated builds, is there really any need to force your customers to wait months or years between releases?
• Anonymous
  August 30, 2004
  Phil, Microsoft has already done something like that for Whidbey. There have been three pre-alpha (one at last Fall's PDC, one in March, and another in May), along with the Beta 1 build from last month. While customers may not have had a chance to make many suggestions that made it into the product, that may only be because the process of releasing pre-alpha builds to the public occurred to late in the cycle.
• Anonymous
  August 30, 2004
  Andy - I didn't know about the pre-alpha releases of Whidbey, but if they were released too late to enable any of their users' feedback to make it into the product then you've made my point for me. JetBrains start releasing versions of Idea as soon as there's something remotely usable there. Yes, the early releases are buggy, yes they're often unstable, but by releasing them in this way not only does users' feedback get incorporated into the final product, but the product also gets extensive testing by real users. I wish Microsoft would do the same thing: as an IDE, Visual Studio is literally years behind Idea and it seems to take a painfully long time for long-discussed improvements to appear in the product.
• Anonymous
  August 30, 2004
  Back when I was a test lead at Microsoft, I'd often barter for bugs. "Look, I really need this bug fixed, so I'll trade you these bugs for these." Needless to say, triage meetings were fun...
• Anonymous
  August 30, 2004
  Ideally I'd like to somehow explicitly communicate the trade-off decision such that even bug reporters are thinking about it.
  
  Could you show some sort of chart/graph/picture that shows how much capacity is available until the scheduled release date and the corresponding number of bug fixes that can therefore be selected. I suspect it's more an expectation thing than anything else.
  
  A comment on IDEA, since Eclipse does something similar though slower. Their interim releases, though buggy, are still remarkably usable where nightly Eclipse builds tend not to be. So there's probably something to say in terms of development practices as well.
• Anonymous
  June 12, 2009
  PingBack from http://cellulitecreamsite.info/story.php?id=7191