Whidbey Beta 2 Tell Mode
Two weeks ago, we entered a phase of the Beta 2 product cycle we call “Tell Mode”. This is the point which marks the transition to the end game for Beta 2. Here’s what has happened so far, and an idea of what’s to come for Beta 2.
- ZBB : 11/19/2004
- Security Push : 11/15/2004 through 01/14/2005 (rest assured, these are not the only weeks we thought about Security in Whidbey! We spent this time doing concentrated efforts and implementing new processes which are now a part of our daily activities. The push let us take Security to the next level.)
- Tell Mode: 01/17/2004 through 02/04/2005
- Ask Mode: 02/07/2005 … until we meet our release criteria which defines all of the various quality measures for Beta 2.
Tell Mode is when the Division begins to meet daily. Every team reports on the number of bugs fixed the previous day, with those fixes to be available in the current day’s build. The idea behind Tell Mode is to start the shut down process. Teams move into a centralized triage process, where every bug that comes into the product team is scrutinized as to whether or not is should be fixed for Beta, moved to the next milestone, or punted from the project completely. Over time, we start to look at samplings of bug fixes and begin to set the bar, ensuring that teams are fixing the right set of bugs and that we're positively affecting the quality of Beta 2. This additional process slows the churn on the product because the more fixes we take, the more chances of regressions or build breaks. Our goal is to slow the number of check-ins, stabilize, take the necessary fixes to meet our release quality criteria, give the product a little time to “bake”, and then ship it.
Tell Mode is scheduled to last three weeks, through 02/04. We expect to then move into Ask Mode, where every bug that we want to be fixed has to be approved by our centralized ship room. The first question we ask on every one of these bugs is “what is the customer scenario?” Every fix needs to have a positive, tangible, and critical customer impact or else we don’t take it in Beta 2 at that point. Once we are in Ask Mode, we’ll do another test pass and we’ll finish driving up our quality criteria such as our Stress pass rates.
So far, we’re doing pretty well. We’re running a little hotter than I would like, but all the right things are happening. The next week will really give us a good indication on how well we’re tracking towards our goals.
Chris
Comments
- Anonymous
January 30, 2005
"This additional process slows the churn on the product because the more fixes we take, the more chances of regressions or build breaks. "
While I understand that a fix can break a build and that's not good either, a bug isn't good also. However, you don't know if a fix breaks a build, but you do know that a bug is there. I sometimes get the feeling for some bugs teams aren't willing to take the risk and fix it, which is VERY unfortunate because fixing a bug after release is almost impossible, as Microsoft releases service packs for .NET at most once a year. So if a bug is left in the code, a fix for that bug will not appear at least for a year. An ISV which runs into such a bug in their product is then left with no options.
So I'd beg you here: please take risks in fixing bugs: if it takes some more weeks to get it done, so be it, but fix as much bugs as possible. We, developers, are far better off with a product that is 1 month late and with no bugs than on time and with bugs. - Anonymous
January 30, 2005
The triage process is essential.. What Frans Bouma is describing will deliver the opposite of what he is expecting. To see this, compare the stability of later CTPs to the first Beta. The idea of the triage process is to fix only those bugs that are likely to have an imapact on the customer. - Anonymous
January 31, 2005
The comment has been removed - Anonymous
January 31, 2005
Interesting finds today - Anonymous
February 01, 2005
Thanks Chris, for your insightful elaboration. :) I was mixing beta with final release.