Milestones, Cuts, and what you aren't going to ship.
At Microsoft, there is a mantra repeated often by people who have been at the company a long time: "Shipping is a feature."
Like many things in life, what you don't say is often as or more important than what you do say, and there are positive sides to this and negative sides to this mantra. In line with my last post (9 months ago - I know), I wanted to go down into the rabbit hole and take you with me on a tour of some of the indicators that you are on a team which knows how to ship. There is a corollary to "shipping is a feature", and that is:
You don't know what you are shipping unless you know what you are not shipping.
When you build a product, you have a limited number of resources and a limited amount of time from the moment that you formulate your plan to the time you sign off on the gold master to give to marketing and sell to the customer. Building a product suite as large as Office is done much the same way that building a smaller product is done: in milestones. The next several paragraphs explain how the team which builds Office works together to ensure that the right stuff gets into the final product from this perspective and how cuts happen throughout the development cycle.
Microsoft has three major divisions of labor in the "development team". These divisions of labor have layers of management reporting to one overall manager (General Manager, PUM, and VP are typical titles), and each of the "disciplines" has peak amounts of work in the beginning ("Program Manager"), middle ("Developer"), and end ("Tester"). This is generally speaking - employees work all the time during the cycle doing different tasks. In general, the PM outlines what the product needs to do, the Developer implements the product/features, and the Tester makes work for the PM/Dev when the product doesn't work as expected. The PM is also tasked with balancing the work, prioritizing features in the schedule and bugs in the **bug database.
In the beginning of the milestone, Program Managers are busy planning, producing, refining, and reviewing specs. Developmers are busy estimating work/entering estimates on work items into the schedule databases, building prototypes to answer important design and implementation questions, fixing up bugs from previous milestones or product releases (service pack and hotfix work included), and reviewing specs. Testers are busy drawing up test plans, testing bug fixes from development, writing more automated tests for previous feature work, learning the latest technology and methods, etc. Before the coding part of the milestone starts, the team meets and cuts features based on the estimates, priorities, and plans to fit the schedule.
In the middle of the milestone, Program Managers are prioritizing bugs, investigating cross-team issues, and taking care of managing the work done by teams such as localization/globalization, documentation, design, etc. Developers are writing code and tracking progress in bug databases and schedule databases. Testers are busy testing, writing automation, and filing bugs against their developers, keeping developers honest about what they actually accomplished and tracking unfinished or unanticipated work (in the bug database).
In the end of the milestone, Program Managers are prioritizing/triaging/cutting work, making tradeoffs for what bugs need potentially destabilizing changes and making decisions about how to deal with bugs which are less likely to affect the customer - these become documented and fall into categories - won't fix (ever) or postponed (maybe to the next milestone if there is one, maybe to a service pack if customers actually run into the problem and there is no readily available workaround, maybe to a next release, etc). Developers are fixing the oldest and/or worst bugs. Testers are preventing developers from checking in fixes that don't meet increasingly "higher standards" and filing bugs directly to Program Managers instead of Developers (for prioritization and pre-triage). They are also running test pass after test pass - security, performace, legacy, accessibility, localization, stress testing, etc.
The final stage of the milestone/product cycle is a code freeze which includes a triage process where the source control for the project is forked and only the very worst bugs get fixed/checked into the deliverable code. Once this stage is entered, all non-milestone blocking bugs will be postponed to the next milestone of the project. The final milestone is like an extra end to a normal milestone except that "The quality bar is raised" until only real show stoppers get fixed - the kind which would cost the company millions to recall the gold master disk. These are what I like to call "billion dollar bugs" they might cost millions to postpone, but postponing them is delaying work that cost billions. When you have no "billion dollar bugs" for your team, then you are said to be "in escrow". In the final milestone, all bugs are either fixed, triaged/postponed, documented, or resolved won't fix. Eventually, when all teams have been in escrow for some given amount of time, management signs off that the quality of the product is good, and you ship it, throw a party, take a break, and begin working on another project (the best possible time to switch teams/jobs is just after you complete a project).
That's a rough outline of what happens in a normal milestone for Office or more generally over a release cycle for a smaller product. The last couple Windows versions of the Office suite (Office XP, Office 2003, Office 2007) all had about 2-3 years of development. Since typically we have something like 6 months in a milestone, we typically had something like 4-6 milestones. At the end of a milestone, we have a "deliverable" product. Between Major Milestones (MM1, MM2, MM3, etc) we have "dogfood builds" that are internally tested and tested by some customers with special relationships. In addition to major milestones, we have Beta testing Milestones, where the deliverable is tested by many customers outside the company (Beta 1, Beta 2, Beta 2TR, etc), and finally there are shipping and maintenance milestones (RC1, RTM, SP1, SP2, etc). The point of a milestone is that it has a "deliverable", be it a dogfood build, a beta build, or a shipping product or patched product (service pack). Each milestone ends with triage. The RTM milestone is special in that it generally begins in "team triage" and ends with multiple stages of triage (multiple stages of triage allow individual teams to synchronize their level of cuts/"quality bar").
Office is a very successful team at producing a product. All of the above background is important because in it I have littered the various ways which define "what you aren't going to ship". When customers feel frustrated with product issues, bugs, design of some feature, etc. They often write with feedback (via several mechanisms) about "Why can't I achieve X by doing Y with product Z?" The answer which is most likely the case and I always feel a little frustrated that I can't give you a direct example is: "We planned to deliver Y in release Z. It was cut with change # for bug # because of reasons R1, R2, R3, and R4 which are documented in the bug database. It is still visible to the team and management."
Remember what I wrote in the beginning of this post: "Shipping is a feature." We often cut your feature Y for shipping. When it comes right down to it, no other features are as important as shipping. That's the downside of what it takes (cutting) to ship a product like Office to about 500 Million paying customers. The upside is that you have a clearer goal - you know for each stage in the product development cycle that you are doing work in a methodical, targeted fashion, and at the end of the cycle there's on the order of more than a billion reasons why you will ship. Imagine for a moment that you have a team of people the size of Office - let's say for argument sake that the numbers are on the order of 5,000 employees on average costing the company $100,000 each per year for a 2 year development cycle. So assume you have a 2 * 5K * $100K = $1B cost to develop Office.The bottom line is that it costs over $1 Billion to develop a single release of Office (today). At the end of all that work, any bug which prevents you from shipping is a billion dollar bug. Teams which cut work earlier ship better features because they spend less time overall on the things that they won't ship and more time on the things that they will. By the time that they are in their last milestone, they have fewer billion dollar bugs. Very rarely does a billion dollar bug slip through the process (when it happens, it makes news). This takes me back to the top of this message:
You don't know what you are shipping until you know what you are not shipping.
Epilogue: It is painful to spend weeks and in some cases months or years of your time working on a feature to have it cut in whole or in part, but in the end, it is better for the customer to have your imperfect product in their hands than to still be working passionately on a product they will never get because it will never be perfect or bug free. Trust me when I say that developers feel customer pain. We probably feel your pain more than you realize - we had to suffer through cuts of some really cool features in order to ship the features that you have in time to make a release date. Key danger signs that your group is not going to ship is when you talk to your senior managers and find that they don't have the ability to say, "No, we're not doing super cool feature #. We spent # time investigating it, but couldn't get it to work right, and it is cut in favor of getting more basic and more important work that we just can't ship without to be high quality and stable." Managing a successful product means knowing how to make the painful cuts necessary to ship the right product.
** Disclaimer: A bug in the context of this post means any feature of a product which works poorly from the perspective of the person who reported it to the product development team or the person in the product development team who filed the bug in the bug database.
This posting is provided "AS IS" with no warranties, and confers no rights.
Comments
Anonymous
October 23, 2007
PingBack from http://www.soundpages.net/computers/?p=4115Anonymous
March 31, 2009
From elsewhere in the collective.Anonymous
March 31, 2009
Due to pressure from friends after being linked from today's post from Raymond Chen (his blog is great, go read it sometime), I have re-enabled comment posting.Anonymous
April 01, 2009
Wow, at first I thought this was going to be a dry article but I actually read it all the way through and found it very interesting. Thanks for taking the time to shed the light for all us customers out there who have been disappointed in a missing feature or a "why did they do it that way?". At least now I can see things from the other party's perspective.Anonymous
April 01, 2009
The comment has been removed