Principles 5: End-to-End Development Process (for Large Projects)

Artikkeli
04/21/2011

Series Index

+++

This fifth post discusses and end-to-end development process for large software projects.

The only difference between large and small projects is that large projects require more “supporting infrastructure” (more processes, etc.) than small projects, due to the fundamentally imperfect communication between the team members. The “supporting infrastructure” is a tax we have to pay when we develop software. We can minimize it, but we can never really avoid it completely.

Larger projects require larger teams, which (due to accumulated communication imperfections) require larger process tax.

On one hand, we like short product cycles, because they get the product out in the market quickly; on another hand, the shorter the cycle, the higher the ratio between process tax and actual productive time. This is a classic case of an optimization problem, which can be defined as…

“Find the shortest possible project cycle, that keeps the process tax at a reasonably low level“.

Clearly, there’s a variety of other factors that we need to consider when we talk about project and product cycles. To list a few, we need to factor in the general product business model, pre-existing contractual obligations, specific calendar events such as conferences or trade shows, the competitive landscape, etc. For the purpose of this article we can assume that all of these additional factors have already been taken into consideration.

Single Project Rhythm

I am a big believer in having a single, transparent, well-socialized and strictly enforced project rhythm. All products that an organization delivers in my opinion should be aligned to this single rhythm. The rhythm (or cycle) should be short enough, so that it can accommodate all products in development. In this model, different products can take different number of cycles. At the end of every cycle, one or more products get released.

Having a single cycle in my opinion dramatically decreases the complexity (and ultimately the process tax) that a team has to pay. It helps create well-aligned, synchronized and cohesive organizations, reduces dramatically the release management costs of a product, practically eliminates context switching for the individual engineers, drives ownership deep into the organization, improves communication with supporting disciplines (docs, localization, support, sustained engineering), and so on.

Agility

In this model, agility is achieved by having an appropriately-sized cycle. For example, for a desktop OS the right cycle may be 6 months or 12 months. In contrast, a web site would probably need to have a release valve every 2 to 4 weeks. Agility doesn’t mean lack of planning – agility means a reasonably short cycle capable of getting high quality products out in a predictable, controlled fashion.

Process Summary

The process of developing a large software project can be organized in 4 major phases (see Fig. 1 below):

MQ
Pre-planning and Planning
1 or more Development Cycles
Stabilization and Release

(A) and (B) happen simultaneously. (A) is “Milestone for Quality” targeted at constructing a high-quality “software production factory”. This milestone gives the engineering organization a chance to optimize and simplify the tools and processes that it uses to product the product. (B) follows a predefined planning process. Planning is a whole other topic in itself. (C) consists of one or more close-looped development cycles.

Figure 1 Development Process Outline

Obviously this is just a general outline. The size of the different phases has to be determined by the development organization, based on deep understanding of what needs to happen in these phases and on the current state of the business and of the development factory. Some phases (specifically A and B) may shrink if there is no need for them.

Below I show an actual implementation of this general outline. Each small rectangle represents one week. Weeks in green are “stabilization weeks” (stabilization has different meaning for the different milestones – see tables below). Note again that the size of the actual milestones and phases that I show below are just a specific implementation of the general outline. You can have development cycles that are as short as a week, if you think that the ratio of dev time to process tax associated with managing every cycle is reasonable. You can also have longer cycles, although I personally wouldn’t recommend cycles that are significantly longer that 18 weeks or so.

Figure 2 Development Process - Detailed View

With these sizes we have…

About 18 weeks (4 months) of pre-planning and planning activities.
One or more dev cycles of 15 weeks (3.5 months)
Final stabilization of 10 weeks (2.5 months)

I.e. a release with 1 dev cycle takes about 9.5 months; a release with 2 dev cycles takes about 13 months, etc.

The number of the actual development cycles can vary and is a purely business decision. At the end of every development cycle, the organization delivers shippable functionality. When management deems that the available payload makes sense as a next major version of the product, they switch the team to the RTM (RTW) phase, and drive to an actual release.

Before we delve in MQ, it’s important to note that all of these phases are divided in “red weeks” and “green weeks” (a block denotes a week). “Red weeks” are weeks of active development, “green weeks” are weeks of stabilization. Obviously the actual activities that happen in these weeks depend on the phase of the project.

MQ

MQ is the Quality Milestone. The role of MQ is ensuring that we have a working high-quality “production factory” before we embark on producing the next version of the product.

Figure 3 MQ and Planning

The breakdown of MQ is as follows:

Figure 4 Breakdown of MQ

MQ is all-inclusive collaborative effort between the 3 disciplines. The majority of PM, Dev and Test will be working on MQ activities during MQ. A few PMs and Devs will be working on planning / prototyping in parallel. MQ is planned and rigorously project-managed. MQ has a set of exit criteria that the org needs to hit before exiting the milestone.

The Dev Cycle

All features are developed during the dev cycle, which looks as follows:

Figure 5 Dev Cycle

There is a single prioritized feature backlog that captures all features that are being worked on in the current development cycle or can be worked on in future cycles. The list of features that comprise the current development cycle is stable throughout the cycle i.e. Management does not change the list beyond M0 – they have to wait for the next M0.

Any feature that cannot be completed is “cut” until the new dev cycle. Features that take more than one dev cycle to complete will be split into smaller manageable “featurettes” can be completed within the constraints of a dev cycle. Several featurettes may comprise the final feature. At the start of the development new cycle the feature is re-evaluated again against the other features in the backlog.

The start of every new dev cycle is a clean slate. All bugs and features are automatically “cut” and need to be reevaluated and brought back (can be evaluated and brought back during)

IMPORTANT: The development cycle is a contained unit of delivery of added value. We can have as many development cycles as necessary to build a compelling value for the “big release”, but every one of them ends with high quality features that are ready to ship.

The breakdown of the development cycle is as follows:

Figure 6 Breakdown of a Dev Cycle

The M0 to M4 cycle can be repeated multiple times within a release. Every time in M0 (but only in M0), executive management gets to redefine the feature backlog.

Stabilization and Release

The final phase in the development process (disregarding the various post-release activities for now) is the “Stabilization and Release” phase or “RTM”.

Figure 7 Stabilization and Release (“RTM”)

This phase is about fixing bugs, integration (if necessary) and managing the final quality of the release. The breakdown is as follows:

Figure 8 Breakdown of Stabilization and Release

QFEs (Hotfixes)

Every large software project needs to have a mechanism to deal with hotfixes (also known as QFEs). The nature of these fixes requires reactive delivery (as opposed to the proactive approach described above). Teams typically maintain a separate source code branch (or branches) for QFEs and that is typically the same as the branch that contains the currently shipping version of the software.

Due to their reactive nature QFEs are delivered on demand with a release schedule that looks as follows:

Figure 9 QFE Release and RI Schedule

Typically, fixes introduced in the QFE branch must make their way into the Main product branch, which is done with a series of regular reverse integrations (or RIs) as shows above. QFEs also get packaged as larger packages and released as service packs (or SPs).

Every QFE is subjected to a root-cause analysis, following a simple standard template, to help the team avoid similar situations in the future.

Templates and Artifacts

Last but not least I wanted to provide a quick list with some of the fundamental documents that are required to bootstrap a development process for a larger product. Note that this list is nothing more than a reasonable starting point – at least according to me.

Figure 10 Key Templates and Artifacts

Conclusion

There are many ways to ship software products. Large products require more pampering and more scaffolding, at least until a critical mass of the team adopts a coherent product development culture. If you want to ship high quality products, you need to invest in rationalizing your product development cycle. The best way to do that is to plan, using a series of thought experiments (e.g. “imagine that we have achieved code complete, what are the steps from CC to release ready and how long are they realistically going to take”), and a guiding principle of lowest possible overhead costs and highest possible simplicity.

Comments

Anonymous
April 25, 2011
The comment has been removed
Anonymous
April 26, 2011
The comment has been removed
Anonymous
April 27, 2011
This is a great explanation which makes sense. This model must fit especially well to API design but I could imagine doing the same with any kind of user-facing front-end. You could create the UI and stub out all real events with messages boxes. The cost of doing variations would be dramatically lower. If you want any input on what to write about, I would appreciate a few comments on specifications. There is a lot of dubious advice around about this topic. Working inside of Microsoft seems very attractive because you get to learn to maximize your output and be really efficient. Amazing that such a big and old company can still get things done. Think of Oracle, SAP, ...
Anonymous
April 27, 2011
The comment has been removed

Jaa