Today is Patch Tuesday

Taking a departure from my promised post on C2R for a minute to celebrate the passing of another Patch Tuesday.

 

A term recognized across the IT industry is "Patch Tuesday."

I'm hopeful this post will remind folks about Patch Tuesday stuff they already know, and hopefully to ring the bell of "common sense." Unfortunately Patch Tuesday isn't universally known, and most people aren't aware of the background or rationale for our approach. It is my ongoing hope that people reading this content are either learning new things, benefitting from having the basics explained to them, or surrendering myths or misperceptions about our update businesses. In any case the history of Patch Tuesday is interesting, and explaining our approach may help. In summary, the question asked and answered is "how do you update hundreds of millions of endpoints every 30 days without creating massive disruption?" The answer: "Very carefully."

 

Vague metaphor: Some people see streaks of light. Others see two boats passing each other at night in a busy harbor reading each other's navigation lights and following some basic conventions.

 

Patch Tuesday History

Patch Tuesday emerged as part of a broader security initiative undertaken by the company in 2002-2003 as a result of an increase in the number of threats to the Windows environment. Where Microsoft had previously shipped updates on a weekly schedule, it was decided that a monthly schedule combined with advance notice for time to prepare would be a more tenable customer experience for installing updates. At that time all products moved to the Patch Tuesday schedule.

Concurrently, the Windows Update, Microsoft Update and Office Update methods were evolving. Originally released for Windows 95 as a web site, Windows Update allowed users to manually visit a web page, connect and download updates via the web. Eventually the concepts of critical updates and notifications were added to Windows 98 products, giving way to Automatic Updates added for Windows ME and beyond.

In 2005, Microsoft announced a change to Windows Update, introducing the transition to Microsoft Update, to include updates for other products. This began the deprecation and eventual demise of Office Update, leaving Microsoft Update as the primary channel for updating Microsoft Office.

Fast forward 8 years, where today we have hundreds of millions of regularly updating clients, and billions of downloads each year. SAP, Adobe and other software companies have adopted a similar update method, and Patch Tuesday is almost taken for granted. Patch Tuesday has been proven as an effective method of keeping desktops up to date. We have invested quite a bit in update management, beginning with technologies designed to facilitate easy updating, as well as policies and communication guidelines to make the updates more understandable.

Office and Patch Tuesday

Microsoft Update delivers updates to hundreds of millions of users, scaling to large installation numbers very quickly after publishing. The scale and speed of distribution of these updates necessitates a level of caution when deciding what to ship and when. If even a small %age of customers are affected, that can quickly scale to millions or tens of millions of people. We watch the uptake of an installation with a microscope. We carefully monitor installation successes, failures, their causes and we try to learn as much as we can about the environments in which they manifest.

The combination of what ships when can be a tricky area. Our goal is to go as fast as possible. When vulnerabilities or critical updates span platforms or products, however, an alignment of schedules an priorities is necessary to ensure we don't unnecessarily expose one product for the sake of patching another. In some core components, we must sim-ship our updates. For example our Mac products, IE, Office and SQL might need to ship an update on the same day. Keeping 4 or 5 major product lines on cadence with update releases can be daunting, given the multitude of branch, build, automation, development environments in use.

For Public Updates / Patch Tuesday updates, we typically offer only those updates that are most critical to our global customer base. When everyone is affected by an issue, such as a security vulnerability, we offer that update globally. Through the history of Office Updates, you can review the list of what we've shipped, and quickly determine that little falls outside the domain of critical or security. The "Critical" vs. "Important" designation is a criteria of MSRC documented on the Security TechCenter.

In other words, if we take the risk to offer it to everybody, it is an update which we feel strongly that should be applied as early as possible.

How Office365 and Services are Changing Public Updates and Patch Tuesday

An interesting area where we're seeing some change relate to who Office365 and other services that have heavy interaction with Office client applications. Services evolve, as data is understood from their usage, we learn more about how to better optimize client interactions with the services. For example, we encountered a handful of scenarios where sync to online folders wasn't always working as expected, or performed poorly in certain configurations. This update was a change we made for the benefit of service customers, and for the benefit of SharePoint customers syncing document libraries.

In many ways, this is "just another update," but in many ways it is a signal of the future of updating Office. We learned about these issues partly from customer reports or external reports, but performance and other aspects of service interaction are things we learn from telemetry. We are investing heavily to ensure that we are aware of any potential trouble before our customers are aware. And when we are discovering those things, we'll use the Public Update channel to distribute those updates. As our services grow, we want to ensure Office has a successful interaction and users are satisfied with the experience.

This is a departure (an addition) to our traditional response-oriented critical update strategy, and it is part of how Office begins to look more and more like a service. We are supplementing our common escalation channels (the usual mix of Microsoft Support, Communities, MSRC, Dr. Watson, etc.) with more direct line of sight into service health and customer experience. This addition of new sources of telemetry and incident information will increase our speed of issue resolution. The more we listen, the more we learn.