Bureaucracy or Professionalism?
I've been struggling to crystallize my thoughts about the numerous discussions, mostly leading back to our anonymous colleague minimsft, about the bureaucracy / stifling process at Microsoft. The posters have been mostly Microsoft veterans; perhaps it would be useful to offer a perspective on this from a relative newbie (albeit one who has spent 30 years in software development, at 10 different organizations in industry and academia).
I was a process weenie back in the days when I was a developer, so I was rather pleasantly surprised to discover that there is a very hardcore software engineering culture in the SQL Server group at Microsoft. I'm not sure how long that has been in place throughout the organization as a whole, but as an outsider and fairly dis-satisfied Windows consumer 5-10 years ago (and a rabid Windowphobe Unix and OS/2 geek before that) I had assumed that something like the barely organized chaos described in Microserfs was the norm. It's been obvious from the outside for some time that the engineering process must have been improved, but I had no idea how rigorous it is.
The gist of the pushback seems to be that the pendulum has swung too far, from chaos to stasis. The illustrative, possibly apocryphal example, is the one-line bug fix that takes a month of process to get checked in. In that same month, the frustrated hypothetical developer sees friends creating and deploying AJAX apps on the Web, getting VC funding for their Web 2.0 business plan, building a Firefox or Eclipse plugin .... and feeling as though the world is passing us by.
As I sort this all out and think about what some call the bureaucracy ...
- It's not so bad. What I see looks like textbook software engineering, with an extra emphasis on Writing Secure Code. Resisting the temptation to throw in one-line "fixes" late in the ship cycle can be the result of learning painful lessons about how much damage that one line can do. My more experienced colleagues say it's gotta be streamlined or we will continue to have overly long release cycles, but that seems to be in progress. We shall see.
- It's working. The Slashdot crowd likes to yuk-yuk about the Blue Screen of Death, but I have not seen one for about three years, and that was when a disk drive was intermittently failing. I was originally blown away by the "it just works" experience with my 2003-vintage Apple Powerbook, but the 2004-vintage Dell Latitude I got when I started here has been offered just as good of an out of the box experience, maybe even better at switching from one wireless network environment to another without hassle. Likewise, note the trends in the relative number of vulnerabilities discovered in SQL Server vs Oracle, or even in IE vs Firefox. As a consumer of MS software, I definitely don't want any of those teams to go back to the "good ol' days"!
- Sooner or later the successful agile innovators of today will have to follow suit. My thoughts on this rather confusing subject finally did crystallize the other day after reading Nicholas Carr's piece on Web 2.0. "The promoters of Web 2.0 venerate the amateur and distrust the professional." Turning that around, once "amateur" innovators start taking money for their software or service, accumulate legacy customers that they have to keep happy, get enough success to be worthwhile lawsuit targets, and build things that lots of people depend on to get their jobs done, they will have to adopt a culture of "professionalism" to meet all those constraints. Or get bought out by a big company, of course, and suffer through all that stifling bureaucracy until they're free to cash out :-)
Finally, as with everything else in engineering, it comes down to making tradeoffs. Obviously quality is a Good Thing, but so is agility, innovation, openness to new ideas, and a culture of empowerment. Likewise, some of the stuff the various bureaucracy / middle management / stasis posts touch on are clearly Bad Things, such as slimy political moves and "massaging the message" to keep early warnings of impending failure hidden from upper management. But many are much less clearly good or bad. For example, where does one draw the line between encouraging risk-taking and making those responsible "accountable" for taking risks that didn't pan out? How long should Big Ideas (e.g. WinFS, Indigo, LINQ ...) that are driven more by individual vision than clear market demand be allowed to go on before they have to be allowed to sink or swim? Bill Gates noted in his PDC keynote that it takes about 10 years from "hot new technology" to "solid piece of infrastructure." Who knows which of today's underachieving products will be the cash cows of 2015? How many great people will we lose by making them "accountable" for being off by a couple of years in a 20-year bet? The best we can do is put processes in place to ensure that someone is providing a plausible answer to "how many resources will it take to achieve which benefits form whom" before going off into the unknown.
The experience of having to look at the XML technology landscape in "how many resources will it take to achieve which benefits for whom" way vs a "wouldn't it be cool if?" way has been a rather humbling experience. I came here with a certain amount of hope for doing things like persuading the team to support RELAX NG, promoting a loosely typed approach to XML processing, maybe taking XML's processing/transmission inefficiency more seriously, and even, just maybe, getting MS to take the lead in persuading the rest of the industry to clean up / refactor the XML specs together. I haven't abandoned *all* those hopes, but I have definitely come to appreciate the depth of the knowledge and analysis behind what I viewed from the outside as stasis for stasis' sake.
For example, Microsoft's opposition to Binary XML standardization is based on actual experience wrestling with the different ways in which XML is inefficient and in analyzing how to improve that inside various XML-based products in ways that wouldn't impact interoperability. I suppose that to many on the outside it looks like stasis -- we recognize the XML is inefficient but can't endorse an industry-wide attempt to improve efficiency; from the inside I see the analysis -- the cost of XML's inefficiency seldom outweighs the benefit of data interoperability, and when it does, the optimizations for one product domain haven't been useful in another. We can probably do better, but this will take research and experience, not a committee compromise.
Likewise, the experience of writing the MS position paper for the W3C Schema Experience Workshop was particularly illuminating: as "bad" as most XML developers believe the XML Schema specification to be, it's awfully hard to make a rational case that we would make real customers' lives easier in a reasonable time horizon by replacing it as opposed to cleaning it up, implementing it properly, and learning to live with it. An even more extreme example is OPML -- the wretched spec just oozes badness from an XML aesthetic perspective, but the whole point is that almost nobody ever has to look at an OPML file, and the software that produces and consumes the stuff just doesn't care. It may be broke, but there doesn't seem to be enough payoff in fixing it to justify the disruption and confusion of replacing it unless the alternative is a LOT better.
That's not to demean "amateur" efforts of the sort Nicholas Carr talks about .... or those driven by a vision of what would be right / cool as opposed to what will meets some more mundane criteria. The big breakthroughs tend to come from people who do not conform to our conceptions of how to do things properly and professionally: Darwin was an divinity student and amateur naturalist who joined the Beagle's company largely to give Captain Fitzroy an dinner companion of the appropriate social class; Newton devoted most of his brainpower to matters of alchemy and theology; Kepler was literally a court astrologer. Everyone -- even Microsoft -- benefits from an innovation-driven competitive landscape where stasis is not an option because of cost curves going down and benefit curves going off into new dimensions.
Likewise, we probably need to do more to encourage other companies and communities to fill in the XML technology niches that our cost/benefit analyses and requirements for long-term support don't really allow us to fill directly. Lots of people want to use RELAX NG, client-side XQuery, RDF/OWL, etc. in the Microsoft environment. There's gotta be a bunch of fun projects or profitable business models in there for someone smaller and more specialized, or more passionate about the long-run potential of something the market is not yet demanding.