What's the big deal with Vista Betas?
In this somewhat silly Channel9 post about unsubstantiated rumors of Microsoft canceling Vista Beta 2, "Sabot" wrote the following question:
What's all the fuss about a Beta? I don't actually get it? It comes when it's ready and I'm happy with that.
I'm one of those guys that's not really all that fussed about knowing the release date of any product, I always architect systems using existing technology, it's a school-boy error to do otherwise because the plain truth is, there is a 50/50 risk it won't be released by the time you need it in the project plan, and that in terms of risk is to high a percentage to be acceptable
As a consumer of Microsoft's operating systems, I'm 100% behind him. I don't get the hype (although I think seeing the purported screen shots is cool).
As a developer who is working on Microsoft's operating systems, I have a rather different view.
You see, the betas of Microsoft's operating systems are really the only opportunity that Microsoft's developers have to see the "real world" outside the ivory towers of Microsoft.
Within Microsoft, you see, we all tend to have more-or-less the same types of machines. They come from maybe a dozen different manufacturers, and they all have reasonably different capabilities (for example, many don't have any audio). But they tend to be pretty homogenous - 1.5 to 3 GHz machines with between 256M and 2G of RAM, an "ok" video card, and an on-the-motherboard audio and network solutions. The thing is, they don't even come close to representing the diversity of systems that our customers have. We typically don't have the $200 low-end computers in our office, neither do we have $7,000 watercooled ultra high-end gaming rigs.
We also don't have all the myriad applications that our customers have. We've got an amazingly huge library of apps to test (I know, I've been looking at appcompat failures), but we still get reports of problems in apps that we've never heard of. We also don't have the applications that haven't yet shipped, so we have no ability to test them.
As a result, the OS developers within Microsoft LOVE betas. We love it when real customers get a chance to use our stuff, because their environments are so much more diverse than any we could possibly have. In reality, we do a pretty darned good job of weeding out the bugs before we ship (for instance, we've only found a handful of new customer reported audio bugs since Beta1), but there's no substitute for honest-to-goodness in-the-field tests.
It's kinda interesting - Platform software is very different from just about every other engineering endeavor, because the compatibility matrix is SO complicated. You can build an automobile totally in isolation - you can solve every engineering problem you're likely to ever find before it leaves the plant, because you can pretty accurately gauge how your customers are going to use your product. But you can't do that with a software platform, because the number of potential use scenarios is essentially limitless. So for software platforms, betas are indispensable tools that provide critical feedback for the developers of the platform.
Comments
- Anonymous
December 01, 2005
Very interesting article. I just have to say that your automotive analogy is not quire right. From outsite the industry, it may look that "simple," however, consider that you can go down to the dealership and order a pickup truck with whatever options you want, and that causes the factory to have to deal with > 10,000 possible combinations - all of which need to be designed to work well together.
But it is true, however, that there is no responsibility on the part of the automotive OEM to support in-field updates/upgrades to the vehicle configuration, so from that perspective, it is simpler than what Windows faces, but it still overall is not "simple." - Anonymous
December 01, 2005
How has virtualization impacted the feedback loop? I don't install anything beta on my 'real hardware' anymore. It is all VPC's. Has this skewed the amount/quality of compatibility feedback? - Anonymous
December 01, 2005
I love new toys to play around with, and even if I'm not convinced Vista will be right for me I still have to try it to be sure.
I don't like the toys as a developer for the most part, but just as a day to day user that spends way too much time on their computer. :) - Anonymous
December 01, 2005
If Microsoft paid a nominal bug bounty, perhaps betas would be more effective? - Anonymous
December 01, 2005
I agree with Manip, actually. As a developer at work, we only ever consider installing new software after it's been out for a few months at least (even better is after the first service pack comes out :p~)
But on my laptop at home, it's got betas of anything I can lay my hands on! It's always good to stay ahead of the curve :) - Anonymous
December 01, 2005
The comment has been removed - Anonymous
December 01, 2005
It just proves my argument that building software is a very hard beast to contain. It is my hypothesis that software building is hard is because we as humans are a visual animal, by that I mean that if we are building a physical structure, be it a house, bridge etc, during this build if something is going belly up, most times we have ability to correct this before it is completed. (Having said this, there has been some disasters in the past and shall continue in the future.)
Now, when it comes to software building, we can only see the results (ie does the software/system match the requirements that was intended) once the application has been built – That is why it I agree in releasing Beta’s.
What would be ideal is during the application/system build process a “build monitor” is positioned between the compiler and the requirements that check what is getting built matches to what has been specified in the requirement parameters for the application/system. - Anonymous
December 01, 2005
Dazhel: I was thinking the bounty amount should be at the developer's discretion, actually - Anonymous
December 02, 2005
I think the car analogy is a bit flawed because MS has ninety-odd percent of the operating system market, and ships a new OS version only every few years. There are a lot of commercially significant car companies, issuing in total dozens of new models every year, some of which people simply don't buy. Or buy and then hate. What MS has to do is more like packing three years' worth of new models of Accords, Corvettes, BMWs, and Prion^H^H^H^H^HPriuses into a single release. But they've probably got a couple of orders of magnitude more people on the project, too.
The car companies' problem isn't trivial; if it were, GM would still be viable. - Anonymous
December 02, 2005
"What would be ideal is during the application/system build process a “build monitor” is positioned between the compiler and the requirements that check what is getting built matches to what has been specified in the requirement parameters for the application/system."
Robert: Isn't that what software testers do?
You're talking about an extremely complex problem for computers to solve. If you're going to write a program that eliminates software testers by checking a build against the requirements, why not just go the whole nine yards and write a program eliminates software developers as well by writing a program that takes the requirements and creates the actual program? :) - Anonymous
December 02, 2005
Maurits:
Speaking as a developer myself, we already work our rear ends off prepping a product for release. I think I'd speak for a lot of developers when I say that the last thing I want to have draining on my time is mediating disputes over who gets what bug bounty. - Anonymous
December 04, 2005
> You see, the betas of Microsoft's operating
> systems are really the only opportunity that
> Microsoft's developers have to see the "real
> world" outside the ivory towers of Microsoft.
Which means that Microsoft's developers STILL never have an opportunity to see the real world outside the slightly extended ivory towers of Microsoft plus beta testers.
Microsoft doesn't need to pay a bounty for bugs. Microsoft only needs to stop CHARGING a bounty, accept bug reports from the real world, and allow victims to get the fixes. - Anonymous
December 05, 2005
The comment has been removed - Anonymous
December 07, 2005
The comment has been removed - Anonymous
December 08, 2005
Nick, that happens eventually. But we've learned from VERY hard experience that you can't open wide betas early - look at the number of people who put the Longhorn Alpha code on their day-to-day machines. - Anonymous
May 31, 2009
PingBack from http://indoorgrillsrecipes.info/story.php?id=3484