The Beta vs. CTP Dilemma
Over the last couple of years, the CTP or Community Technology Preview has been one of the bigger changes in the way Microsoft release software. Microsoft introduced the concept of CTPs with the “Whidbey” .NET Framework and Visual Studio release but I was happy to see the concept spread to SqlServer, Expression, WinFX and even Windows!
CTPs have been very valuable to us as we have been able to get some great feedback in near real time from customers rather than waiting for a long Beta cycle or worse until after the product has been released to hear the feedback.
But, of course, CTPs have not been without their paint points... Frequent releases come with variable quality and are often incompatible with other pre-release software putting developers in a state of “CTP Madness” where it is often difficult to figure out what two builds work together. JasonSu and I did create a cure – or at least a treatment – that I have heard helps a lot. But the pain certainly does not go to zero.
So, as we begin planning the interim release schedule for Orcas I find myself torn between the traditional value (and costs) of betas and the new value (and risk) of CTPs. The downside of betas is that in order to drive up the quality we need pretty much all hands on deck working on the beta... if there are a few features that are not quite done, it is hard to keep those features moving forward during the beta lock down. This effectively means we extend the release schedule by the Beta lockdown time. We also attempt to “line-up” Betas of related products so that customers can use them together. The plus side of betas is that because we drive the quality up and make them work together, conventionally wisdom is that more customers are able to try it out and get us feedback.
But I wonder if we did a good job with quality throughout the cycle making the builds fairly usable as they come out, could we replace the two or even three beta cycle with less betas and more regular CTPs? This of course only works if we get customers to give us the good quality, real-world feedback on CTPs.... if we don’t get the good feedback until very late in the cycle then the plan didn’t really save us anything.
So what do you think? Do “betas” offer you\your customers something specific that CTPs don’t? Are you able to pick up the CTPs and try real deployments on them or is that something you can only do with Betas?
Comments
- Anonymous
April 03, 2006
The only thing that did bother me was the incompatibility between VS and SQL2005.
Giving us the opportunity to test out those products before release does help us to minimise the learning curve and deliver our products using this technology earlier. As we are giving feedback's back to you guys also makes your product better. So keep the CTP's with better co-ordination if needed between products. - Anonymous
April 03, 2006
CTP's are generally a good thing. While you may not get the feedback in time to address something for the final public release, it does provide the development team with feedback they might not otherwise get. - Anonymous
April 03, 2006
The comment has been removed - Anonymous
April 03, 2006
The way I see it, quality in betas has not been substantially higher than the ones in ctp. Plus I'd rather have fairly unstable CTP on which i can upgrade my code to make it ready for the big day than have a separate branch cleaned up for quality but getting more and more distant from the actual code (and hence getting people to work on a version where the change delta is higher and higher as the CTP go forward).
I see beta as being important only for major milestones where a go-live license is available, hence making whatever we write available for the public to try. That also means that go-live should exist much earlier in the process.
That said this is only true for things like winfx, not really for visual studio. For that, i see no point in betas whatsoever. - Anonymous
April 03, 2006
Me, I prefer the CTP over the
Beta. It allows me to test items and give good information regarding... - Anonymous
April 03, 2006
The comment has been removed - Anonymous
April 03, 2006
both are good, but i actually prefer Betas to CTP. my preference would be for you to release CTPs for the people that want to stay up to the minute; but still have the periodic Beta release for somebody that wants something a little more solid. - Anonymous
April 03, 2006
I like having both options. If MSFT is going more to a CTP release-cycle then please, at the very least, ensure a stable installation. I'd rather spend my time reviwing the features of the product than trying to get it to install. Especially onto a virtual image.
Go the CTP route, just make sure it installs well - Anonymous
April 03, 2006
Personally, I don't mind a CTP every 4-6 weeks. The DevDiv did a good job with CTPs, it's the Windows division that doesn't have a clue what they're doing. CTPs are just public build drops, Betas are milestone releases.
Keep an alpha, two betas and a release candidate, and put out CTPs whenever it is appropriate. And keep up the great work! - Anonymous
April 03, 2006
Having had access to BOTH all along, I must STRONGLY weigh in favor of CTP. This allows much more open discussion and solution of problems inside the community, since you don't have to worry if the "other guy" really is safe for NDA disclosure. - Anonymous
April 03, 2006
I really liked the CTPs. I like the frequency of the CTPs.
I think that for frequent releases I would like to see some more work on the install story. Midway through the WinFX ctps we got a "zap everything prior" tool that helped a lot. I don't know why the ctp installer couldn't run this before hand.
I also found it difficult because the CTPs come in a lot of pieces, and so their is a lot of work to get them in the right order.
It seems a few days work would create an app that would 1) Unstall the prior CTP. 2) Download the new one, and 3) Install it with default options. Thant way I could just leave my machine on overnight and be upgraded in the morning. - Anonymous
April 03, 2006
The comment has been removed - Anonymous
April 03, 2006
I prefer CTPs vs Betas, and a CTP every 30/40 days is ok for me, what I'd like is a better install/removal tools so that i don't have to recreate a new machine every time, sometimes this process takes a long time (in special case if you don't/can't use a virtual machine) - Anonymous
April 03, 2006
Personally I prefer CTPs to be more frequent than quarterly, in an ideal world every 6 weeks would be pretty good.
Certainly the Whidbey beta process was MUCH better than the VS2003 process which felt like it was being driven by the need to get the Compact Framework in place and the bug triage process was being pretty brutal.
Possible the most important bit of having CTP builds available is that if you report a bug that someone marks as "fixed in next release" you can at least test again on the next CTP rather than waiting for many months for the next full "beta" version.
Most of us have PCs that can run a virtual PC for testing with so even the "uninstall" pain of a CTP version goes away. - Anonymous
April 03, 2006
Brad Abrams, a Group Program Manager at Microsoft, asks for feedback regarding customers' preference... - Anonymous
April 03, 2006
I like CTP's as they give us more looks at the progress of software. - Anonymous
April 03, 2006
To be honest, they're just terms. Sure, they're handled differently internally to MSFT; but to customers they're the same thing: software distributed pre-release in the hopes of feedback about defects and usability. With no real warranties/guarantees there's no other reason to install/use these things.
I've tried to be involved a several betas, and the impression I get from MSFT is they don't want to hear any feedback. IE 7 beta 1, for example, I could find nowhere to provide feedback directly to Microsoft.
Call it "Beta" or call it "CTP" but be organized and make it REALLY easy to gather that priceless feedback about defects and usability. And be consistent with gathering feedback; if it's a "Beta" then always collect feedback and always in the same places--don't have one beta that has a place to give feedback and another that doesn't. You've got an army of people willing to work without pay, in their spare time, with a specific focus and emotional commitment to the product--Microsoft should be falling over itself trying to use these people.
Have a look at some of the bugs logged in Product Feedback. FDBK44025, for example, has 13 people validating the bug, 18 people voting for it, and an average rating of 4.72 yet it's resolved as "Not Reproduced". That comes across as "we don't want to hear it", reducing people's motivation to give feedback. There's no way for customers to query stats; but, a quick search for validated, "Not reproduced" maxes out at 100 where all have a rating of 5. - Anonymous
April 03, 2006
Go with CTPs. The product is unfinished, so the "variable quality" is to be expected. There is little that I can or expect to do in a beta over a CTP.
Too many issues were postponed during the Whidbey cycle due to beta schedules. I do not mind frequent (montly or whatnot) CTPs. It is crucial to verify in a timely manner if specific reported issues have been fixed, and if not to redirect attention there. Maybe a beta a few months before the product is to be released is in order, but please focus on CTPs for most of the development cycle. - Anonymous
April 03, 2006
I am all for CTPs, but I might not be the best person to look at as an example. I typically flatten my laptop every other weekend anyhow, so I am all about having the newest semi working bits installed ;) - Anonymous
April 03, 2006
If you ask me a 3 month delay between CTP is fine, however it might be nice if these CTPs got a week or two of "pre-ctp" testing incase there is some blocking install or other major issues.
But what I'd most like to see is ability to log on to some virtual machine that is updated much more frequently and automatically. These could be used to see whether a bug fix actually did the job, by allowing the customer to upload their project to the virtual machine. After logoff the machine would reset to the state before the customer was there.
That would be very good in testing whether for example a customer solution/project has had it's problems fixed in latest build, without waiting for the next CTP to see that. - Anonymous
April 03, 2006
In an ideal world, beta software (including CTPs) should not be installed on critical systems. However, many individuals do not have extra systems hanging around to utilize just for testing. That is one reason I think some people would rather wait for a beta than a CTP.
In that regard, the news is reporting that Virtual Server 2005 is going to be offered for free. Maybe Virtual PC would be better, but perhaps the CTPs can be installed there. If that could work, it would provide an excellent means to encourage end-users to test the CTPs, as they could be removed easily without risking the rest of the system. This may be an avenue to explore. - Anonymous
April 03, 2006
I think CTPs are really great. I learn so much more about a product by using CTPs and seeing the product evolve.
Though I also think strong beta milestones are very important for people who don't want to get 'CTP Madness'. - Anonymous
April 04, 2006
An number of people have commented on the install story. I'd like to see the uninstall story improved. That way, those of us who don't have the luxury of machines that we can flatten periodically can play along more easily. - Anonymous
April 04, 2006
The comment has been removed - Anonymous
April 04, 2006
I'll weigh in with support for the people that say that both are good. Frequent CTP releases of variable and unpredictable quality, and somewhat less frequent betas with a rather higher quality bar and more coordination with other teams to make sure that a given set of beta products all work together.
Perhaps 2-3 CTPs per beta. Or more. Or less. Whatever works :)
For me personally, I'll try CTPs on products I really care about and want to see what's going on as soon as I possibly can (at the moment that's pretty much WPF and C#3/LINQ). Products that I'm fairly interested in and want to get a general idea of what's happening, I'll wait for a beta, assuming the option's available (this, incidentally, is part of the reason I didn't try SQL2005 prior to its release - they stopped doing betas in favor of CTPs only).
And I agree with sam.jack about uninstall. Clean uninstallation is tremendously important for prerelease products. If it's too hard to get that right for CTPs, that's another reason for having betas as well. - Anonymous
April 04, 2006
Go with having 2-3 betas. The CTPs were just a huge pain and I'll probably just use betas from now on to try out new version of Visual Studio. - Anonymous
April 04, 2006
for the most times, i install ctp's on a test system (or virtual servers) and betas on my machines, or even my(privatE) live environment (as with asp.net2.0, yukon). the hardest problem for me is the upgrade process. when theres every month a new release, and it always takes hours to compelete (since update to another beta is allways not easy) its too much time for me. i like ctp's very much, but there are also some bugs in it, that shouldn't be there (and in most cases the "anoying" bugs, when you use it daily. not really the "big" bugs) - Anonymous
April 04, 2006
What you call them is a separate issue to what types there are.
Unstable = play around for short(er) periods of time to uncover new features.
Stable = as above + port existing applications, make sure they run, test/report bugs
Stable + goLive = as above + start enhancing existing applications
Released = as above + start brand new applications
The frequency at which you provide the above and which type you provide at each drop is entirely up to you (given that you know what we do with each).
Whatever you drop please allow make it uninstall cleanly. - Anonymous
April 05, 2006
ctps are the best in my opinion because with this shorter release cycle i get versions with important bugfixes faster than need to wait for the next beta - specifically with winfx it's great with ctps... - Anonymous
April 05, 2006
I prefer CTPs, but if the GoLive licenses apply only to Betas, then I want a really good final beta. - Anonymous
April 08, 2006
The comment has been removed - Anonymous
April 17, 2006
This probably reflects poorly on me, but I don't know the difference between Beta and CTP release. When I am downloading either, I usually look for the latest release date and go with that. - Anonymous
October 22, 2008
PingBack from http://payre.wordpress.com/2008/10/23/le-dilemme-de-la-beta-release/