Long-term goals/challenges of community tech previews
I was forwarded a great email thread from VB customers. Here are some excerpts (from multiple people):
"I'm beginning to think "Community Editions" of products as large and complicated as Whidbey are NOT a good idea. Yes I said, NOT.
Reason? I believe the average non-NDA, Joe Blow developer (esp. those
new to .NET or those who've not upgraded yet) out there get the wrong
impression. These editions are just not up to the quality-bar that the major betas. And *I* understand that. Most of us do. I just wonder if they don't hurt the products.”
“I'm disappointed in how Microsoft is handling this "release". Each time they've done it (MVP summit, Dev conf in Orlando, Whidbey preview
workshop this week), I've counseled them to tell the developers that
anyone installing this software must be prepared to format their drive
and re-image. Invariably, they agree but say nothing.
This can only cause hard feelings as more "Technology Previews" arrive. Fewer developers will dare to try them--simply because they don't know how to protect themselves.”
“Why not offer these drops in the form of a VPC image with the build
installed? Not only would this give the community a safe way to work
with the builds, it would also let you get up and running in 2 minutes
instead of the 20-40 it takes to get a preview installed.”
“I agree that VPC images would be a _lot_ more compelling. Unfortunately, there are lots of issues with distributing VPC images.”
It’s great feedback, partly because it shows that we need to explain our long-term intent better. I’ve touched on some of this before, but I’ll try to elaborate here.
Our ideal is that every build is publicly available. Another way to say it is that we want to enable a scenario where a customer finds a bug, reports it, the team/developer gets the report essentially immediately, sees it’s an easy fix, fixes it, resolves the bug, the customer gets notified, and with no extra effort from anyone (beyond the customer installing) the customer can install the fixed version within a few days. I have tried to do that within our existing processes and it requires a significant amount of extra effort to get the fixed build distributed to even one customer. There are similar (but different) scenarios around getting feedback on features before the features are so baked that it will be a whole product cycle (!) before the feedback can be addressed. Frequent public builds are a means to the end of enabling us to be much more responsive to some kinds of customer input.
We’re in a very significant transition phase that will be challenging due to people’s expectations (both internally and externally). For a long time we’ve released builds very infrequently and gone to great lengths to make sure “builds work as well as possible” by having long lockdown/test periods before each beta. (More on “builds work as well as possible” below) This March Tech Preview is a step towards much more frequent builds, but the transition is causing turbulence because of past expectations. Once we’ve been putting builds out frequently (think weekly or every day) for a while, there will be less confusion about which ones have been through comprehensive testing – many people will learn to ignore most and look for the Beta indicator of quality, while bleeding-edge customers have the option of being more involved. I’d love input on how to get through the transition phase better.
I put “builds work as well as possible” in quotes above because the downside to the long lockdown periods is that we’d regularly get customer feedback several weeks prior to the release and decide not to fix it simply to avoid the risk of destabilization. That has been a source of great frustration for many of us for prior releases – we’d get builds out to some customers at the earliest possible moment but we’d already be well into lockdown so feedback we got prior to beta N often didn’t get addressed until a beta N+1 many months later. This is the tension – minimizing change so we don’t break things that worked for customers yesterday, or addressing the feedback they gave us based on their experience yesterday. Community drops are about getting feedback in time to address it.
We’ll be releasing another community technology preview in a few weeks. You will probably find that the quality will be significantly better than the March on. This is a function of where we are in the release cycle. The March one happened very shortly after a significant number of changes went into the build. Since then we’ve fixed a huge number of bugs, done extensive testing and not added/changed functionality. We will explain this to people. There will still be significant bugs – otherwise we’d call it a beta.
The feedback above on messaging at conferences, etc., is great – I’m following up on that. We did put a fairly dramatic warning on the cd envelope - I know that had some effect because someone in my management chain asked a VSLive attendee whether they’ve installed the build, and the person said “Are you kidding?!? Have you read the warning?” We’ve also had relatively limited downloads on MSDN (in the thousands, vs. many times that for betas). Some obviously got the message but again, given expectations, we have to do stellar here to avoid problems and we’re not stellar yet.
In the long run we need to do five things well for this to really come together:
First, we need to solve the distribution problem. We’re looking at several different options – shipping media, VPC images, the p2p solution suggested earlier. The one I’m most intrigued by (and probably the farthest from reality today) is incremental updates. Anyway, we need to figure out how to get a large quantity of bits to customers in a quick, reliable, and cheap way, otherwise it’s impossible to do this frequently.
Second, we have to expose what we know about the build’s quality. Internally we survive by running build verification tests on every build. At a minimum we can start exposing that level of information for these builds. Part of the problem is that the information is very hard to digest, e.g. it comes out as 83% of the tests passed, but what you really want to know is that you can’t open a project or debug because there are significant failures in those areas, i.e. the test results can look good when any real user would say it’s broken.
Third, we need to be smart about letting the community help in communicating build quality. I described this in an earlier post, but next to the “Download this build” button, I would love to show how people who’ve installed so far feel about the build. If I’m a customer about to download the bits and I see that 21 out of 23 people who’ve installed so far say “It’s great” (or “it sucks” as builds sometimes do), I can make much better decisions about using my time to download and install. There’s an opportunity here to promote and benefit from community self-organization.
Fourth, we need to be more transparent about where we are in the dev cycle, what features are coming in now, etc.. We’ve made some progress here, but we need to do more. One thing about the March tech preview was that it happened pretty quickly and we missed opportunities to explain where we were in advance, to set expectations better. In the coming months expect us to do things like cover the project status on Channel 9 and be even more explicit in blogs. We’re doing a review of the March tech preview now to figure out what to change and what specific steps we can take today to help with the next one.
Lastly, we need to give customers a dependable, effective mechanism for submitting feedback and finding out what we’ve done with it. Few customers have this today, and historically we’ve done a poor job responding (although we’ve done much better with feedback on the Whidbey alpha from last summer). If customers can submit feedback, get a response within a few days, and see the results a few days later, my guess is they’ll be more interested in tracking our progress more closely.
I think if we do those things – make the logistics easy, make it easier to assess quality, make it easy to understand whether the builds have features/bug fixes that are interesting enough to me – then much of today’s pain will go away (or be offset by benefits). The problem right now is that none of these things are automatic. The many many month release cycle has gotten us in the habit of doing little of it as we go, and then trying to pull a bunch of stuff together at the last minute – it’s pretty classic for us to send out a beta with 0 guidance on what new features there are or what has been fixed. The current community drop approach is still too much like infrequent betas – but we are starting to figure out how to build it in and make it part of what we do every day.
In the end, if we do all this and 99% of installs still happen with betas (but 1% continue to make the effort to get more frequent drops), we’ll be satisfied because two things will have changed:
- We will have shifted much of the control/decision-making over how involved customers can be from us to our customers.
- It will enable every person within our division to be much more responsive to customer feedback, without a lot of incremental extra effort.
That’s the big picture. I’m very excited about frequent community/public builds not because people are clamoring for them, but because of the implications of making them work well for us and for our customers. There is some skepticism internally about these – no one will install, etc.. And there are stages in our product cycle where builds generally are bad. But even scoping this down to just the time between a beta1 and RTM would be a huge improvement on what we do today. The great thing is that that period is coming up soon and it's a great chance to figure it out, so that when we get ready for the next product cycle we have enough of the infrastructure/processes/philosophy in place that we can continue.
Thoughts?
Comments
- Anonymous
May 07, 2004
The comment has been removed - Anonymous
May 07, 2004
The comment has been removed - Anonymous
May 08, 2004
I hate it when customers go to a huge amount of effort to report a bug and then we say "oh yeah, we knew about that one". While we have significant improvements planned in the near term for being much more transparent about feedback we get from customers (which addresses to some extent your interest in filtering), I don't have a great answer today about internal bug status - there's always been a concern about revealing customer confidential information. (Plus some other concerns which could be addressed through, um, professionalism...)
One thing I didn't say in the post above is that this is a many month (if not years) process. If we waited until we got the process/transparency 100% right before we started, it would never happen - there's simply too much internal inertia. Thanks to some very strong management support, we've gone from releasing builds very infrequently to somewhat more often. We have lots of gaps in what we need to make it really work, but by taking the first step we're now getting very concrete feedback from customers on what is missing.
The suggestion about MSDN download ratings is a good one - I was thinking just about Whidbey but you're right, this would generally useful. I'll follow up with them. - Anonymous
May 08, 2004
I think the VPC approach would be best. I would also suggest a whitepaper (which I can write for you) that outlines how to test beta software and not have to buy a new hard disk each time (which actually works pretty well) ;)
Of course, the VPC approach assumes a great deal. I have moved VPC images around on my systems but they're all pretty healthy. We would need a minimum testbed system to host the VPC or the performance would be "challenged". - Anonymous
May 09, 2004
The comment has been removed - Anonymous
May 09, 2004
.. OTOH it could be that people would just shut down the p2p part of the thing after being done with their daily update, making the updates mostly being fed from MS servers. - Anonymous
May 10, 2004
The comment has been removed - Anonymous
May 11, 2004
The comment has been removed - Anonymous
May 14, 2004
I develop with both C# and VB.NET, but I like VB.NET to be better because I am old VB developer.
I just installed VS.NET 2005 (Technology Preview), and this my prompt feedback
------------------------------------------------------------------------------
I surprised that I found productive feature in C# and not
in VB.NET, this features:
1) Refactoring
I tried Refactoring in C#, It is really very productive
feature.
2) IntelliSense/Auto Completion (Keywords)
Now only in C# debugger feel the Language keywords like
(private, public, foreach ...), and auto complete it to
you.
However VB.NET have longer Keywords such as (MustInherit,
NotInheritable,.)
----------------------------------------------------------------------------
I dreamt that I can find these features in VS.NET, but
unfortunately I frustrated when I found this feature in C#
not VB.NET ?
Microsoft always say that it concentrate on Productivity
features in VB.NET, How does VB.NET 2005 lack these
very productive features ?!
Looking forward for your reply.
Amr Essam
www.Verizon.com
Consultant & Team Lead
MCSDT + MCT
Dallas, Texas
amr_essam@yahoo.com - Anonymous
May 18, 2004
>> Anyway, we need to figure out how to get a large quantity of bits to customers in a quick, reliable, and cheap way, otherwise it’s impossible to do this frequently.
What about using BitTorrent? - Anonymous
June 13, 2009
PingBack from http://hairgrowthproducts.info/story.php?id=4062