Last year's VSTS 2010 CTP - Feedback still needed!

Last year's CTP of VSTS 2010 was one of the major milestones on the road to shipping what we think will be a great release. There's a lot of value, and much of it previewed in the CTP, but I've been feeling like we didn't get a huge amount of community feedback to help us fine tune the plan. Selfishly - I'm interested in your thoughts on two of the major innovations our team was responsible for:

Eliminate the “no-repro” bug with application flight recording.

How many times has a developer used the phrase “works on my machine” when responding to a bug? I know I have :) In many cases that may be true. By default the VSTS 2010 testing system will record a host of information:

1) Screenshots of the test in progress so developers can see the symptoms

2) An environment snapshot to aid in configuration mismatch diagnosis

3) An application flight recording that can be replayed in the debugger to allow developers to quickly get to the root cause of any issue

The application flight recorder technology is a key element to actually finding and solving bugs faster. Not only can developers see symptoms, they can understand what their code was up to at the time and get to the problem faster. To do this, we took advantage of one of Visual Studio’s core experiences – the VS debugger (which we are responsible for too). Many developers spend a large portion of their time using the VS debugger for both software construction and debugging. So we now provide a new set of debugging experiences that focus on looking at the application through the recording, and seamlessly integrate that with the debugger’s existing views, as opposed to building a new tool that people would have to learn and get some experience with. Any developer will be able to get value directly out of the gate. This also frees testers from extra rounds of validation (did that bug really exist), or working to reproduce a bug that only show up intermittently or even going to the trouble of setting up a test environment for the dev to repro and debug on live.

Understanding the test impact of code changes

A common question that testers in software organizations ask is “which tests should I run to ensure quality”.  This is especially critical for teams with a test bed of many thousands of tests as they approach the release point of their software.  Code changes can break the product, but also simply break the tests associated with them.  All of this is cost, and as time progresses from the date the issue was added, it becomes more expensive to understand and fix.

VSTS 2010 intends to provide guidance around which tests have been impacted by changes in newer builds.  As tests run in either automated or manual test scenarios, information is being recorded about which tests touch which code elements in a given project.  This mapping is then transmitted to the test server which maintains a database of mappings.  Team System then makes this information available to all the participants in the software process.

1) Test managers have a built in report of the test impact of any given new build.  As well as understanding the tests impacted by changed source code, they can see which requirements have changed, or which bugs have been resolved, and by extension the test cases that are related to those.  They can then decide to promote a new build to be “the testable build”

2) An individual tester can see this same information for their area, and make decisions on which of their tasks they ought to work on first

3) Developers can see the upstream impact of any change they make, and potentially run the impacted tests to understand if there is a breaking change, or if they have broken the product.

As we get closer to Beta, we should be going deeper on how these features work and what you can (and can't  :) ) do with them, but until then, please go test drive them and give us the feedback!

Keep on fixing,

John

Comments

• Anonymous
  February 05, 2009
  The comment has been removed

• Anonymous
  February 05, 2009
  I hate to me too, but kfarmer has it nailed. Deliver a reasonable release I can install and actually do work with and I'll file feedback all day long. But a VM? Unless it's just to "check out" some specific little thing, it's pretty useless as far as me getting into it.

• Anonymous
  February 05, 2009
  I tried to use the Historical Debugger, only to find that it didn't work with our Native C++ code.  Many other new features I tried either caused crashes with our code base or didn't support Native C++ apps.

• Anonymous
  February 06, 2009
  Thanks folks.   I totally appreciate where you are coming from wrt VMs.  It was no fun at all trying to lug that thing around when doing demos at PDC.  When we get to Beta I would expect us to not rely on VMs, but it isn't my decision so that's my opinion. On native code - jsamuel, could you ping me directly via the blog link.  I've been trying to get a handle on how many of our Team System customers are using native code, what size their teams are etc.  No native support this time around was a hard cut for the team.  Of course we personally want all of our features to work with every technology.  But when you are deeper in the stack (debugger, profilers, flight recording) the platform variation really affects you - what works for one platform is not merely a "refactor" away.

• Anonymous
  February 12, 2009
  Steve St. Jean on Team System Web Site Remodeling Jeff Beehler on Check out our updated website and Welcome