My Favorite Features: Unit Testing Enhancements in Visual Studio 11

I’ve been writing a set of posts on some of my favorite Visual Studio 11 features that I’m using in my personal development. In my last post, I talked about JavaScript tooling enhancements. In this post, I’d like to talk about the new unit testing features. Unit testing is an important step in the development process, and is something we are doing throughout the team in Visual Studio. I’ve also been writing unit tests while working on my personal coding projects.

Third-Party Test Frameworks

Developers who are passionate about unit testing will often tell you why their favorite framework is the best. With Visual Studio 11, we wanted to give people a top-notch, developer-focused experience that let them use whatever framework they wanted. To add a unit testing framework to your development environment, just install the plugin from the Visual Studio Extension Manager (pictured below) or the Visual Studio Gallery online.

There are many test framework plugins available at this point, including:

• NUnit
• xUnit.net
• MbUnit
• QUnit
• Jasmine

And of course we still have the built-in test “MS-Test” framework for .NET code, as well as a new framework for C++ code.

Writing Tests with NUnit

Let’s take a look at the example of NUnit. The NUnit team shipped an adapter for Visual Studio 11 Beta on the same day the beta was released and just recently released an update with some important bug fixes. Getting started is as easy as installing the NUnit Test Adapter for Visual Studio 11 Beta here.

In this example, I’m using a test project, which is a .NET Class Library with a reference to NUnit.Framework.dll. Here’s a quick NUnit test to show how this works.

Then click “Run All” in the Unit Test Explorer window and see the results.

This ability to use different third party unit testing frameworks, and even to mix them within a single solution, was a major ask from our customers over the years. Now you can easily extend your experience to let your team use the unit testing engine they prefer.

Running Tests Continuously

As you’re writing tests and code, and running the tests in-between each step, you may find it gets a bit tedious to constantly switch back and forth. In Visual Studio 11, you can now configure the Test Explorer to automatically run unit tests after each successful build. This is a real time-saver because it makes the test run feel like it’s part of the build.

Thinking about your unit tests as part of the build is a natural thing. For a long time, teams have used continuous integration servers to build and run tests for every check-in. Now you can get that same outcome on your local machine.

To turn it on, just toggle the “Run Tests After Build” option in the Test Explorer toolbar.

The nice thing about this feature is that it lets you focus on the code and not on the other tool windows in Visual Studio. Only when things fail do you stop to look around and see what happened.

Debugging a Test When Something Goes Wrong

When a test fails, there are a few ways to figure out what went wrong. First, you can check the test runner, which provides rich exception information. In cases when that’s not enough, it can be useful to view the test with the debugger. With Visual Studio 11, you can debug NUnit tests the same way you debug other managed tests. To do this, simply right click on the test in the Test Explorer and choose “Debug Selected Tests”.

Analyzing My Test Results

There are a number of analytics you may want to perform on your unit tests, and the most common one people use is code coverage. We significantly simplified the experience around collecting code coverage information for your unit tests, and enabled it to work with your 3^rd party unit test projects. As pictured below, let’s select “Analyze Code Coverage” in the Test Explorer.

Now when the run completes, it will open the Visual Studio Code Coverage Results window, and display editor highlighting so I can easily determine where more test coverage may be required.

You can also easily check the coverage for a test (or a set of tests) by right clicking on what you want to analyze and selecting “Analyze Code Coverage”. This can be very handy when adding tests to existing code. The in-editor coverage coloring let you see exactly what parts of your code were run by the tests you selected.

Unit Testing Browser-based Javascript

Part of my recent application was browser based, and used a lot of the new stuff shipping with ASP.NET MVC in Visual Studio 11. If you want to unit test Javascript, you can use QUnit (the unit testing framework used by JQuery) with the Chutzpah adapter available on the Visual Studio Gallery.

As you can see, the QUnit tests are discovered and executed in the same experience as other unit tests. The extensibility of VS 11 unit testing makes supporting additional unit test framework like QUnit as simple as getting (or writing) an appropriate plugin that supports that system.

Unit Testing Native C++ Applications

As mentioned earlier, we also have the ability in Visual Studio 11 to do native unit testing, with a new C++ unit testing framework shipping in the box. We’re really happy to enable this support for our C++ developers, so you no longer need to use the “/clr” flag or fall back to 3rd party frameworks.

Here’s a quick example:

For a full-fledged sample application, I recommend checking out the Hilo project, by the Patterns & Practices team, which now includes native C++ unit tests.

To learn more about native unit testing in Visual Studio 11, please visit MSDN.

Conclusion

In this post, we reviewed a range of new improvements for unit testing. These came about by the team focusing on a couple of key customer goals:

1. Flexibility – Use whatever test framework your team prefers

2. Simplicity & Consistency – Get a simple, consistent experience that lets developers focus on their work

I hope you will have an opportunity to try out the new Visual Studio 11 unit testing features. For more information, please visit MSDN.

Enjoy!

 

Follow me at twitter.com/jlzander .

Comments

• Anonymous
  May 22, 2012
  Awesome!  I love the run tests with successful build feature!  This will make running unit tests much easier, but I think there is a huge opportunity for better education in the MS Developer space about what makes a test a unit, or integration, or other kind of test.  Do you have any comment on this and/or any plans to lead developers into the "pit of success" in here?  I have a post from earlier this year that elaborates a bit on this topic: ardalis.com/Unit-Test-or-Integration-Test-and-Why-You-Should-Care It seems to me that few teams/developers who mainly write integration tests masquerading as unit tests will turn on the "run tests after successful build" feature, since these tests will take a long time to run.  At which point, either the feature will simply go unused, or worse, Microsoft will get complaints from users about this when in fact the issue is that they're not actually writing unit tests (and hopefully if one does have different kinds of tests in a given solution, you can turn on the feature just for the fast/unit tests). Great post.

• Anonymous
  May 22, 2012
  Great stuff!

• Anonymous
  May 22, 2012
  The comment has been removed

• Anonymous
  May 23, 2012
  That was a great overview. I didn't know that you could test the javascript like that. Thanks a lot!

• Anonymous
  May 23, 2012
  This is a fantastic set of features. I know that people like to emphasize the negatives of VS11, but this is definitely an area of massive improvement. Good job!

• Anonymous
  May 23, 2012
  Will we be able to specify test category or similar to be run on build? I don't want to run all integration and gui tests after each compile :)

• Anonymous
  May 23, 2012
  To repeat an earlier sentiment: Awesome! I've been waiting for Microsoft to do this for almost a decade, very pleased to see this vital bit of the native ALM loop closed.

• Anonymous
  May 23, 2012
  The comment has been removed

• Anonymous
  May 23, 2012
  The comment has been removed

• Anonymous
  May 24, 2012
  @Martin - We are planning to provide better filtering (including by Categories) for the Run on Build, but haven't completed that yet. We know it is important. Thanks!

• Anonymous
  May 24, 2012
  In the vs11 beta attempting to debug native unit tests frequently causes vs to crash. Really hope that has been fixed for release as apart from that the integration into vs is fantastic

• Anonymous
  May 24, 2012
  The comment has been removed

• Anonymous
  May 24, 2012
  Simplicity is an admirable goal but the current VS11 Native offering is too simplistic. Please please extend it to include parameterised testing. Without richer tests, we're probably going to have to stick with Google Test (aka gtest). We can't tell what your intentions are with the test syntax because the documentation page msdn.microsoft.com/.../hh694604(v=vs.110) is still blank as of 25th May. Looking in the source is the only help and that's a bit depressing.

• Anonymous
  May 26, 2012
  If we use NUNIT and TFS, will the TFS test reports (SSRS), and test impact analysis work with non MSTEST frameworks? If so AWESOME!!!!!

• Anonymous
  May 26, 2012
  Is unit testing code of native C++ applications cross-platform? If not do you plan to make it cross-platform? Thank you in advance.

• Anonymous
  May 28, 2012
  The comment has been removed

• Anonymous
  May 29, 2012
  @Allen Fienberg Test impact would work for Nunit tests run as part of build. Regarding test reports, the test results for Nunit tests would also be published to TFS in the same TRX format. You would be able to get test run statistics – like passed tests, failed tests and so on out of it.” -- Bhuvaneshwari Krishnamurthi [MSFT]

• Anonymous
  May 29, 2012
  The comment has been removed

• Anonymous
  May 29, 2012
  Hi Michael Kochetkov, The Unit test code for C++ is not cross platform. It has to be compiled targetting specific platforms - x64, x64 and ARM (similar to the development code). Thanks Bhuvaneshwari MSFT

• Anonymous
  June 03, 2012
  This native C++ testing is a great improvement over VS2010! But our unit tests are actually Behavioural Tests, i.e. they use a mocking framework to test higher level, inter-object collaboration, not just simplistic function calls. (1) Will VS11 allow testing at the C++ class level or will we have to wrap everything in DLLs and use C functions to call our tests? (2) Will VS11 give us native C++ developers nice, visual code coverage? Thanks!

• Anonymous
  June 04, 2012
  @dripfeed, To answer your question: (1) Both are supported. You can test at C++ class level by including the product source code or linking the product code as an obj file. You can also wrap things into DLLs and use export/import for testing if you want. It's up to you. (2) Yes, C++ unit test framework works with code coverage the same way as .NET. So you will get the nice, visual code coverage. :) Rong Lu Program Manager

• Anonymous
  June 04, 2012
  @ AndyDent. Thanks for the note. We will update the doc in the next doc refresh. Regarding parameterised testing, can you please post an entry on User Voice (visualstudio.uservoice.com)? This will help the product team prioritize. There are already some ideas listed on native unit tests. You can vote for these ideas also. – Mathew Aniyan, Program Manager, Visual Studio ALM Team.

• Anonymous
  June 05, 2012
  @RongLu Thank you for clarifying that. At last, we can stop wrapping our tests in C++/CLI! Wait...what's this in the VS2012 RC Readme? 1.3.8 Test Tools 1.3.8.1 Code coverage for native C++ doesn't work Code coverage for native C++ doesn't work in Visual Studio 2012 RC. Nooooo!

• Anonymous
  June 14, 2012
  @RongLu The C++ code coverage worked in 11 beta. What happened? When will it be fixed?

• Anonymous
  June 18, 2012
  @HansH, @dripfeed, Sorry for the confusion. Yes, Code coverage worked for C++ in Beta and it will continue to work. We were having a few issues with the RC build but good news is those have been fixed since RC is out. Thanks, Rong