Jaa

Writing Unit Tests in Visual Studio for Native C++

  • Artikkeli

My team frequently creates wizards that run in Windows PE as part of a Configuration Manager task sequence, and we write them in C++. I’m a big fan of TDD, so when I started to work on a new architecture for our wizards, I wanted to write all the code using TDD.

When I asked around about writing C++ tests, I was told there isn’t any support for C++ unit tests in Visual C++. Not true. In this blog post I’ll describe what’s required in order to write unit test for native C++ code using Visual Studio.

For new C++ projects, you can set them up as I’ll describe below. However, for your existing code, you may have to reorganize your project structure before you can start writing unit tests.

Here is a quick overview of the setup for testing C++ code:

  • The unit tests need to be written in C++/CLI in order to use the .NET unit test framework
  • The code you’re testing should be in a static library that is linked into both your production EXE/DLL, as well as the test DLL

image

Setting up the projects

As I mentioned above, the first step is to setup the projects correctly:

  1. Create a new Win32 Project and select Static library in the Application Settings screen. This is where you’ll add your production code
  2. Create a second Win32 Project for your production EXE or DLL
  3. Create a Test Project (in the C++ section of the Add New Project dialog box)
  4. In the Properties dialog box for the test project, change the Target Name to the name you want to use for you unit tests. The default is always DefaultTest, rather than the name of the project you just created

I’m using Visual Studio 2010, but these instruction will most likely work with prior versions, especially Visual Studio 2008.

Next we need to link the static library into the test and production projects, and provide both of those projects access to the header files in the static library:

  1. Right click the test project and select Properties
  2. In the Configuration combo box, select All Configurations to ensure the changes below are made to both Debug and Release configurations
  3. Click the Common Properties node and then Framework and References
  4. Click Add New Reference… , select the static library and click OK
  5. Click the Configuration Properties node, then click the C/C++ node
  6. Click in the Additional Include Directories property and type something like this (substitute the name of your library project for ProductLibrary):
    $(SolutionDir)\ProductLibrary;%(AdditionalIncludeDirectories)
  7. Change the Common Language RunTime Support property to Common Language RunTime Support (/clr)

You’ll want to repeat steps 1-6 for your EXE/DLL project.

Writing C++/CLI Unit Tests

At this point you should be able to build your solution, so the next step if you’re using TDD is to write a test method. If you’ve written test methods in C#, most of this should be familiar to you, although with a different syntax. But if you’re a C++ programmer and you’ve never written C# unit tests, there is a bit of a learning curve. Here I’ll explain enough to get you started.

Creating a new C++ Unit Test project initially creates a file called UnitTest1.cpp that contains one unit test. I usually delete the extra code and just leave the single test method. If you have a simple class called SomeClass with a method called SomeValue that returns an int, you can write a test that looks like this:

 [TestMethod]
void ShouldReturn1()
{
    std::unique_ptr<SomeClass> pClass(new SomeClass());
    Assert::AreEqual<int>(1, pClass->SomeValue());
};

There are several elements to this test that you may not be familiar with, but you can find full details on MSDN. First is the TestMethod attribute. This identifies the method as being a test method. When you run the tests in Visual Studio, it will run any public methods marked with this attribute.

Next, notice the use of the Assert class static method called AreEqual. There are a number of static methods that allow you to validate the results of running a command. Again, you’ll find all of these documented on MSDN.

Some Tips and Tricks

The most significant limitation is the lack of IntelliSense in C++/CLI in Visual Studio 2010 (the C++ team plans to correct that in a future release). Fortunately, there is a third-party product called Visual Assist X from Whole Tomato Software that brings back IntelliSense to C++/CLI in Visual Studio 2010. I’ve been using this now for a few weeks and I really love it.

There are some tricks you’ll want to be aware of when you’re using Assert. Because Assert is designed for .NET code, there are some methods that won’t apply, such as IsNotNull, which expects to receive a managed pointer rather than a native pointer. Here are some ways you can deal with this limitation:

 Assert::AreNotEqual<DWORD_PTR>(0, (DWORD_PTR) pClass, "Should not be null");
Assert::AreEqual<DWORD_PTR>(0, (DWORD_PTR) pClass, "Should be null");
Assert::IsTrue(pClass == nullptr, "Should be null");

Finally, if you want to test a string value returned by a function, you can do something like this:

 Assert::AreEqual<String^>("Expected", gcnew String(pClass->StringValue());

Note in particular the use of gcnew , which creates a new instance of a managed class. Likewise, you’ll notice the ^ at the end of String in the AreEqual. This tells the compiler that we’re working with a pointer to a managed instance of type String.

Update

I’ve create a new blog post with more tips and tricks: C++/CLI to C++ Tips and Tricks.

For code coverage, see Capturing C++ Code Coverage with Visual C++.

Final Thoughts

I’ve been using C++/CLI unit tests with native C++ code now for about 1-1/2 months and I find it a really nice environment. I can debug my unit tests just as easily as C# unit tests. At this point I have about 240 tests and they run in about a 1-2 seconds, which means I can easily run all these tests after making changes to ensure I haven’t broken anything.

After so many years writing in C#, I never thought I would enjoy C++ programming again. I was wrong. Using TDD to write C++ code is almost as nice as writing C# code, and I’m really enjoying the experience.

I want to thank my colleague Mike Schmidt for getting me pointed in the right direction. He had some C++/CLI unit tests, but they were testing public functions of a DLL. I did some research and added the part of about using a static library, which provides full access to the internals of the code—just the thing you need for writing code with TDD.

Comments

  • Anonymous
    November 21, 2010
    FYI, to use IntelliSense with managed code all you need to do is to toggle the Common Language Runtime Support switch of the C++ property pages:
  • when writing: select "No Common Language Runtime support"
  • when compiling: use "Common Language Runtime support (/clr)" Some IntelliSense is better than no IntelliSense. :) Maybe someone could make an addin or a macro to toggle the switch based on compilation.

  • Anonymous
    November 22, 2010
    Thanks for the suggestion, Michael. I'm going to play around with this and see if I can figure out how to make this automatic since that would be a really nice experience. Having IntelliSense for the code that I'm testing, but not of .NET, would be a very nice improvement.

  • Anonymous
    November 22, 2010
    John, thanks for the writeup.  It got me up and running very quickly with TDD, C++ and VS 2010. I'm also targeting 64 bit platforms, so I noticed that the pointer tests should be using _PTR types, such as Assert::AreNotEqual<DWORD_PTR>(0, (DWORD_PTR) pClass, "Should not be null"); Thanks again.

  • Anonymous
    November 22, 2010
    John, Thanks a lot. Seems like a good idea. I'll be moving from CppUnit to VSUTF very soon.

  • Anonymous
    November 23, 2010
    I thanked you a little too early there. Write unit tests with no intellisense support? Are you kidding me? I'd rather use notepad.

  • Anonymous
    November 23, 2010
    The comment has been removed

  • Anonymous
    November 25, 2010
    The comment has been removed

  • Anonymous
    November 29, 2010
    I did some research over the weekend and found various posts talking about a product called Visual Assist X that is supposed to have support IntelliSense in managed C++. I just installed a trial copy and I can report that it does, in fact, provide full IntelliSense when writing unit tests in managed C++. With this add-in, I now have the best of both worlds!

  • Anonymous
    December 10, 2010
    Actually, you cannot use MSTest and C++/CLI to test native x64 code at all.  In order to call native x64 code from C++/CLI, you need to build using the x64 platform.  The test project will build fine, but MSTest cannot load x64 assemblies.  This very unfortunate limitation of MSTest means that you can only test Win32 native code using MSTest and C++/CLI.  To test native x64 code with MSTest, you would need to write your tests in C# and call native functions with P/Invoke, and compile your C# test project with the Any CPU platform.  See this blog post for more info: blogs.msdn.com/.../visual-studio-team-test-load-agent-goes-64-bit.aspx

  • Anonymous
    December 13, 2010
    Adam, thanks for pointing that out. We don't have a requirement to deliver a 64-bit version of our application, so I hadn't run into hte x64 limitation you described. Do you find many cases where 64-bit code doesn't behave properly while the 32-bit code does?

  • Anonymous
    December 14, 2010
    Hi John, ideally one would write code in such a way that the same source code compiles and executes correctly on 32 bit and 64 bit platforms.  However in some cases this isn't possible, and even when it is, code doesn't end up being portable between 64-bit and 32-bit platforms automatically.  So there is always the possibility of errors on one platform or another.  I have never seen a case where a large body of code that had only been tested on 32-bit platforms was converted to 64-bit by just recompiling and having everything work correctly.  So I would not consider it an option to ship 64-bit code that had only been tested in 32-bit mode.   I think whatever test technology you choose must support testing on all of the platforms you plan to support, so I think the MSTest with C++/CLI approach unfortunately isn't useable by people who need to ship 64-bit code.

  • Anonymous
    December 21, 2010
    native c++ unit testing: www.boost.org/doc/libs/1_45_0/libs/test/doc/html/index.html

  • Anonymous
    December 24, 2010
    till IntelliSense become available on C++/CLI, I chose WinUnit@CodePlex

  • Anonymous
    January 04, 2011
    It's incredible that there are people who don't know Visual Assist. I've been using it since VC6!! VS intellisense is/was/will be(?) nothing compared with VA.

  • Anonymous
    January 29, 2011
    You can found here codebasequality.com a useful technique to avoid limitations due to use of c++CLI.

  • Anonymous
    February 09, 2011
    This is actually the same technique you can use to test native C++ projects with NUnit; we've been doing this with the Visual Lint codebase for several years now so you could say that we've got form in this area. The most serious difficulties we're encountered are twofold:

  1.  If the project you're testing uses native resources, you will probably have to write a resource DLL to host them, since persuading to managed assembly (i.e. the test fixture) to also host unmanaged resources seems to be an exercise in frustration. That can make writing some tests harder, which can't be good.
  2.  To debug a failing test you are going to have to use mixed mode debugging, which in VS2008 is still exceptionally brittle (can't speak for VS2010, but I would be surprised if it has changed much in that regard). That can make running tests more painful, which is definitely not good. As a result of finally having enough of the problems above we're now in the process of ditching NUnit for our C++ tests and moving them to Googletest. Despite the lack of a decent UI, the resultant tests are proving to be far easier to work with.
  • Anonymous
    February 10, 2011
    It seems to me there are two motivations for using C++/CLI for unit test code. Maybe three.
  1. You get reflection, which makes writing and working with unit tests much easier. C++'s lack of reflection has traditionally been a road-block to easy-to-use unit test frameworks.
  2. You can get integration with existing test runner tools (MSTest, NUnit etc).
  3. If you also maintain a managed code base you can achieve some continuity of framework, test style and tools. I have attempted to address (1), and will follow-up with (2) with my own, purely native, C++ Unit Test framework - Catch: www.levelofindirection.com/.../unit-testing-in-c-and-objective-c-just-got-easier.html I can't directly address (3), but I would argue that, while it may be a useful thing if you are maintaining .Net codebases, the xUnit style of test framework is not a great fit for native C++ and a more adapted approach is preferable (which is the approach I have taken).

  • Anonymous
    February 21, 2011
    I should warn everyone from using MSTest to test C++ code on any version of the Studio prior to 2010. It simply does not work. After you create certain number of tests (around 50 or more), the Studio will start to die on you when loading unit test projects. It means you will not be able to open the project or it will take ridiculously long time. The problem is in Intellisense getting confused while trying to parse unit tests' code. At some point it just seem to go into the infinite loop, and there is no solution for it. And, surprise, you can't turn intellisense off, as it will also turn off the MSTest. 2010 is better, we now have a solution with ~500 unit test and did not have that problem (yet). Regardless of that, using MSTest for C++ code can be a big pain by itself. C++ class cannot be a member of the unit test class, managed library does not like C++ static objects very much, additional code is required to have asserts on strings etc, you name it. Debugging can be another issue. In our case, our unit tests sometimes look as a bunch of workarounds for another workarounds.

  • Anonymous
    February 21, 2011
    There are large (dare I say insurmountable) problems with attempting to use the MsTest framework for unit testing in c++. Let me list them in order.

  1. Your unit test class will be managed. This means that if you want to create a local member you will need to either have a pointer instance or, create some kind of struct to hold unmanaged instances.
  2. All your asserts are designed for .NET and so everything that is unmanaged must be marshalled to managed to be used in the assert (or just use Assert::True everywhere).
  3. Intellisense prior to visual studio 2010 causes large unit test projects to hang for 5 minutes or more at a time.
  4. As the Unit test is running in a mixed-mode environment, debugging the test can be very difficult. Stack variables may not be visible, containers not viewable. As well as this, it will constantly drop to the disassembler when stepping between the managed and unmanaged environment.
  5. Compilation and running of the tests are extremely slow compared to other C++ unit test frameworks. This is an absolute killer for TDD. Good TDD frameworks will let you make a change and run the test in under a 5 seconds, in MSTEST, your looking at minutes+. If you are doing TDD I would warn you in the strongest terms to avoid MSTEST for C++ unit testing. It may look an attractive solution initially, but it is the worst possible solution you could choose. There are other unit test frameworks that will integrate with Visual Studio (and even the ones that don't really add no real overhead). For me, I must use MsTest every day, because of the decision to go with it, and it is the bane of my existence.
  • Anonymous
    February 22, 2011
    Ben, perhaps you could tell more about the environment where you ran into these conditions. I've been using this approach with TDD for over 4 months, and except for 1, I haven't encountered the problems you describe. We have over 400 unit tests, and the compile/run test cycle is about 5 seconds most of the time. The exceptions are when I make a header change that effects all the code. Even then, the build time is mostly the time to build all the native code.
  1. Yes, this is true. It hasn't been a big deal for me.
  2. I use tests like Assert::AreEqual<HRESULT>(... and this works just fine. Can you provide a case where you have to use IsTrue?
  3. I'm using 2010.
  4. I debug all the time and haven't encountered problems.
  5. I haven't seen this. I write all my code with TDD, and the performance has been fine.

  • Anonymous
    February 22, 2011
    I have worked on several projects using MsTest with C++ development, and have hit different issues in different environments. If we restrict the discussion to 2010 only (and indeed the most egregious problems have been resolved here), the issues that remain may be specific to our setup. Still, let me go into more specific details. The compilation time I am talking about is actually mostly link time; it takes nearly 2 minutes to link on my quad core 8 gig machine, when the main project links nearly instantly. I've not been able to work out the issue. Our project makes liberal use of templates the STL and BOOST, which may be a factor. If it works for you, fine, but I've never seen this issue with other test frameworks. When debugging, I often get errors such as  "Children could not be evaluated", and "Error: cannot obtain value" from the debugger (when looking at the this pointer for instance). Now, I'm used to this sort of thing when dealing with mixed mode code, but who wants this pain in their unit tests? The assert issues have been covered by one of your previous posters, and using his C++ assert wrapper does resolve these kinds of issues, but again, why would you want to do this when every other c++ unit test framework already does this for you? Most of these issues that I listed can be worked around, but at the end of the day MsTest was never designed for native C++ unit testing, and the flaws are many. As there are so many other better solutions, I cannot understand why anyone would chose to use MsTest in a production environment. For one developer working alone, sure. For a team in a company, or code that will be maintained by others, it's a bad choice.

  • Anonymous
    March 23, 2011
    Ben - the only reason we chose to stick with MSTest is code coverage. Visual Studio gives you detailed code coverage stats so you can find the 'dark corners' in your C++ classes. I cannot find any other integrated Visual Studio product that gives us this. We use GoogleMock for mocks. John - when you use this static lib approach, do you still get code coverage stats in Visual Studio 2010?

  • Anonymous
    March 24, 2011
    Sean, you can get code coverage using static libs. However, since the code coverage is for the unit test DLL, you also get coverage information on your unit tests. I have another blog post I'm almost finished writing that shows how we handle these issues. And with SP1, code coverage coloring is back.

  • Anonymous
    March 24, 2011
    Thanks John - I'm looking forward to that next post of yours.

  • Anonymous
    March 31, 2011
    Hmm. I gave this a try, and I am having problems getting it to work. In particular, I'm using std::string in my production code, and when I include my header, which itself includes <string>, I get linker metadata errors presumably because it gets compiled once as native code and once as CLR: 1>XML Database.lib(XML Database.obj) : warning LNK4075: ignoring '/EDITANDCONTINUE' due to '/INCREMENTAL:NO' specification 1>LINK : warning LNK4098: defaultlib 'MSVCRTD' conflicts with use of other libs; use /NODEFAULTLIB:library 1>MSVCMRT.lib(locale0_implib.obj) : error LNK2022: metadata operation failed (8013118D) : Inconsistent layout information in duplicated types (std._String_const_iterator<char,std::char_traits<char>,std::allocator<char> >): (0x02000019). 1>MSVCMRT.lib(locale0_implib.obj) : error LNK2022: metadata operation failed (8013118D) : Inconsistent layout information in duplicated types (std.basic_string<char,std::char_traits<char>,std::allocator<char> >): (0x0200003d). 1>MSVCMRT.lib(locale0_implib.obj) : error LNK2022: metadata operation failed (8013118D) : Inconsistent layout information in duplicated types (std.basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >): (0x02000063). 1>MSVCMRT.lib(locale0_implib.obj) : error LNK2022: metadata operation failed (8013118D) : Inconsistent layout information in duplicated types (std._String_iterator<char,std::char_traits<char>,std::allocator<char> >): (0x02000080). 1>MSVCMRT.lib(locale0_implib.obj) : error LNK2022: metadata operation failed (8013118D) : Inconsistent layout information in duplicated types (std._String_val<char,std::allocator<char> >): (0x02000081). 1>MSVCMRT.lib(locale0_implib.obj) : error LNK2022: metadata operation failed (8013118D) : Inconsistent layout information in duplicated types (std._String_val<wchar_t,std::allocator<wchar_t> >): (0x02000083). 1>LINK : fatal error LNK1255: link failed because of metadata errors Any hunch about what I'm doing wrong or how to fix it?

  • Anonymous
    September 16, 2011
    Is there any way to test private methods?

  • Anonymous
    October 15, 2011
    I wrote a Load() to load a text file. There's a test run error when I tried to test it. What can I do? The test adapter 'UnitTestAdapter' threw an exception while running test 'TestLoad'. Exception has been thrown by the target of an invocation. Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

  • Anonymous
    October 17, 2011
    The comment has been removed

  • Anonymous
    October 17, 2011
    @Cathal. I have seen this problem, and there are a couple of things you can do. First, try upgrading to SP1 on your build server. There was a bug that sometimes caused test runs to fail on the build server. Second, you can view the test results from you build sever by expanding the test run results and clicking on the View Test Results link in the build summary. Finally, you can use the TestContext.WriteLine method to add extra information to the test results, which might help you track down the problem. When we were getting intermittent failures on our build server, it wasn't always the same test. I don't think we've run into this problem since we upgraded the build server to SP1.

  • Anonymous
    October 17, 2011
    @Carl. I'm not sure what you're asking. It sounds like there might be a bug in the test code that you wrote. The test adapter can't catch all C++ errors, so it does crash if the C++ failure is too severe.