Managed and Native Interop - C# or C++/CLI?

This is a topic that I've been thinking a lot about lately. And I would really like to get all of your opinions as well (especially if you disagree with me).

This debate came up recently (again) on my trip to Tech Ed Developer in Barcelona. If I am having to write interop code, should I use C# and p/invoke, or should I use C++/CLI? I had some very smart people like Kate Gregory and Daniel Moth to chat with this about. Kate and Daniel are definitely good people to chat with because, honestly, they have completely different opinions on this topic. Kate is on the C++/CLI side of things, especially for serious interop tasks, while Daniel believes that you should keep it in C# no matter how complicated that C# gets. So, what side of the fence do I fall on?

This is a question I've been pondering for a while now. Since I first came in to this Windows Server 2008 Technical Evangelist, I've had to write a lot of interop code. And I personally believe in using the right tool for the job. And while p/invoke is great for smaller interop jobs between native code and managed code, it can "get out of hand" quickly. For instance, to p/invoke into the new Transactional NTFS APIs, I could look at having to create a bunch (several hundred lines of code or many more) of interop structures merely to call DeviceIoControl to get issue a single TxF command. Here's the thing: all those structures are all ready defined on the native side and could be used with ease (as in, a simple "#include" statement) on the C++/CLI side of things. No extra code necessary.

But it's not just the structure definitions that are the problem, it is also the programming paradigm used in the native code you are trying to interop with. When dealing with a simple Windows API function, this may not be a problem at all. Of course, it gets a little scary when dealing with COM interop, but at least there are still the tools built into C# to interop with COM. Where the train really starts to derail is when you start writing interop code for a native library that uses a different paradigm.

For instance, let's look at DeviceIoControl. It's not really one function call, it's a complete API in itself. I've had to get familiar with DeviceIoControl when doing my work with Transactional NTFS. DeviceIoControl uses a much more conversational paradigm than managed developers are perhaps used to. Here's a common "calling session" with DeviceIoControl (minus all the fun exception handling that should be there):

 while (true)
{
    // Clean-up previously allocated memory

    // Allocate memory

    // Issue command
    if (!DeviceIoControl(...))
    {
        if (GetLastError() == ERROR_MORE_DATA)
        {
            // Determine how much more memory we need

            continue;
        }
    }
    else
    {
        // There was enough memory finally

        // Pull data out of buffer we issued to DeviceIoControl

        break;
    }
}

When writing managed code against this DeviceIoControl paradigm, you will write more code that is harder to maintain and debug than you will if you simply used C++/CLI to do the interop. With C++/CLI you can use the API natively exactly like it's meant to be used, and then "expose" the code to the managed world via C++/CLI. When I moved my Transaction NTFS wrapper from C# to C++/CLI, I was able to remove a large chunk of code, and it was much easier to read and understand. Using the right tool for the job will do that for you.

The DeviceIoControl paradigm is just one example though. Another "hang up" when writing interop code in C# is when you need to do some very detailed memory and pointer manipulation. Take this following piece of code that is used within the DeviceIoControl pattern above and is used to get the list of running transactions on a system (assuming the "list" has already been populated):

 // Loop through the transaction entries and store them in a list
TXFS_LIST_TRANSACTIONS_ENTRY *txEntry;
txEntry = (TXFS_LIST_TRANSACTIONS_ENTRY *)((char *)txList + sizeof(TXFS_LIST_TRANSACTIONS));
for (DWORD i = 0; i < txList->NumberOfTransactions; i++)
{
    TRANSACTION_ENTRY *newEntry = AddTxFEntry(txEntry->TransactionId, txEntry->TransactionState);

    // Each transaction is attempted to be loaded with their files ASAP (if any).
    LoadLockedFiles(rmDirectory, newEntry);
    txEntry++;
}

This code is a PITA to write in C#. Not only do you have to write all the interop structures for TXFS_LIST_TRANSACTIONS_ENTRY, TXFS_LIST_TRANSACTIONS, and TRANSACTION_ENTRY, but you have to do all the pointer fun via the Marshal helper in C#. Sure, you could pop into unsafe code and deal with pointers directly, but I think if you are willing to do that, you should be more than willing to just use C++/CLI. Why would I go through all that, if I can just write the code and expose the results to managed code via C++/CLI? As soon as I switched the wrapper from C# to C++/CLI, my job got a LOT easier. And this is coming from a programmer that is relatively new to the whole C++ world. If I can write in C++/CLI for interop, I honestly believe any committed C# developer can as well.

Let's look at a small snippet from the Transactional NTFS Wrapper:

 FileStream^ TransactedFile::Open(System::String ^path, FileMode mode, FileAccess access, FileShare share) {
                
    ...

    ////////////////////////////////////////////////////////////////////////
    // We need to marshal our values into native types since we are
    // going to call into a native Windows API based on CLR parameters
    marshal_context ctx;
    const wchar_t *marshalledPath = ctx.marshal_as<const wchar_t *>(path);

    HANDLE fileHandle = CreateFileTransacted(marshalledPath, 
        marshal_as<DWORD>(access), 
        marshal_as<DWORD>(share),
        NULL, 
        marshal_as<DWORD>(mode), 
        FILE_ATTRIBUTE_NORMAL, 
        NULL, 
        kernelTransactionHandle, 
        NULL, 
        NULL);
    
    ...
}

This is the actual segment of code that does the main interop piece for me. Is the above that intimidating? A couple of things to note to those C# developers out there. You can think of the "^" as denoting "by reference". Yes, it's a bit more complicated than that, but just think of it that way for now, that's all you need to know. And the "::" operator is your namespace operator. So instead of "System.String", I have "System::String". Once again, a very simple syntactic difference.

The marshal_context object is a new object introduced with Visual C++ 2008 as part of the new marshalling library. By using the marshal_context, the context will take care of memory allocation and de-allocation for the strings we are are interop'ing. Very simple. The CreateFileTransacted API expects the path to be a "const wchar_t *", so I simply marshal our passed System::String as "const wchar_t *" and we're good to go. Since marshal_as<> is template based, it's very easy to extend yourself. And that's what I've done with marshalling System::IO::FileMode/Access/Share as DWORD (which I will show in a future post).

Now, honestly, is that piece of code that scary? If you have questions about it or things that still scare you, please let me know and I will look at writing a future post on the topic.

Back to the debate at hand though: C# versus C++/CLI for interop. I can certainly understand where Daniel is coming from. C# developers can quickly become intimidated when seeing C++/CLI code. It isn't the C# code they know and love. But I'm not sure he gives C# developers enough credit. It's an important step for developers to realize that, syntactically, C++/CLI and C# are not that different. If you are familiar with reference types in C#, you are probably more familiar with pointers than you realize. There are some "gotchas" around topics like pinned pointers, but they are concepts you need to know in C# as well if writing interop code there, so they aren't that big of a deal in the end. I'm not saying to not use P/Invoke and C# though. I'm simply saying to use the right tool for the job. If it is a very simple Windows API you are writing interop code for, C# and P/Invoke is probably "good enough". For more complicated interop jobs though (as in with most of the new enhancements in Vista and Windows Server 2008), why not use C++/CLI since it will make your life easier one you get over the small learning curve?

Did I have this same opinion before Visual Studio 2008? Yes, and no. Yes, C++/CLI code was still built for interop, but I honestly believe that the introduction of the extensible marshal_as<> construct (combined with other technologies like STL CLR) make interop in C++/CLI even more powerful and easier to use than before. For instance, marshalling strings was not fun to do before Visual Studio 2008 (especially for a developer not extremely aware of the different ways to use strings in native code). However, out of the box, the marshal_as<> construct provides a lot of different ways to work with and to interop with strings. And in all of these ways, you don't have to worry about allocating or de-allocating memory at all (ah, the same joys I get when working in a garbage collected world). And in the case that you are interacting with C++ libraries that are possibly STL-based, STL CLR provides you with a library that you simply don't get in C# (interop'ing with C-based Windows APIs are one thing, interop'ing with proper C++ libraries are a completely different beast when using C# for the interop code).

If you are a C# developer that writes _any_ interop code, you owe it to yourself to learn C++/CLI. Trust me, it will make your life much easier, and it's not that hard to learn. And being scared of pointers is no excuse. If you are comfortable with reference types in C#, you are probably more familiar with the concept of pointers than you realize. Plus, with STL and Boost (even more so with the upcoming TR1 work), there really is no need to have a pointer anywhere in your modern C++ program (ah, the sweetness that is the shared_ptr).

So with the launch of Visual Studio 2008, I believe that the "right tool" for the "interop job" is C++/CLI for anything that is beyond trivial. Constructs and libraries like marshal_as<> and STL CLR only drive this point home more. You can feel free to continue to write thousands of lines of managed-code-copying-already-defined-in-header-files simply to p/invoke into an existing Windows API, but me? I will be using Visual C++ 2008 and enjoying every minute of it since I don't have to do any re-invention of the wheel.