Share via

WARNING: VS 2012 might break your broken P/Invoke

  • Article

I'm seeing many people reporting that they are seeing strange P/invoke issues when they moved their code to VS 2012. Typically, they have P/invokes like this:

 [DllImport("Win32Project2.dll", PreserveSig = true, CharSet = CharSet.Unicode)]
static extern int MyPInvoke(out string ret);

If you attach a native debugger (or enable native debugging), with the right symbols (public symbol would be just fine), you would see that it is actually crashing inside combase.dll!CoTaskMemFree:

>    ntdll.dll!_RtlReportCriticalFailure@8()  + 0x33 bytes   
     ntdll.dll!_RtlpReportHeapFailure@4()  + 0x21 bytes   
     ntdll.dll!_RtlpLogHeapFailure@24()  + 0xa2 bytes   
     ntdll.dll!_RtlFreeHeap@12()  + 0x2677b bytes   
     combase.dll!CoTaskMemFree(void * pv)  Line 475    C++

Problem like this are usually caused by a mismatch between the managed declaration and the native implementation. The usual suspect here is the 'out string' signature. In .NET world, having a out string in P/invoke means:

  1. Native code pass a LPWSTR out to managed code, allocating the memory using CoTaskMemAlloc
  2. CLR creates a managed System.String instance from LPWSTR, and frees the LPWSTR using CoTaskMemFree (because the ownership has already transferred from native code to managed code, which the 'out' implies)

So, if MyPInvoke pass a LPWSTR that is not allocated from CoTaskMemAlloc, say from HeapAlloc, new, or a string literal, you are potentially asking CLR to do CoTaskMemFree on your pointer, which would result in undefined behavior.

Basically, this means the P/invoke was already broken before.

But why it didn't crash before and suddenly starts to crash now after upgrading?

This is exactly the kind of thing you could expect with undefined behavior, because that could change anytime. It turns out in Vista, Windows guys decided to make CoTaskMemFree a tiny bit better: by actually reporting errors. Before that, since it returns void, it would silently fail if a random pointer is passed to CoTaskMemFree (because CoTaskMemAlloc has some book keeping, so in 99.99% of the case it would know it is not theirs). In Vista, if the app is indeed compiled for Vista, which means its subsystem version would be 6.0 in the PE file, you would get the better behavior: you would crash!

How's VS 2012 involved in this puzzle? It actually pass /subsystemversion:6.0 to C# compiler (this is a new switch), which would enable this new crash-if-error behavior.

Fixing this issue is actually pretty straight-forward:

  1. If you can change the native implementation, change the native function to allocate the memory using CoTaskMemAlloc
  2. Otherwise, you'll need to change the P/invoke declaration in managed code. Since the only there are probably a thousand kind of allocators out there, and we only support one (CoTaskMemAlloc/CoTaskMemFree), your best bet is to pass it as IntPtr, and then free it (or simply not free it, if it is a const string, for example) yourself, either by using one of the Marshal APIs, or expose the specific free function of the allocator using another P/invoke.

For more information, you can refer to an MSDN article that I wrote about 4 years ago here: https://msdn.microsoft.com/en-us/magazine/cc164193.aspx

Comments

  • Anonymous
    January 24, 2013
    Hi, thanks for sharing this. I have one question though, does it affect code where instead of out string I pass StringBuilder ? Thanks!

  • Anonymous
    January 25, 2013
    @pinvokenoob If you are passing 'out StringBuilder', the same rule applies. The native side of signature would be 'wchar_t **' If you are passing 'StringBuilder', the native signature would be 'wchar_t '. There are no memory ownership transfer in this case. CLR owns the wchar_t * buffer (it comes from pinned StringBuilder), and native code writes to that wchar_t buffer.