CLR SPY and Customer Debug Probes: The PInvoke Calling Convention Mismatch Probe

Defining a PInvoke signature and using DllImportAttribute correctly can be difficult to do, and you normally get little-to-no diagnostic information if you make a mistake.  Some mistakes that can be made with PInvoke would be impossible for the CLR to detect, but many mistakes pass through without validation because PInvoke is designed for high-performance access to unmanaged APIs.  Thanks to the PInvoke Calling Convention Mismatch probe, however, at least one source of errors can now be caught.

One of the named parameters that can be set on DllImportAttribute is called CallingConvention, which can be used to specify the calling convention of the unmanaged DLL export.  The following enumeration values can be used (defined in the System.Runtime.InteropServices namespace):

• CallingConvention.Cdecl.  The caller is responsible for cleaning the stack.
• CallingConvention.StdCall.  The callee is responsible for cleaning the stack.
• CallingConvention.ThisCall.  Used for calling unmanaged methods defined on a class.
• CallingConvention.Winapi.  This isn't a real calling convention; it's an alias for the platform's default calling convention. On Windows (excluding Windows CE), the default calling convention is StdCall.

DllImportAttribute assumes CallingConvention.Winapi if none is specified, so users of Win32 APIs typically don't need to specify any calling convention.

Suppose, however, that you want to use PInvoke to call the C runtime library's Bessel function _j0 because you're interested in electromagnetic wave theory, yet there's no equivalent managed API for this.  (Not surprisingly, I've never heard anyone complain about this omission!)  You might write C# code like the following:

  [DllImport("msvcr71.dll")] // There's a bug here!

  static extern double _j0(double x);

  public static void Main ()

  {

  double result = _j0(2.345);

  }

With the PInvoke Calling Convention Mismatch probe enabled, you would get the following error message when running the program:

  Stack imbalance may be caused by incorrect calling convention for method _j0 (msvcr71.dll)

This probe reports such errors whenever it detects that the calling convention of a PInvoke signature does not match that of the target unmanaged method.  The problem here is that the CLR treats the function _j0 as if it has the Winapi calling convention (since none was explicitly specified) yet the header file for this function (math.h) shows that it really has the Cdecl calling convention:

  _CRTIMP double __cdecl _j0(double);

Therefore, the correct managed definition for the _j0 function would have been the following:

  [DllImport("msvcr71.dll",

    CallingConvention=CallingConvention.Cdecl)]

  static extern double _j0(double x);

Without this probe enabled, this type of error can be very easy to make since, depending on the exact calling convention mismatch, the CLR may still recover without any problems!  But in general, this is a serious problem that could cause stack corruption.  With this probe enabled, the CLR performs various heuristics to determine if the callee's behavior doesn't match the calling convention that the CLR is told to follow.  Note that sometimes this probe detects a signature problem other than an incorrect calling convention, if it still causes a stack imbalance.  But when you see a message from this probe, you know there's some kind of bug present!

Because this this is an "error probe," you can force a debug break whenever this situation is detected.  This is the "Break on Error Messages" feature in CLR SPY.

I also want to repeat that you'll see this probe reporting a problem when running Windows Forms applications that take advantage of the new v1.1 Application.EnableVisualStyles feature (which gives you Windows XP themes without the use of a manifest).  This is due to a bug in a PInvoke signature inside System.Windows.Forms for the Win32 DeactivateActCtx API.  There are 3 workarounds:

1. Disable the PInvoke Calling Convention Mismatch probe, or
2. Uncheck "Break on Error Messages" in CLR SPY so you can ignore this message and not provoke the crash, or
3. Use an XML manifest to enable XP themes, rather than using the EnableVisualStyles API.

Comments

• Anonymous
  June 04, 2004
  But when you see a message from this probe, you know there's some kind of bug present!
• Anonymous
  July 27, 2004
  The comment has been removed
• Anonymous
  November 02, 2005
  The comment has been removed