Partager via

Tips for writing an Interop Debugger

  • Article

I've had a growing number of people inquire about how to write an interop-debugger with ICorDebug. I strongly advise anybody considering writing an interop-debugger to reconsider for reasons listed here. However, for those who can not be dissuaded, and promise to do something really really cool, here are some key tips:

  1. Interop-debugging means both the managed + native debugging engines coexist, and your debugger stitches them together to try to make it look like a unified debugging operation.

  2. Supply an ICorDebugUnmanagedCallback to ICorDebug::SetUnmanagedHandler. This callback is then invoked to dispatch DEBUG_EVENT structures (which are normally retrieved by WaitForDebugEvent in native debugging).

  3. You're extremely restricted for what you can do during this callback. Specifically, most of the ICorDebug API can't be called. In .NET 2.0, we started enforcing this better by checking and returning CORDBG_E_CANT_CALL_ON_THIS_THREAD.  Never block in this thread. Just update your state and get out.

  4. Most importantly, you can't do an in-band continue on the thread dispatching the ICorDebugUnmanagedCallback. This means the unmanaegd callback thread has to get another thread (perhaps the thread that continues for managed events) to make the in-band continue. There's actually a good technical reason for this; although it's not intuitive (witness this forum post)

  5. If the callback is for an Out-Of-Band event, just update internal state and immediately continue it.

  6. For Whidbey, use ICorDebugProcess2::SetUnmanagedBreakpoint and ClearUnmanagedBreakpoint instead of setting the raw int3s yourself.

  7. Don't call Native debug APIs directly. Instead, call the corresponding API on ICorDebug.
     

    Native Debugging API from kernel32.dll What to use on ICorDebug
    CreateProcess ICorDebug::CreateProcess with DEBUG_ONLY_THIS_PROCESS flag set. Note that child-process debugging (DEBUG_PROCESS && !DEBUG_ONLY_THIS_PROCESS) is not supported.
    DebugActiveProcess ICorDebug::DebugActiveProcess, with win32Attach = true.
    WaitForDebugEvent Dispatched via the ICorDebugUnmanagedCallback callback, which was set via ICorDebug::SetUnmanagedHandler. Pay close attention to in-band vs. out-of-band events.
    ContinueDebugEvent ICorDebugProcess::Continue. Call ICorDebugProcess::ClearCurrentException  to get DBG_CONTINUE. Else this acts as DBG_EXCEPTION_NOT_HANDLED
    Set/GetThreadContext ICorDebugProcess::Set/GetThreadContext. Only use when the thread is in native code and stopped at an inband event.
    Write/ReadProcessMemory ICorDebugProcess::Write/ReadMemory.Use ICorDebugProcess2::SetUnmanagedBreakpoint and ClearUnmanagedBreakpoint instead of writing int3 for native breakpoints.
    DebugActiveProcessStop Not supported.
    Suspend/ResumeThread Use directly, but only when the debuggee is stopped and the thread is in native code.
    DebugBreakProcess ICorDebugProcess::Stop.
    DebugSetProcessKillOnExit Not supported.

    This is because in order to make interop-debugging work, ICorDebug needs to intercept the native debugging APIs. Note that the native debugging APIs should only be used in native code. Use managed debugging APIs for operations in managed code. Since threads can cross between managed and native, you may need to figure out where the thread is (via a callstack) to determine which operation to use.

  8. Remember that ICorDebug is for debugging managed code, not native. So the ICorDebug* object hierarchy represents managed constructs, not native ones. For example, native threads don't get an ICorDebugThread object, and native modules don't get an ICorDebugModule object. This is why you'll notice above that all the native-API equivalents are on ICorDebug or ICorDebugProcess.  In contrast, since managed constructs are usually built on native ones, you may see native debug objects for managed elements. For example, a managed thread is built on a native thread, so you'll see a native thread entry for managed threads.

And don't forget there's the "Building Development Tools for .Net" forum for further questions.

Comments

  • Anonymous
    January 29, 2007
    Hi Mike, do you know a way to modify existing pdbs to preserve the debugging experince when doing a full ILDASM/ILASM round trip like here: http://geekswithblogs.net/akraus1/archive/2007/01/23/104288.aspx When I load the changed dlls into the debugger my breakpoints stay gray. I guess this is because the debugger compares some hashes at module level which I would have to change to get a decent debugging experience. Is that the only obstacle or are more things lurking inside a pdb file? Yours,    Alois Kraus

  • Anonymous
    January 31, 2007
    The comment has been removed