Share via

Out-of-band events and Mixed-mode debugging

  • Article

Interop-debugging splits all debug events into "In-band (IB) " and "Out-of-band (OOB)". Inband events are the good ones: at an inband event, the debuggee is stopped and you can use the rest of the ICorDebug API normally to do inspection and other things like setup funcevals (aka property-evaluation). This is because the debugger's helper thread (which is needed to do all interesting managed-debugging operations) can still execute at an IB event. All managed debug events are effectively IB events (this unification is one of the goals of interop-debugging). Most native exception debug events in user code are also IB events. This is why you can stop at a native breakpoint and run a managed callstack.

However, some native events block the helper thread (such as a native exception on the helper thread). Such events become OOB. Once ICorDebug dispatches an OOB event (via ICorDebugUnmanagedCallback::DebugEvent when fOutOfBand = true), then almost the entire debug ICorDebug API is blocked until that event is continued. This is because OOB events block the helper thread, and most ICD APIs are blocked on the helper-thread (blocking is a transitive operation).
In other words, you may be in the middle of an innocent API like ICorDebugThread::GetAppDomain() and that API may suddenlu block because a callback thread just dispatched an OOB event. The API will be blocked until you continue the OOB event (via calling ICorDebugProcess::Continue(fIsOutOfBand=true) ).  To make matters worse, the timing here is completely nondeterministic and therefore this can be a source of races.

Another thing is that OOB events can come at any time. They can come when the process is starting up. They can come in the middle of a managed attach. They can come in the middle of calling Continue. They can even come when you think the process is stopped at an inband event!  I repeat, OOB events can come at any time.  (IMO, ICorDebug's API design around interop-debugging does not properly deal with OOB events, but that's another issue.)

When a debugger gets an OOB event, the only thing it can safely do is update internal bookkeeping and then continue the process. It is very strongly recommended that interop-debuggers always immediately continue the OOB event before returning from the callback. Un-continued OOB events have been the source of quite a few hangs in the debugger.

Unfortunately, this all makes it difficult to write an interop-debuggers properly because:
1) The threading model introduces reentrancy issues.
2) OOB events breaks the traditional debugger model of a unified event stream.

Comments

  • Anonymous
    September 18, 2005
    ICorDebug (ICD) in managed-only debugging mode does not need to be a reentrant API. In other words, you...
  • Anonymous
    January 15, 2007
    I've had a growing number of people inquire about how to write an interop-debugger with ICorDebug. I