x64 calling convention and the disappearing process syndrome

Raymond Chen describes the parameter passing aspect of the x64 calling convetions.   But there is more than parameter passing to the calling convention.   Exception handling is an important part of the calling convention.

A function that calls another function or needs to allocate stack space or requires exception handling (e.g. has a try statement) must have a prolog and an epilog.  It also has to have an entry in a special function-table.   The function table includes unwind information – information that enables the exception-handling routings to unwind the stack and undo the effect of the function prolog.    In order for exception handling to work, there are limitations on function prolog and epilog.

The fun begins when a function does not have correct unwind information.  If in addition to that, there is no debugger attached to the process, the system notifies the Win32 sub-system about the exception.   The Win32 sub-system will simply kill the process. You will not see any Watson or JIT debugger dialog box.   The process will just disappear.

This happened to me last week.  I had an assembly thunk function that called some C++ code that had a race condition (which seems to happen only when a debugger is not attached).   Debugging would have been much easier if I did not have an assembly thunk that did not play by the rules and did not have a function-table entry.

Comments

• Anonymous
  August 24, 2004
  You need to create a function table somehow and call RtlAddFunctionTable. If the code is very dynamic, use RtlInstallFunctionCallback.
  
  I don't actually own a 64-bit system but I've given it a little study. I'm a Pocket PC programmer mostly; Windows CE doesn't offer any way to dynamically add to the exception table.