Share via

Inside 'Image File Execution Options' debugging

  • Article

There are times that you need to debug the startup code for an application, but something else is launching the application. Classic examples might be services or setup custom actions. Luckily, the OS team came up with a way to debug these problems: 'Image File Execution Options' debugging. Junfeng Zhang blogged about this a while back. Today, I wanted to talk in more detail about how to use Image File Execution Options with Visual Studio, and a bit more details about how it works.

Using Image File Execution Options with VS 2005

Setup:

  1. Run regedit.exe
  2. Goto HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options
  3. Create a new key for your exe (example: foo.exe)
  4. Create a new string value under your exe. The name of the string value is 'Debugger', and the value is 'vsjitdebugger.exe'

Here is a sample registry script to do this:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\sample.exe]
"Debugger"="vsjitdebugger.exe"

Using Image File Execution Options with Visual Studio v7.x

MSDN gives a good explanation for how to set this up.

How does Image File Execution Debugging actually work

The operating system implements this feature inside the user-mode part of CreateProcess. What happens is that after the operating system determines what executable is going to be started it checks the registry key for a debugger. If it finds one, it will launch the debugger instead of the application. The only way to skip this is to call CreateProcess with DEBUG_PROCESS. The debugger is then expected to launch the original application again, but this time under a debugger.

This has some very interesting implications:

  • Although this feature is for debugging, it is really just a general mechanism of redirecting what application gets launched, which is kind of interesting.
  • It becomes very difficult to start the application _outside_ of a debugger. Usually trying to start the application outside of the debugger will just launch an instance of the debugger.
  • This can confuse whatever called CreateProcess. Instead of getting back the handle/process id of the debuggee, it will instead get back the handle/process id of the debugger. Probably not what was expected.
  • On Win 64, there are two copies of HKEY_LOCAL_MACHINE\Software (one for 32-bit apps, and one for 64-bit apps), and therefore there are two copies of these options. However, where the operating system looks isn't dependant on the bit-ness of the application that is going to be debugged (which is what you would probably expect). Instead, it is dependant on the bit-ness of the application that called CreateProcess.

Limitations of Image File Execution Debugging

Visual Studio 7.x had very basic support for image file execution options:

  • To debug services or other 'session 0' processes, someone must be logged onto the console.
  • Only works for native or native+managed debugging
  • Only works if the process you are trying to debug is visible (example: does not work for services unless they are allowed to interact with the desktop).
  • Only works if the process you are trying to debug is an administrator or member of the 'debugger users' group
  • Only works if devenv.exe is on the path
  • Does not work for Ctrl-F5
  • If you detach from the started process and shutdown Visual Studio, you might confuse whatever started the debugged application.

VS 2005 got rid of most of these. What is left:

  • To debug services or other 'session 0' processes, someone must be logged onto the console.
  • Only works on 32-bit computers

In VS 2005, the other thing to be careful of is that debugger will guess what kind of code to debug by looking at the exe, so if you have a native exe which loads managed code, and you want to debug the managed code, make sure that you check the 'Manually choose the debug engines' checkbox when the Just-In-Time debugger dialog appears.

Comments