Virtual Kernel Mode Debugging

A while back I discovered that it was possible to do kernel mode debugging of an OS on a virtual machine (VM) running under Virtual PC using a single computer. Even if you are not concerned with debugging a device driver, kernel mode debugging can be useful for debugging applications on Windows Embedded Standard when a failure occurs in the system rather than the application itself. I have personally used this technique to verify that a Windows Embedded Standard runtime OS has at least began the boot process. I highly recommend the following book for more information on debugging: Advanced Windows Debugging (Addison-Wesley Microsoft Technology Series), by Mario Hewardt, and Daniel Pravat.

Typically for kernel mode debugging, we use two physical devices with a null modem cable connected between the serial ports. In Virtual PC, we can create a virtual null Mmodem cable between the development machine and the Target VM using Named Pipes.

The steps below will walk you through how to setup and begin kernel debugging from a development machine running WinDbg (available here) and a Target VM running a Windows Embedded Standard runtime OS.

Configure the Virtual Null Modem cable

1. Run the Virtual PC Console on the development machine by selecting Start-All Programs-Microsoft Virtual PC

2. Select your Target VM and click Settings

3. In the Settings dialog click COM1

4. Select Named Pipe and enter “\\.\pipe\mydebugpipe” for the value (without the quotes)

5. Click OK

Configure the Target VM for Kernel Debugging

6.  In your Target VM add the following line to the c:\boot.ini file (either boot the VM and use notepad or use the VhdMount utility available with Microsoft Virtual Server). If you have the Server Command Line Tools in the target OS Configuration you can see how to use bootcfg.exe to modify the boot.ini file here.

Note that this is one line in the file.

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows Embedded Standard" /noexecute=optin /fastdetect /DEBUG /DEBUGPORT=COM1 /BAUDRATE=115200

7. Shutdown the Target VM

Launch WinDbg

We are going to use WinDbg to kernel debug the Target VM. Download and install it now if you do not have it already.

8. On the Development machine, open a Command Prompt

9. Launch WinDbg using the following command

<install-path-of-windbg>\windbg -k com:pipe,port=\\.\pipe\mydebugpipe,resets=0,reconnect

Debug with WinDbg

10. Start the Target VM from the Virtual PC console and select the Microsoft Windows Embedded Standard [debugger enabled] operating system to start

11. At this point, WinDbg will connect and start to spew some information to the screen

12. In WinDbg, select Debug->Break

13. Now select View->Call Stack from the WinDbg menu to see the current call stack of the OS

Well, I hope this introduction to kernel mode debugging was interesting and enjoyable. I did spare you the need to have two machines and string a null modem cable between the two. With this new knowledge, I suspect that you will find, or create, a need to do some kernel mode debugging in the near future.

- Preston

Technorati Tags: XPe,Standard 2009