A look at the WoW64 layer in the debugger
[1st edit 2 Jan 2007- updated to make the exercise slightly harder]
When I first started using Windows Server 2003 x64 as my day to day operating system of choice, I did a bit of digging to understand how the WoW64 layer provides support for 32-bit processes to run in the 64-bit world.
Here is a walkthrough of what I found, you should be able to follow the steps yourself if you are running an x64 version of Windows.
Download and install the 64 bit version of Debugging Tools for Windows appropriate for your platform from:
www.microsoft.com/whdc/devtools/debugging/install64bit.mspx
Now run the 32 bit version of notepad:
Start->Run, enter "%systemroot%syswow64notepad.exe" (without the quotes) and click Ok.
Select Start->Programs->Debugging Tools for Windows 64-bit->WinDBG
In task manager, locate the 32 bit instance of notepad.exe on the "Processes" tab. This is displayed as notepad.exe*32
Ensure the PID (process ID) column is visible. (If it is not choose "Select Columns" from the View menu and check the checkbox for "PID (Process identifier)"). Note the process ID for the notepad.exe*32 process.
In WinDBG, on the File menu select "Attach to process". Use the process ID you noted above to select the 32 bit notepad instance in the list of processes and click OK.
If a message box saying "Save workspace?" appears click "No".
If there is no MDI child window titled "Command" visible then select "Command" on the "View" menu and one should appear.
You should see output something like this:
Microsoft (R) Windows Debugger Version 6.4.0007.2
Copyright (c) Microsoft Corporation. All rights reserved.
*** wait with pending attach
Symbol search path is: srv*D:symbolssymcache*\symbolssymbols
Executable search path is:
ModLoad: 00000000`01000000 00000000`01014000 E:WINDOWSsyswow64notepad.exe
ModLoad: 00000000`78ec0000 00000000`78ff9000 E:WINDOWSsystem32ntdll.dll
ModLoad: 00000000`78be0000 00000000`78c26000 E:WINDOWSsystem32wow64.dll
ModLoad: 00000000`78b90000 00000000`78bda000 E:WINDOWSsystem32wow64win.dll
ModLoad: 00000000`78b80000 00000000`78b89000 E:WINDOWSsystem32wow64cpu.dll
ModLoad: 00000000`7d600000 00000000`7d6f0000 ntdll.dll
ModLoad: 00000000`7d4c0000 00000000`7d5f0000 KERNEL32.dll
ModLoad: 00000000`762b0000 00000000`762fa000 comdlg32.dll
ModLoad: 00000000`77f50000 00000000`77fec000 ADVAPI32.dll
ModLoad: 00000000`7da20000 00000000`7db00000 RPCRT4.dll
ModLoad: 00000000`7dbd0000 00000000`7dcd3000 COMCTL32.dll
ModLoad: 00000000`77ba0000 00000000`77bfa000 msvcrt.dll
ModLoad: 00000000`7d800000 00000000`7d890000 GDI32.dll
ModLoad: 00000000`7d930000 00000000`7da00000 USER32.dll
ModLoad: 00000000`77da0000 00000000`77df2000 SHLWAPI.dll
ModLoad: 00000000`7c8d0000 00000000`7d0d3000 SHELL32.dll
ModLoad: 00000000`73070000 00000000`73097000 WINSPOOL.DRV
ModLoad: 00000000`4b8d0000 00000000`4b921000 MSCTF.dll
ModLoad: 00000000`7df50000 00000000`7dfc0000 UxTheme.dll
ModLoad: 00000000`77d00000 00000000`77d8c000 OLEAUT32.dll
ModLoad: 00000000`77670000 00000000`777a4000 ole32.dll
(8e0.e70): Break instruction exception - code 80000003 (first chance)
ntdll!DbgBreakPoint:
00000000`78ef3320 cc int 3
In the command area at the bottom of the command window enter the following command:
.symfix +c:symcache
This tells the debugger to download needed symbols from Microsoft and to cache them locally in c:symcache so they are not downloaded a second time.
Notepad is currently stopped by the debugger at a breakpoint. Notepad will not respond to user interaction.
In the command area, enter the following command:
g
Notepad should now be responsive again.
From the Format menu of Notepad select "Font". The font dialogue should appear.
Switch back to WinDBG and select "Break" from the Debug menu.
This interupts Notepad at a breakpoint on thread 1 which in this case is the "debugger thread", a special thread that the debugger injects into the debuggee (notepad in this case).
Now enter the following command to switch the debugger context to thread 0:
Thread 0 is the only other thread in this process and is the thread on which the main user interface of Notepad is running. This is the thread that has made a system call to display the Font dialog and is awaiting input.
You should see output like the following:
0:001> ~0s
SYMSRV: +c:symcache*msdl.microsoft.com/download/symbols needs a downstream store
*** ERROR: Symbol file could not be found. Defaulted to export symbols for E:WINDOWSsystem32wow64cpu.dll -
wow64cpu!TurboDispatchJumpAddressEnd+0x684:
00000000`78b842d9 c3 ret
Enter the following command to output the call stack of thread 0:
kp
You should see output like the following:
0:000> kp
Child-SP RetAddr Call Site
00000000`0007ede8 00000000`78b8428e wow64cpu!TurboDispatchJumpAddressEnd+0x684
*** ERROR: Symbol file could not be found. Defaulted to export symbols for E:WINDOWSsystem32wow64.dll -
00000000`0007edf0 00000000`78be6a5a wow64cpu!TurboDispatchJumpAddressEnd+0x639
00000000`0007ee60 00000000`78be5e0d wow64!Wow64SystemServiceEx+0x2ca
00000000`0007ee90 00000000`78ed8501 wow64!Wow64LdrpInitialize+0x2ed
00000000`0007f6c0 00000000`78ed6416 ntdll!LdrpInitializeProcess(struct _CONTEXT * Context = 0x00000000`00000000, void * SystemDllBase = 0x00000000`00000000)+0x17d9
00000000`0007f9d0 00000000`78ef3925 ntdll!_LdrpInitialize(struct _CONTEXT * Context = 0x00000000`00000000, void * SystemArgument1 = 0x00000000`00000000, void * SystemArgument2 = 0x00000000`00000000)+0x18f
00000000`0007fab0 00000000`78d59630 ntdll!KiUserApcDispatch(void)+0x15
00000000`0007ffa8 00000000`00000000 0x78d59630
*** ERROR: Module load completed but symbols could not be loaded for E:WINDOWSsyswow64notepad.exe
00000000`0007ffb0 00000000`00000000 0x0
00000000`0007ffb8 00000000`00000000 0x0
00000000`0007ffc0 00000000`00000000 0x0
00000000`0007ffc8 00000000`00000000 0x0
00000000`0007ffd0 00000000`00000000 0x0
00000000`0007ffd8 00000000`00000000 0x0
00000000`0007ffe0 00000000`00000000 0x0
00000000`0007ffe8 00000000`00000000 0x0
00000000`0007fff0 00000000`00000000 0x0
00000000`0007fff8 00000000`00000000 0x0
00000000`00080000 00000020`78746341 0x0
00000000`00080008 00005370`00000001 0x20`78746341
Note that the addresses in the Child-SP and RetAddr columns are 64-bit addresses, like "00000000`0007ede8"
Note also in the Call Site column that the functions nearest the top of the call stack are functions in the wow64 module and the wow64cpu module. The lower part of the call stack is not correctly resolve due to a lack of symbols in this case.
However that is not important. The main point to observe here is that at a system level, the 32 bit process is actually a 64 bit process that uses the Wow64 layer to accept 32 bit system calls from the program's 32-bit EXE and DLLs and forward on those calls to the 64-bit operating system.
Now enter the following command to list the modules loaded in the process:
lm
You should see output like the following:
0:000> lm
start end module name
00000000`01000000 00000000`01014000 notepad (deferred)
00000000`4b8d0000 00000000`4b921000 MSCTF (deferred)
00000000`73070000 00000000`73097000 WINSPOOL (deferred)
00000000`762b0000 00000000`762fa000 comdlg32 (deferred)
00000000`77670000 00000000`777a4000 ole32 (deferred)
00000000`77ba0000 00000000`77bfa000 msvcrt (deferred)
00000000`77d00000 00000000`77d8c000 OLEAUT32 (deferred)
00000000`77da0000 00000000`77df2000 SHLWAPI (deferred)
00000000`77f50000 00000000`77fec000 ADVAPI32 (deferred)
00000000`78b80000 00000000`78b89000 wow64cpu (export symbols) E:WINDOWSsystem32wow64cpu.dll
00000000`78b90000 00000000`78bda000 wow64win (deferred)
00000000`78be0000 00000000`78c26000 wow64 (deferred)
00000000`78ec0000 00000000`78ff9000 ntdll (deferred)
00000000`7c8d0000 00000000`7d0d3000 SHELL32 (deferred)
00000000`7d4c0000 00000000`7d5f0000 KERNEL32 (deferred)
00000000`7d600000 00000000`7d6f0000 ntdll_7d600000 (deferred)
00000000`7d800000 00000000`7d890000 GDI32 (deferred)
00000000`7d930000 00000000`7da00000 USER32 (deferred)
00000000`7da20000 00000000`7db00000 RPCRT4 (deferred)
00000000`7dbd0000 00000000`7dcd3000 COMCTL32 (deferred)
00000000`7df50000 00000000`7dfc0000 UxTheme (deferred)
Note that there are two copies of NTDLL loaded in the process:
00000000`78ec0000 00000000`78ff9000 ntdll (deferred)
00000000`7d600000 00000000`7d6f0000 ntdll_7d600000 (deferred)
This is quite unusual, to have two copies of a module loaded in the process.
It happens here because we have loaded both the 32 and 64 bit versions of NTDLL.DLL
Exercise (answer at the end of the post):
How can you tell which NTDLL is which?
Wow64Exts Debugger Extension
You are now going to use a special debugger extension to look at the 32-bit process as if you were using a 32-bit debugger. Enter the following command to load the debugger extension:
!load wow64exts
Enter the following command to tell the debugger extension to switch to x86 (32-bit) mode:
!sw
You should see output like the following:
0:000> !sw
Switched to x86 mode.
Enter the following command to dump the call stack again. Note it is the same command that you used before:
kp
You should see output similar to the following:
0:000:x86> kp
*** ERROR: Symbol file could not be found. Defaulted to export symbols for USER32.dll -
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
0016fa58 7d957625 USER32!WaitMessage+0x15
0016fa84 7d957b9b USER32!DrawStateW+0x92d
*** ERROR: Symbol file could not be found. Defaulted to export symbols for comdlg32.dll -
0016faa4 762be42a USER32!DialogBoxIndirectParamAorW+0x36
0016fadc 762be305 comdlg32!ChooseFontW+0x14e
0016fb1c 01003100 comdlg32!ChooseFontW+0x29
0016fd94 01003947 notepad+0x3100
0016fdb8 7d9472f8 notepad+0x3947
0016fde4 7d9475e3 USER32!WindowFromDC+0x3b
0016fe5c 7d947816 USER32!WindowFromDC+0x326
0016fed4 7d947858 USER32!WindowFromDC+0x559
0016fee4 01002a32 USER32!DispatchMessageW+0xf
0016ff1c 0100752b notepad+0x2a32
*** ERROR: Symbol file could not be found. Defaulted to export symbols for KERNEL32.dll -
0016ffc0 7d4e6e1a notepad+0x752b
0016fff0 00000000 KERNEL32!BaseProcessInitPostImport+0x8d
Note that in the ChildEBP and RetAddr columns, the addresses are now 32-bit, like "0016fa58".
Note also that the call stack looks very different. In particular the wow64 and wow64cpu module are no longer present.
This demonstrates how, from the point of view of the 32-bit code, the WoW64 layer is transparent, as if it did not exist.
If you attached a 32-bit debugger to this an instance of the 32-bit version of notepad you would see the same call stack.
The wow64exts debugger extension allows you to switch between x86 and x64 (or IA64) modes and debug the 32-bit process with a 64-bit debugger. This could be useful if you are trying to troubleshoot problem with a particular 32-bit process and you suspect the problem are to do with the behaviour of the WoW64 layer within that process.
The debugger extension also has special commands to allow you to do things such as set breakpoints in the 32-bit code.
Reference
WinHEC Presentation about the Wow64 architecture:
download.microsoft.com/download/1/8/f/18f8cee2-0b64-41f2-893d-a6f2295b40c8/DW04040_WINHEC2004.ppt
(Look at slides 5 and 6 in particular)
Cheers
Doug
Answer to the exercise (how can you tell which NTDLL is which):
If you calculate the size you can tell.
For the one listed as ntdll, 78ff9000 is the upper address of the mapped region of the module. 78ec0000 is the lower end. Use calc.exe to work out the difference. Check this against the sizes of the two versions of ntdll.dll on your system. Now check the size of the other one just to confirm.
The one shown as ntdll is the 64-bit one from windowssystem32.
The one shown as ntdll_7d600000 is the 32-bit one from windowssyswow64.
(the fact the debugger appends the _7d600000 to the one from syswow64 indicates it loaded second after the 64-bit one from system32)
Comments
Anonymous
April 09, 2007
Hi Doug, Your post is very informative! I've found it while searching for a way to solve my problem: I can't get the kernel32.dll symbols (debugging a 32-bit application) I have _NT_SYMBOL_PATH properly configured, and I'm getting some symbols (user32.dll, gdi32.dll, msvcrt.dll, etc.), but not for kernel32.dll, ntdll.dll, comctl32.dll and some others... I've tried to use symchk.exe on kernel32.dll and it sayd the .pdb is downloaded (c:WindowsSymbolskernel32.pdb8600FA0E2CE1463C8CDD0CBC26A8D84C2kernel32.pdb), but when I try to debug a 32-bit application, it will not load. I appreciate any information on this. ThanksAnonymous
April 09, 2007
Hi "e", Try setting your symbol path explicitly in the debugger like this: .symfix c:windowssymbols This should tell the debugger to use the public symbols server with c:windowssymbols as your downstream cache. You could then turn on verbose symbol loading: !sym noisy and trigger a symbol reload: .reload HTH DougAnonymous
September 04, 2007
Hi Doug, It seems that I cannot get the proper syswo64 symbols for ntdll (wntdll.pdb), the pdb I have been able from microsoft symbol server into my cache do not load into windbg. I have success with wkernel32.pdb though. Could you check if the correct wntdll.pdb is posted on the microsoft symbol server. I am using server 2003/64 bits, SP1. Thanks JoseAnonymous
September 04, 2007
Jose In the debugger, can you do lmvmwntdll !sym noisy .reload /f wntdll.dll and post the output that you get? DougAnonymous
September 19, 2007
OK, got it, thanks to the !sym noisy The symbol store I had setup here had white spaces in the directory name, windbg is able to read the pdb files, but is not able to write them in case they have to be downloaded from the microsoft symbol server.Anonymous
July 21, 2008
Hi all, Imagine we are running a 32bit .NET app in a x64 machine . This app is failing so we have takenAnonymous
September 03, 2008
Imagine we are running a 32bit .NET app in a x64 machine . This app is failing so we have taken a memoryAnonymous
September 29, 2011
Great article. I followed the steps and now can see tread callstacks in my 32-bit app.