How to use Windbg to debug a dump of a 32bit .NET app running on a x64 machine
Imagine we are running a 32bit .NET app in a x64 machine. This app is failing so we have taken a memory dump of the app when a specific CLR exception gets raised.
We open the dump with our 64bit version of Windbg (Windbg x64), and we verify that we actualy got the dump when the exception happened:
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(27e8.2904): CLR exception - code e0434f4d (first/second chance not available)
wow64!Wow64NotifyDebugger+0x9:
00000000`78be6369 b001 mov al,1
We check the unmanaged call stack and it looks like this:
0:000> k
Child-SP RetAddr Call Site
00000000`0012e170 00000000`78be64f2 wow64!Wow64NotifyDebugger+0x9
00000000`0012e1a0 00000000`78be6866 wow64!Wow64KiRaiseException+0x172
00000000`0012e510 00000000`78b83c7d wow64!Wow64SystemServiceEx+0xd6
00000000`0012edd0 00000000`78be6a5a wow64cpu!ServiceNoTurbo+0x28
00000000`0012ee60 00000000`78be5e0d wow64!RunCpuSimulation+0xa
00000000`0012ee90 00000000`78ed8501 wow64!Wow64LdrpInitialize+0x2ed
00000000`0012f6c0 00000000`78ed6416 ntdll!LdrpInitializeProcess+0x17d9
00000000`0012f9d0 00000000`78ef3925 ntdll!_LdrpInitialize+0x18f
00000000`0012fab0 00000000`77d59640 ntdll!KiUserApcDispatch+0x15
00000000`0012ffa8 00000000`00000000 0x77d59640
00000000`0012ffb0 00000000`00000000 0x0
00000000`0012ffb8 00000000`00000000 0x0
00000000`0012ffc0 00000000`00000000 0x0
00000000`0012ffc8 00000000`00000000 0x0
00000000`0012ffd0 00000000`00000000 0x0
00000000`0012ffd8 00000000`00000000 0x0
00000000`0012ffe0 00000000`00000000 0x0
00000000`0012ffe8 00000000`00000000 0x0
00000000`0012fff0 00000000`00000000 0x0
00000000`0012fff8 00000000`00000000 0x0
Well, all this wow64 stuff doesn't tell us much... This is the 64bit look of our 32bit app. Let's try to take a look to the managed call stack. We need to load SOS extension for that:
0:000> .loadby sos mscorwks
The call to LoadLibrary(C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos) failed, Win32 error 0n193
"%1 is not a valid Win32 application."
Please check your debugger configuration and/or network access.
Well, this error makes sense. We are trying to load 32bit version of SOS into 64bit version of Windbg. 64bit version of SOS should be here: C:\Windows\Microsoft.NET\Framework64\v2.0.50727.
Let's try our 32bit version of Windbg (Windbg x86) then. We can also see the exception and the unmanaged call stack with all that wow64 stuff. Same as before. Let's try to load SOS then:
0:000> .loadby sos mscorwks
Ok, it seems to be loaded. Let's check it:
0:000> .chain
Extension DLL search Path:
...
Extension DLL chain:
C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos: image 2.0.50727.1434, API 1.0.0, built Thu Dec 06 05:42:38 2007
[path: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll]
...
Great. We are doing business. I don't want to see all that wow64 stuff, I want to see the managed (.NET) call stack with SOS:
0:000> !clrstack
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of mscorwks.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks___.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.
Weird. We got an unexpected error. Our debugger is the latest version, but it seems our version of mscorwks.dll is different than the one in the machine where we got the dump. Mscordacwks.dll is different too, of course. But Windbg should find mscordacwks in my symbol path... Let's verify that the path is correct:
0:000> .sympath
Symbol search path is: srv*c:\symbolspub*https://msdl.microsoft.com/downloads/symbols
The symbol path is correctly pointing to our Microsoft public symbol server, that should be enough... I'm also supposed to be debugging in the same architecture because I took the dump in AMD64 and my machine is AMD64, too. Let's try that command they mention in the error:
0:000> .cordll -ve -u -l
CLRDLL: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.1434 f:0
doesn't match desired version 2.0.50727.832 f:0
CLRDLL: ERROR: DLL c:\symbolspub\mscordacwks_x86_x86_2.0.50727.832.dll\461F2E2A566000\mscordacwks_x86_x86_2.0.50727.832.dll init failure, Win32 error 0n87
CLR DLL status: ERROR: DLL c:\symbolspub\mscordacwks_x86_x86_2.0.50727.832.dll\461F2E2A566000\mscordacwks_x86_x86_2.0.50727.832.dll init failure, Win32 error 0n87
Interesting. We just confirmed the version we need of mscordacwks is different than the one we have, it seems we are getting the right mscordacwks version from the symbol server, but we can't initialize the dll.
Wait a sec! All that wow64 stuff we saw in the call stack implies that we are in the 64bit part of our 32bit application. Let's try to see 32bit stuff only:
0:000> .load wow64exts
0:000> !sw
Switched to 32bit mode
Let's check the unmanaged call stack:
0:000:x86> kL
ChildEBP RetAddr
002df57c 79f55b05 kernel32!RaiseException+0x53
002df5dc 7a0904d5 mscorwks!RaiseTheExceptionInternalOnly+0x226
002df6a0 79646e0c mscorwks!JIT_Throw+0xfc
002df700 79465607 mscorlib_ni!System.IO.__Error.WinIOError(Int32, System.String)+0x1e17f8
002df71c 79649b75 mscorlib_ni!System.IO.__Error.WinIOError()+0x1f
002df71c 06e1c8f7 mscorlib_ni!System.IO.Path.GetTempFileName()+0x1d9d5d
...
No wow64 stuff in there. This is a typical 32bit call stack. We are good. Can we init mscordacwks.dll now?
0:000:x86> .cordll -ve -u -l
CLRDLL: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.1434 f:0
doesn't match desired version 2.0.50727.832 f:0
CLRDLL: Loaded DLL c:\symbolspub\mscordacwks_x86_x86_2.0.50727.832.dll\461F2E2A566000\mscordacwks_x86_x86_2.0.50727.832.dll
CLR DLL status: Loaded DLL c:\symbolspub\mscordacwks_x86_x86_2.0.50727.832.dll\461F2E2A566000\mscordacwks_x86_x86_2.0.50727.832.dll
It seems so! Let's check the managed call stack:
0:000:x86> !clrstack
OS Thread Id: 0x2904 (0)
ESP EIP
002df604 78be254a [HelperMethodFrame: 00000000002df604]
002df6a8 79646e0c System.IO.__Error.WinIOError(Int32, System.String)
002df708 79465607 System.IO.__Error.WinIOError()
002df710 79649b75 System.IO.Path.GetTempFileName()
...
002df8cc 7b074fef System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
002df8d0 7b082365 [InlinedCallFrame: 00000000002df8d0]
002df93c 7b147ff6 [InlinedCallFrame: 00000000002df93c]
Ok, now we are really doing business. Which exception did we get?
0:000:x86> !pe
Exception object: 000000000293a1cc
Exception type: System.IO.IOException
Message: The directory name is invalid.
InnerException:
StackTrace (generated):
StackTraceString:
HResult: 8007010b
We can now continue from here.
Do you want to know more on debugging wow64? Check these links:
A look at the WoW64 layer in the debugger
Comments
Anonymous
November 15, 2009
Excellent article!!Anonymous
June 11, 2012
this article would be better if it addressed other potential errors that could occur. For example: 0:000:x86> .cordll -ve -u -l CLR DLL status: No load attempts After following every other step in the article exactly.