Setting up managed code debugging (with SOS and SOSEX)
In this post I want to share some basics on how to set-up managed code debugging in windows debugger. I’ll consider that reader already know basics about how to setup windows debugging. I’ll mainly discuss details about what and how need to be loaded in order to be able to use specialized extensions for managed code debugging (mainly sos and sosex). I’ll not be discussing the debugging itself (e.g. commands from sos and sosex extensions).
Local live debugging
In the case of local live debugging we don’t have to much obstacles in our way (since we have access to the used .net framework binaries and data access layers – those I’ll discuss later). You should make sure that you are using same bitness of debugger and extension as is the bitness of the target application (.NET application are usually built architecture agnostic and then run as 32 bit processes on 32 bit OS and 64 bit processes on 64 bit OS).
If you would be about to use mismatching bitness of debugger to attach/start the managed process you might see the following error:
---------------------------
WinDbg:6.13.0014.1618 X86
---------------------------
Could not create process '\\Foo\Bar\Baz.exe', Win32 error 0n50
The request is not supported.
---------------------------
OK
---------------------------
SOS extension
Once you are in the debugger, broken in into the target process, you will want to load sos extension (name is a acronym for ‘Son Of Strike’ as the extension was developed as the next version of the extension named ‘Strike’). This extension is shipped with .NET framework (and also indexed on Microsoft symbols server which will come in handy for other scenarios). An old version (1.1) of this extension also ships with the debugger package – and this can actually get into your way.
For .NET framework 1.1 the SOS extension has all the information about .NET internals ‘baked-in’ itself (some of those information you would normally expect to be in the symbol files) – so all you needed was loading the SOS. From the .NET version 2.0 the definition of data structures was separated into separate binary – mscordacwks (sometimes called data access layer). When loading the SOS extension it will need to also load correct data access layer (for correct architecture and version of .NET framework targeted by the debugged application/dump). Another important change for our purposes was introduced in .NET framework 4.0 – the mscorwks.dll module was replaced with module clr.dll. I’ll show how all those facts affects the managed code debugging.
Loading SOS extension during live local debugging
During the live local debugging you have the luxury of knowing that you computer has the same version and architecture of .NET framework as what is the debugged target using (because they are on the same machine :-)). Same approach also applies when you are debugging remotely or postmortem but know for sure that you have same architecture and version on your computer as what was on the target computer.
In this case you know that your installation of .NET framework will also have appropriate version SOS extension and mscordacwks.dll binary. They will actually in the same folder as where is the core dll for .NET – mscorwks.dll (for .NET up to 4.0) or clr.dll (for .NET 4.0 and higher) in that case you can simply use loadby command that will load an extension from a location of another binary that is already loaded in the debugger (so you need to make sure those modules are already loaded, or force load the with ‘.load /f <module>’):
.loadby sos.dll mscorwks
.loadby sos.dll clr
Loading SOS extension during postmortem or remote debugging
If you would try previous commands on remote or postmortem debugging, you might happen to see the following error message:
The version of SOS does not match the version of CLR you are debugging. Please
load the matching version of SOS for the version of CLR you are debugging.
CLR Version: 4.0.30319.19010
SOS Version: 4.0.30319.269
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of clr.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks___.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.
The message itself already contains few helpful tips. However if you meet this situation you might be able to easily resolve majority of those cases by leveraging the fact that .NET Framework debugging related binaries are being indexed on Microsoft symbol server – internal and also external (for more info or symbol server, setting symbols path and execution image path you can refer to the setup windows debugging blog post). Basically for remote or postmortem debugging of managed application/dump in windows debugger you should follow those simple steps:
1) Set your symbol path to Microsoft symbol server (internal or external)
2) Set your execution image path to Microsoft symbol server
3) Make sure that the core CLR binary (mscorwks.dll or clr.dll) is loaded, reload it (with .reload /f) if it’s not
4) Reload CLR debugging related binaries by using ‘.cordll –ve –u –l’ command (-ve for verbose, –u for unload, –l for load)
5) Load the SOS module from the core CLR binary location (as you’d do that for live debugging)
Here is the example debugging session demonstrating those steps:
Microsoft (R) Windows Debugger Version 6.13.0014.1618 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [\\XXXXXX\FooBar_121030_075015.dmp]
User Mini Dump File with Full Memory: Only application data is available
Comment: '
*** Procdump -ma FooBar.exe
*** Manual dump'
Symbol search path is: YYYYYYYYY
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (32 procs) Free x86 compatible
Product: Server, suite: TerminalServer DataCenter SingleUserTS
Machine Name:
Debug session time: Tue Oct 30 06:50:17.000 2012 (UTC - 8:00)
System Uptime: 16 days 19:11:53.254
Process Uptime: 0 days 23:42:59.000
................................................................
..................................................
Loading unloaded module list
.
Cannot Automatically load SOS
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=000001ec edi=00000000
eip=7784f8b1 esp=0044f164 ebp=0044f1d0 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!ZwWaitForSingleObject+0x15:
7784f8b1 83c404 add esp,4
0:000> !eeversion
No export eeversion found
0:000> .symfix+
0:000> .sympath
Symbol search path is: SRV*https://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*https://msdl.microsoft.com/download/symbols
0:000> .exepath SRV*https://msdl.microsoft.com/download/symbols
Executable image search path is: SRV*https://msdl.microsoft.com/download/symbols
Expanded Executable image search path is: srv*https://msdl.microsoft.com/download/symbols
0:000> lmvm mscorwks
start end module name
709f0000 70f9b000 mscorwks (deferred)
Image path: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
Image name: mscorwks.dll
Timestamp: Sat Dec 24 22:48:13 2011 (4EF6C72D)
CheckSum: 005B474B
ImageSize: 005AB000
File version: 2.0.50727.5456
Product version: 2.0.50727.5456
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® .NET Framework
InternalName: mscorwks.dll
OriginalFilename: mscorwks.dll
ProductVersion: 2.0.50727.5456
FileVersion: 2.0.50727.5456 (Win7SP1GDR.050727-5400)
FileDescription: Microsoft .NET Runtime Common Language Runtime - WorkStation
LegalCopyright: © Microsoft Corporation. All rights reserved.
Comments: Flavor=Retail
0:000> .reload /f mscorwks.dll
0:000> lmvm mscorwks
start end module name
709f0000 70f9b000 mscorwks (private pdb symbols) D:\Debuggers_32bit\sym\mscorwks.pdb\7139B75336C24F7CAA1DC4060608770D2\mscorwks.pdb
Loaded symbol image file: mscorwks.dll
Image path: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
Image name: mscorwks.dll
Timestamp: Sat Dec 24 22:48:13 2011 (4EF6C72D)
CheckSum: 005B474B
ImageSize: 005AB000
File version: 2.0.50727.5456
Product version: 2.0.50727.5456
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® .NET Framework
InternalName: mscorwks.dll
OriginalFilename: mscorwks.dll
ProductVersion: 2.0.50727.5456
FileVersion: 2.0.50727.5456 (Win7SP1GDR.050727-5400)
FileDescription: Microsoft .NET Runtime Common Language Runtime - WorkStation
LegalCopyright: © Microsoft Corporation. All rights reserved.
Comments: Flavor=Retail
0:000> .cordll -ve -u -l
CLRDLL: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.6387 f:0
doesn't match desired version 2.0.50727.5456 f:0
CLRDLL: Unable to find '' on the path
Cannot Automatically load SOS
CLRDLL: Loaded DLL D:\Debuggers_32bit\sym\mscordacwks_x86_x86_2.0.50727.5456.dll\4EF6C72D5ab000\mscordacwks_x86_x86_2.0.50727.5456.dll
CLR DLL status: Loaded DLL D:\Debuggers_32bit\sym\mscordacwks_x86_x86_2.0.50727.5456.dll\4EF6C72D5ab000\mscordacwks_x86_x86_2.0.50727.5456.dll
0:000> .loadby sos mscorwks
0:000> !eeversion
2.0.50727.5456 retail
Workstation mode
SOS Version: 2.0.50727.6387 retail build
This should help you resolve vast majority of issues with loading modules for managed debugging, there still might be some special cases where you can see issues.
Issues with loading SOS extension and related binaries
One of those can happen when the mscordacwks is not properly indexed on the symols server. Then ouw would see similar output after running .cordll:
3: kd> .cordll -ve -u -l
CLRDLL: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscordacwks.dll:4.0.30319.269 f:8
doesn't match desired version 4.0.30319.19010 f:8
CLRDLL: Unable to find 'R?' on the path
CLRDLL: Unable to find 'mscordacwks_AMD64_AMD64_4.0.30319.19010.dll' on the path
CLRDLL: Unable to get version info for '\\ntdev.corp.microsoft.com\wincorerelease13\fbl_core1_kernel\9257.0.121021-1715\amd64fre\bin\indexes\..\netfx\core\win\microsoft.net\framework64\v4\mscordacwks_AMD64_AMD64_4.0.30319.19010.dll', Win32 error 0n87
CLRDLL: ERROR: Unable to load DLL mscordacwks_AMD64_AMD64_4.0.30319.19010.dll, Win32 error 0n87
CLR DLL status: ERROR: Unable to load DLL mscordacwks_AMD64_AMD64_4.0.30319.19010.dll, Win32 error 0n87
This can potentially happen to you inside Microsoft when working with specific build of .NET Framework that wasn’t properly indexed (or the index was already purged) – it should never happen outside of Microsoft. If this happens, contact CLR team, as they have their own share for the build binaries that might have longer data retention.
Other – most common – issue can happen when you accidentally load the improper version of sos (e.g. from the debugger package, by simply running ‘.load sos’ and picking some wrong version that is in the path). .chain command can help reveal those issues:
0:000> !eeversion
Doesn't work with 2.x
0:000> .chain
Extension DLL search Path:
D:\Debuggers_32bit\WINXP;D:\Debuggers_32bit\winext;D:\Debuggers_32bit\winext\arcade;D:\Debuggers_32bit\pri;D:\Debuggers_32bit;D:\Debuggers_32bit\winext\arcade;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Windows Kits\8.0\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\Odd\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\ManagementStudio\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\;C:\Program Files\Microsoft SQL Server\110\DTS\Binn\;D:\Debuggers_64bit
Extension DLL chain:
D:\Debuggers_32bit\clr10\sos: image 6.13.0014.1618, API 1.0.0, built Wed Oct 17 10:41:23 2012
[path: D:\Debuggers_32bit\clr10\sos.dll]
dbghelp: image 6.13.0014.1618, API 6.2.6, built Wed Oct 17 10:42:02 2012
[path: D:\Debuggers_32bit\dbghelp.dll]
ext: image 6.13.0014.1618, API 1.0.0, built Wed Oct 17 10:41:49 2012
[path: D:\Debuggers_32bit\winext\ext.dll]
exts: image 6.13.0014.1618, API 1.0.0, built Wed Oct 17 10:42:19 2012
[path: D:\Debuggers_32bit\WINXP\exts.dll]
uext: image 6.13.0014.1618, API 1.0.0, built Wed Oct 17 10:42:18 2012
[path: D:\Debuggers_32bit\winext\uext.dll]
ntsdexts: image 6.2.9255.0, API 1.0.0, built Wed Oct 17 10:42:28 2012
[path: D:\Debuggers_32bit\WINXP\ntsdexts.dll]
Loading SOSEX extension
SOSEX is helpful extension that extends possibilities of SOS. You first need to have SOS loaded – as described above. Then you can simply download latest version of SOSEX from https://www.stevestechspot.com/ and load it into the debugger and use it:
0:000> .load D:\Tools\sosex\32bit\sosex
This dump has no SOSEX heap index.
The heap index makes searching for references and roots much faster.
To create a heap index, run !bhi
0:000> !sosex.help
SOSEX - Copyright 2007-2012 by Steve Johnson - https://www.stevestechspot.com/
To report bugs or offer feedback about SOSEX, please email sjjohnson@pobox.com
Quick Ref:
--------------------------------------------------
bhi [filename] BuildHeapIndex - Builds an index file for heap objects.
bpsc (Deprecated. Use !mbp instead)
chi ClearHeapIndex - Frees all resources used by the heap index and removes it from memory.
dlk [-d] Displays deadlocks between SyncBlocks and/or ReaderWriterLocks
dumpgen <GenNum> [-free] [-stat] [-type <TYPE_NAME>] Dumps the contents of the specified generation
[-nostrings]
finq [GenNum] [-stat] Displays objects in the finalization queue
frq [-stat] Displays objects in the Freachable queue
gcgen <ObjectAddr> Displays the GC generation of the specified object
gch [HandleType]... Lists all GCHandles, optionally filtered by specified handle types
help [CommandName] Display this screen or details about the specified command
lhi [filename] LoadHeapIndex - load the heap index into memory.
mbc <SOSEX breakpoint ID | *> Clears the specified or all managed breakpoints
mbd <SOSEX breakpoint ID | *> Disables the specified or all managed breakpoints
mbe <SOSEX breakpoint ID | *> Enables the specified or all managed breakpoints
mbl [SOSEX breakpoint ID] Prints the specified or all managed breakpoints
mbm <Type/MethodFilter> [ILOffset] [Options] Sets a managed breakpoint on methods matching the specified filter
mbp <SourceFile> <nLineNum> [ColNum] [Options] Sets a managed breakpoint at the specified source code location
mdso [Options] Dumps object references on the stack and in CPU registers in the current context
mdt [TypeName | VarName | MT] [ADDR] [Options] Displays the fields of an object or type, optionally recursively
mdv [nFrameNum] Displays arguments and locals for a managed frame
mfrag [-stat] [-mt:<MT>] Reports free blocks, the type of object following the free block, and fragmentation statistics
mframe [nFrameNum] Displays or sets the current managed frame for the !mdt and !mdv commands
mgu // TODO: Document
mk [FrameCount] [-l] [-p] [-a] Prints a stack trace of managed and unmanaged frames
mln [expression] Displays the type of managed data located at the specified address or the current instruction pointer
mlocks [-d] Lists all managed lock objects and CriticalSections and their owning threads
mroot <ObjectAddr> [-all] Displays GC roots for the specified object
mt (no parameters) Steps into the managed method at the current position
mu [address] [-s] [-il] [-n] Displays a disassembly around the current instruction with interleaved source, IL and asm code
muf [MD Address | Code Address] [-s] [-il] [-n] Displays a disassembly with interleaved source, IL and asm code
mwaits [-d] Lists all waiting threads and, if known, the locks they are waiting on
mx <Filter String> Displays managed type/field/method names matching the specified filter string
refs <ObjectAddr> [-target|-source] Displays all references from and to the specified object
rwlock [ObjectAddr | -d] Displays all RWLocks or, if provided a RWLock address, details of the specified lock
sosexhelp [CommandName] Display this screen or details about the specified command
strings [ModuleAddress] [Options] Search the managed heap or a module for strings matching the specified criteria
ListGcHandles - See gch
Use !help <command> or !sosexhelp <command> for more details about each command.
That’s it today. Now you are introduced to setting up managed code debugging and next time I’ll try to post some real live interesting case scenario.