Why do I get weird function names on my stack? (a discussion on symbols)
Symbols can contain information about global variables, local variables, function names, parameters, structures and source line numbers.
There are 3 types of symbols, export symbols, pdb symbols (public symbols) and private pdb symbols (private symbols).
The export symbols are part of the dll itself. For example ntdll.dll and kernel32.dll expose a big part of their functions as export symbols so that they can be called as API’s, but most dll’s that you find in a process have a very small set of exported symbols. Generally export symbols don’t contain parameter info for the functions and since very few functions are exposed this way you can’t really relay on the validity of the stacks when you have only export symbols.
Public symbols contain some basic symbols such as function names and global variables, but again, not all function names are exposed in public symbols. The developer of the dll chooses what to expose as public symbols and thus he/she can hide anything that they feel would give away too much information about the implementation.
Private symbols contain pretty much everything listed in the first paragraph
When debugging, symbols are matched up with the respective dlls or exe’s by a GUID linking the dll/exe to a symbol file (thanks Pavan for the correction) . This means that if you have multiple ntdll.pdb’s in your symbol search path, the debugger will know which one corresponds to your particular version of ntdll.dll. The search path is given by .sympath, and apart from what is listed in your sympath the debugger will also look in the directory where the dll is loaded from as well as anything in the paths given in the environment variable _NT_SYMBOL_PATH.
So… what happens if you have the wrong symbols?
Let’s look at this stack with public symbols for mscorsvr.dll: (sorry about the formatting)
54 Id: 62c.1590 Suspend: 1 Teb: 7ffa2000 Unfrozen
ChildEBP RetAddr Args to Child
1212ef44 7c59a030 00000090 00000000 1212ef64 ntdll!NtWaitForSingleObject+0xb [i386\usrstubs.asm @ 2004]
1212ef6c 7c57b3db 00000090 00009c40 00000000 kernel32!WaitForSingleObjectEx+0x71 [D:\nt\private\windows\base\client\synch.c @ 1309]
1212ef7c 791b578b 00000090 00009c40 00000000 kernel32!WaitForSingleObject+0xf [D:\nt\private\windows\base\client\synch.c @ 1217]
1212efa0 791dbe6e 00000000 00000000 00000000 mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x3a
1212ffb4 7c57b388 0d406838 00000002 00000000 mscorsvr!ThreadpoolMgr::intermediateThreadProc+0x44
1212ffec 00000000 791dbe2d 0d406838 00000000 kernel32!BaseThreadStart+0x52 [D:\nt\private\windows\base\client\support.c @ 460]
We are creating a worker thread and then sit and wait for work
Let’s look at the same stack with export symbols for mscorsvr:
0:054> kb
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
1212ef6c 7c57b3db 00000090 00009c40 00000000 ntdll!NtWaitForSingleObject+0xb
*** ERROR: Symbol file could not be found. Defaulted to export symbols for mscorsvr.dll -
1212efa0 791dbe6e 00000000 00000000 00000000 kernel32!WaitForSingleObject+0xf
1212ffb4 7c57b388 0d406838 00000002 00000000 mscorsvr!GetCompileInfo+0x8e99
1212ffec 00000000 791dbe2d 0d406838 00000000 kernel32!lstrcmpiW+0xb7
The debugger is nice and tells us that we couldn’t find the symbol file for mscorsvr, but yet it gives us a function name (from the export symbols) so it appears that we’re calling some function called GetCompileInfo, hmm… very strange, why did it pick that name?
If we were to list the symbols (export symbols for mscorsvr) we would get a list that looks like this
0:054> x mscorsvr!*
791b0000 mscorsvr!Ordinal73 =
791b0000 mscorsvr!Ordinal76 =
791b0000 mscorsvr!Ordinal77 =
791b0000 mscorsvr!Ordinal75 =
791b0000 mscorsvr!Ordinal78 =
791b0000 mscorsvr!Ordinal71 =
791b0000 mscorsvr!Ordinal79 =
791b0000 mscorsvr!Ordinal74 =
791b0000 mscorsvr!Ordinal72 =
791d2fd5 mscorsvr!GetCompileInfo =
791e0920 mscorsvr!GetAssemblyMDImport ()
…
So we are executing something at address 791dbe6e (791d2fd5+0x8e99), if we take a peak at the symbols with public symbols loaded, we would see that that address landed inside the intermediateThreadProc function (+0x44 to be exact)
791bbb78 mscorsvr!SimpleRWLock::LeaveRead =
793eaf50 mscorsvr!ExecutionManager::m_dwReaderCount =
791dbe2d mscorsvr!ThreadpoolMgr::intermediateThreadProc =
793035c5 mscorsvr!VtBoolMarshaler::UnmarshalComToNativeIn =
791f8e0d mscorsvr!LayoutClassPtrMarshaler::ClearNativeTemp =
So in essence, the reason it picked the name GetCompileInfo was that that was the last symbol it could find, preceding the address that we were executing at.
You can see how this can easily get confusing. Oh and btw, not only did it give us a completely wrong function name, but if you take a closer look at the two stacks, we lost a whole function call on the stack with the export symbols (The call to WorkerThreadStart). The reason for this is that the function we were supposed to look at was at address 791b578b and this address is located before our first exported function so it can’t even resolve it to something fake.
How do you know if your symbols are good?
For starters, just by looking at the stacks we can see that the function names seem pretty weird. It doesn’t make sense that a thread would start with lstrcmpiW, they usually start with BaseThreadStart or something to that effect. The second thing that gives it away just by looking at the stack is that we were supposed to be 0x8e99 instructions in to the function GetCompileInfo, that is 36505 instructions, wow, that would be a mighty long function.
The more definite way of knowing would be to run lmv on the executable. (notice the extra m before mscorsvr for match pattern)
0:054> lmv mmscorsvr
start end module name
791b0000 79418000 mscorsvr (export symbols) mscorsvr.dll
Loaded symbol image file: mscorsvr.dll
Image path: C:\WINNT\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll
Image name: mscorsvr.dll
Timestamp: Thu Jul 15 09:26:32 2004 (40F631A8)
CheckSum: 002664D4
ImageSize: 00268000
File version: 1.1.4322.2032
Product version: 1.1.4322.2032
File flags: 20 (Mask 3F) Special
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04e4
CompanyName: Microsoft Corporation
ProductName: Microsoft .NET Framework
InternalName: MSCORSVR.DLL
OriginalFilename: mscorsvr.dll
ProductVersion: 1.1.4322.2032
FileVersion: 1.1.4322.2032
FileDescription: Microsoft .NET Runtime Common Language Runtime - Server
LegalCopyright: Copyright © Microsoft Corporation 1998-2002. All rights reserved.
LegalTrademarks: Microsoft® is a registered trademark of Microsoft Corporation. Windows(TM) is a trademark of Microsoft Corporation
Comments: Microsoft .NET Runtime Common Language Runtime - Server
Lmv will give you a lot of good information about the dll, but in this particular case what we are interested is what is printed after the module name, i.e export symbols in this case. (You can also get this for all modules by running lm)
One last comment on this: As you may have noticed, going to the public Microsoft symbols server you sometimes get public symbols and sometimes end up with export symbols for mscorsvr.dll for example. This is because some cases like certain hotfixes and private builds have not been uploaded to the public symbol server.
Lazy symbol loading and downloading symbols for offline debugging
If you run lm you will notice that a lot of the dll’s and exe’s will have their symbols listed as deferred. This is because the debugger doesn’t actually load all the symbols when the debugger starts (as this would take a looooooooong time especially if you have a remote symbol store). Symbols are loaded on an as needed bases, i.e. when you run kb (to list the stacks) or run x to examine the symbols for a particular executable, or any other command that requires symbol lookup.
Sometimes though, it is useful to download all the symbols for a process, a common example would be if your application is running on a machine with no internet access and you want to be able to live debug it.
In this case what you can do is this:
- Take a dump of the process when all modules are loaded and open it on a machine with internet access.
- Set the sympathy to srv*c:\myappsymbols*https://msdl.microsoft.com/download/symbols;
- run .reload /i /f to force load all the symbols for the dump
This will download all the symbols available on Microsofts public symbol server that match your dlls into the c:\myappsymbols and then you can simply copy this folder over to the machine without internet access and set the symbolpath there to srv*c:\myappsymbols*c:\myappsymbols;c:\folderwithanyadditionalsymbols.
But… some of the functions on my stack only show up as addresses, what is up with that?
This really doesn’t have anything to do with symbols but I thought it was worth mentioning anyways. In the callstack below we can see one function call only listed by it’s address… 0xd44334a, the reason for this is that it is a managed function which doesn’t have a native translation, and after all windbg is merely a native debugger. You can run !clrstack or !ip2md on the address to get the .net function name.
0:045> kb
ChildEBP RetAddr Args to Child
1182ecd0 7c59c5b7 7920d6a7 00000010 0f39c800 ntdll!NtYieldExecution+0xb [i386\usrstubs.asm @ 2108]
1182ecd4 7920d6a7 00000010 0f39c800 00000010 kernel32!SwitchToThread+0x6 [D:\nt\private\windows\base\client\thread.c @ 2611]
1182ed00 791c0283 0f39c800 00000010 00000000 mscorsvr!gc_heap::allocate_more_space+0xe3
1182ef28 791b6930 0f39c800 00000010 00000000 mscorsvr!GCHeap::Alloc+0x7b
1182ef3c 791c0359 00000010 00000000 00000000 mscorsvr!Alloc+0x3a
1182ef5c 791c03a7 0d421928 1182efec 062eff1c mscorsvr!FastAllocateObject+0x25
1182efc8 0d44334a 00000000 1182f01c 1182f040 mscorsvr!JIT_NewFast+0x2c
WARNING: Frame IP not in any known module. Following frames may be wrong.
1182effc 791b33ec 1182f114 791b3c96 1182f050 0xd44334a
1182f004 791b3c96 1182f050 00000000 1182f028 mscorsvr!CallDescrWorker+0x30
1182f114 791b5612 00bf1bdb 79b7a000 00000004 mscorsvr!MethodDesc::CallDescr+0x1b8
1182f1d0 791b56c6 79bf1bdb 79b7a000 79bd1d16 mscorsvr!MethodDesc::CallDescr+0x4f
1182f1f8 791b5c40 1182f210 0ba57968 0f39c7c8 mscorsvr!MethodDesc::Call+0x97
1182f2ac 791b5b97 0f3a6480 00000001 00000000 mscorsvr!AddTimerCallbackEx+0x165
1182f2c0 791b4de8 1182f374 791b3cf4 ffffffff mscorsvr!AddTimerCallback_Wrapper+0x13
1182f308 791b5ae5 0ba57968 791b5b84 1182f374 mscorsvr!Thread::DoADCallBack+0x5c
1182f3c4 791b59ce 0f3a6480 00000001 00000001 mscorsvr!AddTimerCallbackEx+0x165
1182f3d8 791b59a5 0f3a6480 00000001 791b5996 mscorsvr!AddTimerCallback+0x10
1182f3ec 791b5802 0f3a2230 0f3b6bf0 793ea988 mscorsvr!ThreadpoolMgr::AsyncTimerCallbackCompletion+0xf
1182f400 791b57c1 0f3b6bf0 00000000 791b5751 mscorsvr!ThreadpoolMgr::ExecuteWorkRequest+0x19
1182f420 791dbe6e 00000000 00000000 00000000 mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x129
0:045> !clrstack
Loaded Son of Strike data table version 5 from "C:\WINNT\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll"
Thread 45
ESP EIP
0x1182efa4 0x77f88fe3 [FRAME: HelperMethodFrame]
0x1182efd0 0x0d44334a [DEFAULT] [hasThis] Void System.Timers.Timer.MyTimerCallback(Object)
0x1182f2cc 0x791b33ec [FRAME: ContextTransitionFrame]
How about symbols for managed dlls? Do I need them?
Technically for debugging with windbg the answer is that you would need them very seldom or never. If you do have managed symbols and the dlls are compiled in debug mode you can get line numbers and source files. You can also get some parameter info when using !clrstack –p. But all in all, the extra “stuff” you get from managed symbols is very little when using a native debugger like windbg.
Over and out…
Comments
Anonymous
February 07, 2006
Wow thank you! This stuff is just mind blowing. How did you learn all that?Anonymous
April 07, 2006
this blog is one (if not) the best debugging blogs I'ever visited. -- John - TexasAnonymous
October 16, 2006
I have talked about a number of different hang/performance issues in my posts. This post is a genericAnonymous
November 14, 2006
Wow. I was trying to figure out how I was going to load symbols for Windows DLLs that don't seem to match the symbol packs while remote debugging without being able to access the internet to get to the symbol server (stupid VPN software redirects all traffic except DHCP and the VPN connection over the VPN, and the customer site does not allow internet access.) I'm debugging using Visual Studio, and was not aware that I could create a dump file using it. I'm currently waiting for the dump file to transfer across the VPN so I can load it while I have internet access. I don't see why this wouldn't work, though. Thanks! (found this blog entry via a google search, fwiw)Anonymous
November 14, 2006
Apparently, I can create the dump using VS, but in order to download the symbols, I have to load the dump into WinDbg and do the force reload command given above. Once I do that, everything works. Thanks again!Anonymous
July 13, 2007
Sorry pelase : Wrong cdteagory... wippbe carefulAnonymous
February 03, 2008
.NET Debugging Demos This is a series of debugging demos aimed to help you get some hands on experienceAnonymous
May 13, 2008
When you debug .net applications you will sometimes get error messages in windbg. Here are a few of theAnonymous
May 20, 2008
If you have read any of my posts you have probably noticed that I am very partial to windbg and the debuggingAnonymous
July 07, 2008
.NET调试实例 这是一个系列的调式实例，目的是为了帮助你在调式.NET应用程序中最常见的挂起(Hang)、性能(performance)、内存(memory)和系统崩溃(crash)方面获得一些...Anonymous
July 10, 2009
The debugger matches the symbols not by the timestamp and checksum but by a GUID that links the dll and pdb. This GUID is generated by the compiler and is inserted into the dll and pdb headers. Thanks.Anonymous
October 01, 2009
0:054> lmv mmscorsvr shoulde be 0:054> lmv m mscorsvrAnonymous
October 01, 2009
Yanglei, you can do it either wayAnonymous
December 12, 2009
The comment has been removedAnonymous
December 13, 2009
it looks like a problem with sos where you dont have the proper symbols for mscordacwks on the client. Do you have the same framework version on the client and server?Anonymous
March 03, 2010
Tess, I'm getting some problems with loading the symbols for the apps I need to debug. They are a mish-mash of "co-operative" ("promiscuous" is more accurate really) Smalltalk, Win32 C API and .Net C++/C# exes and dlls. I think part of the problem is because the .Net bits are using a point-release/hotfixed version of the .Net Frameworks (v1.0.3705) for which windbg can't find the appropriate symbols (or not at http://msdl.microsoft.com/download/symbols, anyway). First, does this sound like a plausible cause of 'problems' for windbg? If so how badly will it affect what I can reliably do in windbg with these kind of apps if I can't resolve it? And do you know if its possible to get symbols for this version of the .Net Frameworks somehow? thanks Tim PS this blog is really useful - tvm.Anonymous
March 04, 2010
The comment has been removedAnonymous
March 04, 2010
The comment has been removedAnonymous
March 07, 2010
Tim, If I would venture a guess, I think that the 1.0 symbols might have been removed from the public symbol server to save some space, as 1.0 hasn't been supported for quite some time now.Anonymous
June 30, 2011
Hi Tess, what is going wrong if I see the following entry: ... ? <Unloaded_tle.dll>+4dffc8 Best regards, Artjom