Troubleshooting Memory Leaks With Just a Dump
Hello debuggers, the debug ninja is back again. Sometimes we have a scenario where a process is using a lot of memory, and the only data we are able to get at the moment is a user dump. Ordinarily data from tools such as umdh or xperf would be preferable because they provide memory usage data over a period of time and can include call stack information. However, umdh requires restarting the process (which loses the state of high memory usage), and xperf requires the installation of the Windows Performance Toolkit which may not always be an immediate option.
When we have such a dump we may not be able to specifically identify what piece of code is generating the high memory usage, but we may be able to narrow the scope of troubleshooting to a specific dll.
The first thing we need to do is identify what type of memory is using most of the address space. The debugger command !address –summary allows us to do this:
0:000> !address -summary
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 489 7fe`6ff5a000 ( 7.994 Tb) 99.92%
Heap 9094 1`75ed1000 ( 5.843 Gb) 93.47% 0.07%
<unknown> 275 0`12e41000 ( 302.254 Mb) 4.72% 0.00%
Image 937 0`05a6a000 ( 90.414 Mb) 1.41% 0.00%
Stack 138 0`01700000 ( 23.000 Mb) 0.36% 0.00%
Other 14 0`001bd000 ( 1.738 Mb) 0.03% 0.00%
TEB 46 0`0005c000 ( 368.000 kb) 0.01% 0.00%
PEB 1 0`00001000 ( 4.000 kb) 0.00% 0.00%
From this example we can see that most of the memory is used by heap. A process will usually have multiple heaps, each created by a call to HeapCreate. We can examine the size of each of these heaps with !heap –s:
0:000> !heap -s
LFH Key : 0x0000006c1104d280
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
0000000000100000 00000002 16384 12824 16384 1180 254 5 0 3 LFH
0000000000010000 00008000 64 4 64 1 1 1 0 0
00000000003d0000 00001002 1088 708 1088 121 20 2 0 0 LFH
0000000003080000 00001002 1536 700 1536 4 4 2 0 0 LFH
00000000033a0000 00001002 5229696 1377584 5229696 414244 4039 3059 0 2c LFH
External fragmentation 30 % (4039 free blocks)
Virtual address fragmentation 73 % (3059 uncommited ranges)
0000000003380000 00001002 64 8 64 3 1 1 0 0
0000000003600000 00001002 512 56 512 3 1 1 0 0
0000000003c20000 00001002 512 8 512 3 1 1 0 0
0000000003220000 00001002 512 8 512 3 1 1 0 0
0000000003e50000 00001002 512 8 512 3 1 1 0 0
0000000003d00000 00001002 512 148 512 5 3 1 0 0 LFH
From the above output we can see that most of the memory is being used by heap 00000000033a0000.
At this point we need to try to identify what this heap is used for. A brute force method to do this is to search memory with the ‘s’ command.
0:000> s -q 0 l?7fffffffffffffff 00000000033a0000
<snip>
000007fe`f21810a0 00000000`033a0000 00000000`00000001
The output of the ‘s’ command may be verbose. You will need to manually examine the addresses where ‘s’ finds hits. Most of these addresses will probably be in heap memory, we are looking for an address that matches a module. I snipped the above output to just show the relevant hit, an address inside of a loaded module.
The search of memory revealed that the heap 00000000033a0000 is used by the module useheap.dll, specifically it is part of the global ‘Blob’ class.
0:000> ln 000007fe`f21810a0
(000007fe`f21810a0) useheap!Blob::m_Heap
At this point we don’t know specifically what code in useheap.dll has allocated a lot of heap, however we have significantly narrowed the scope of the problem. We can now determine if there is a known issue with heap usage in useheap.dll that is addressed in a later version. We may also know from experience that this module uses a lot of memory under specific circumstances, such as a high volume of work sent to this service.
I hope this example helps the next time you have high memory usage and only have a user dump to troubleshoot with. Good luck!
Comments
Anonymous
April 26, 2012
A more efficient way, if you have private symbols, is x ! And search that output for 33a0000 Which would give you Ushheap!blob::m_heap = 0x33a0000Anonymous
April 26, 2012
Very often heap handles are stored in a global variables and got "heap" at the end of the variable name (like MS CRT's "_crtheap"). I usually try to locate such global variables by running "x *!_crtheap" or "x !_heap" or even "x *!*heap" or "x *!heap". There can be a lot of matches including function names but gloabl heap variables are usually visible. Of course this will not work if there are no symbols available.Anonymous
May 23, 2012
kd> !address summary *** ERROR: Module load completed but symbols could not be loaded for vmci.sys *** ERROR: Module load completed but symbols could not be loaded for pvscsi.sys *** ERROR: Module load completed but symbols could not be loaded for intelide.sys *** ERROR: Module load completed but symbols could not be loaded for spldr.sys *** ERROR: Module load completed but symbols could not be loaded for vmxnet3n61x64.sys *** ERROR: Module load completed but symbols could not be loaded for intelppm.sys *** ERROR: Module load completed but symbols could not be loaded for vm3dmp.sys *** ERROR: Module load completed but symbols could not be loaded for tmtdi.sys *** ERROR: Module load completed but symbols could not be loaded for vmmouse.sys *** ERROR: Module load completed but symbols could not be loaded for dump_pvscsi.sys *** ERROR: Symbol file could not be found. Defaulted to export symbols for VSApiNt.sys - Let me know why i am getting above errors. [Hi Manoj. The command to use is !address -summary . If you leave off the '-' the debugger will attempt to resolve the word "summary" as a symbol, leading to the symbol load failures you are seeing.]