debugging memory problems – a beginner’s view

Unlike native debugging, you don’t need symbols for debugging managed code. SOS.dll can also be used with WinDBG and Visual Studio debugger. Simply use the intermediate window in Visual Studio to load SOS.dll and then you can use the commands provided by the extension.

My favorite blog for managed debugging (and almost everyone else’s) is by Tess Ferrandez-Norlander. Tess is an Escalation Engineer with Microsoft. I don’t think anyone else explained managed debugging better than Tess. It is a good place to start with the basics & I highly recommend reading her blog. Her blog not only teaches the techniques of debugging, but also contains code examples that illustrate various problems. The comments section of each post also has very valuable information.

So here, I explain how I go about looking at high memory dumps:

Big Question: I always have a question in my mind: Where is most of the memory? Then start looking for it. Once I identify what all components are majorly contributing to memory usage, I check to see if the problem can be resolved with a hotfix. While this usually requires identifying patterns and some experience knowledge, for beginners, it is still worth checking the Microsoft support site to see if the problem you are running into is described in a KB article and if a hotfix is available. The easiest way is to stay updated because Microsoft continuously investigates such issues and provides hotfixes. If the problem is traced down to application code, the developers should work to reduce memory usage.

First step - Open the dump in WinDBG. For instructions on setting up WinDBG, please see this post.

First, we need to find out where “most” of the memory is. A debugger command is available that will help us here – !address –summary. To effectively understand and interpret what the output means we need to first understand what the output values represent. Take a look at this MSDN article. In the output of !address –summary command, the column named “KB” indicates in KiloBytes, the amount of memory in each of these “areas” and the Pct(Tots) column indicates a percentage of the value in KB column against Total address space (4 GB on 32 bit).

Thus, if RegionUsageIsVAD is high, it indicates that most memory is in virtual allocations. .NET heaps are made with calls to VirtualAlloc. The GC in .NET uses the Microsoft Win32® VirtualAlloc() application programming interface (API) to reserve a block of memory for its heap. Depending on the flavor of GC in use, it may be 32 MB (Workstation GC) allocations or 64 MB (Server GC). For ASP.net applications running in IIS, by default, the Server GC is automatically selected, if the computer has 2 or more processors.

Similarly, RegionUsageFree indicates most of memory is “Free”. If applications start throwing out of memory exceptions and a captured dump shows most of memory is Free, then it indicates fragmentation within the process address space. Typically if 30% or more of memory is Free, then I’d suggest investigating why memory is fragmented.

If RegionUsageImage is high, then in means you have lots of DLLs being loaded in the process. Ideally 100 MB or less is good. ASP.NET applications should not have debug attribute set to TRUE in web.config. Enabling debugging emits debug information which can be an overhead and cause memory problems. Similarly, there should not be any modules that are built in debug mode.

If RegionUsageHeap is high, it means most of the memory is on heaps. Eg the C Runtime heap, MDAC (Microsoft Data Access Components) heap, heaps used for compression etc. When you use the MSVCRT.dll library, it internally ends up allocating on the C runtime heap by calls to the CAlloc API.

Note that there are many tools out there (not just winDBG) that allows you to identify the allocation profiles of applications. Eg: For .NET applications you can use CLRProfiler (Microsoft) and Ant’s Profiler (RedGate). I do not know about profilers for native C, C++ applications. However Debug Diagnostics Tool v1.1 from Microsoft includes a capability for injecting a DLL – Leaktrack into the process that hooks into the calls like CAlloc and can keep track of who made the allocations. DebugDiag also includes a Memory pressure analysis script that can read a dump with leak track injected and give you a very nice report on where the allocations are – By size, by count and many other parameters. It’s 99% helpful in debugging web applications hosted on IIS. I will discuss this in another blog post.

WinDBG is usually used in post mortem debugging. Profilers should be used only in development and test environments, not production. They are very invasive and will reduce performance by about 10 times.

Continuing a bit more about RegionUsageIsVAD… If you determine that most memory is from Virtual Allocations AND you have ASP.net web applications running in IIS, then almost always it is likely that large amounts of memory is allocated in managed heaps. You will then need to investigate into .NET heaps to determine the top memory consumers. That’s where the !dumpheap –stat command comes into the picture.

More in my next post.