The Infamous Debug Attribute

Our most common issues deal with memory problems. Memory problems come in many flavors, but one of the more common ones is the OutOfMemory exception or OOM.

A lot of the OOM issues we see are due to high memory conditions, but some are not. When an OOM condition is not accompanied by high memory usage, it can be confusing. Hopefully I can shed some light on that by discussing one of the most common causes of OOM in a lower-than-expected memory consumption scenario.

An OOM occurs when a memory allocation fails for some reason. In an ASP.NET application, an OOM is often the result of the CLR not being able to allocate a new segment for the managed heap. These allocations are 32MB for lower generations and 64MB for the LOH, and the memory for the segment must be contiguous. Therefore, you may have 400MB of free memory for the process to use, but if we can't get a large enough contiguous block, you'll see an OOM because of memory fragmentation.

So what causes fragmentation? In our world, one of the most common causes is having a large number of dynamic assemblies loaded into the process, and one of the most common causes of that is the infamous debug attribute in the web.config file.

The debug attribute looks like this:

<compilation debug="true" />

In ASP.NET 1.x, the debug attribute is true by default. When the debug attribute is true, ASP.NET batching is disabled and each page gets dynamically compiled into its own assembly (DLL file) and loaded into the app domain. While each of these dynamic assemblies is small in size, we don't load them into any particular address space so they end up getting scattered thoughout memory. If there are enough of them, you can end up being so badly fragmented that OOM conditions occur. However, we've seen a lot of other problems caused by having debug set to true as well. Rule of thumb; in a production environment, the debug attribute should always be set to false.

So what is this batch compile thing?

When you browse an ASP.NET application, your application executes from a dynamic assembly that is copied into the Temporary ASP.NET Files directory on the Web server. That assembly gets loaded into your application domain as well. When batch compile is enabled (it is by default unless debug is true), ASP.NET will compile all pages in a particular directory into a single DLL. When batch compile is disabled (and it will be if debug is true), each page is compiled into a separate assembly. (In a future blog, I'll go into how you can tell if an assembly it batched.) That means that if you have 1,000 pages, you'll have 1,000 assemblies just for your pages. That doesn't take into account other dynamic assemblies that are created due to XML serialization and for other reasons.

What do I lose by setting debug to false?

I talk to a lot of developers who resist setting debug to false because they tell me that they want to get stack trace information in their exception logging. Good news! The debug attribute has nothing at all to do with getting stack trace information. The purpose of the debug attribute is to ensure that a particular page gets compiled into its own DLL so that you can debug it in Visual Studio, etc. Debug can be set to false and it will have no effect on your ability to get stack trace information and to do post-mortem debugging with Windbg.

As a side-note, you can get managed stack information without symbols as well. Symbols are required for source debugging and for seeing source file names and line numbers, but stack information in a managed application is obtained by other methods.

Okay, if I go on much longer, you'll lose patience and stop reading (if you haven't already), so I'll stop here for now.

Jim

Comments

• Anonymous
  April 12, 2006
  I may have said this before (my memory's not what it was prior to decade number 4), but the most common...

• Anonymous
  February 14, 2007
  I may have said this before (my memory's not what it was prior to decade number 4), but the most common

• Anonymous
  July 25, 2007
  The comment has been removed

• Anonymous
  July 25, 2007
  Tribina, An OOM can happen because of memory fragmentation. When the CLR needs to grow the managed heap, it requires a large block of contiguous memory of up to 64MB in size. If there isn't a large enough contiguous block available, an OOM will occur. There are many causes for a fragmented memory space. Your best bet is to open a case with PSS. We can get a dump of the process when the OOM occurs and debug it to see what's going on. Jim

• Anonymous
  July 26, 2007
  Thank you for replying. How do you the processes dump?  I've used Debug Diagnostic Tool before.  Is this what you mean?  Also, when should the dumps be taken?  When start experiencing the OOMs? I want to preemptively do this before placing a  call to PSS if I can. Thanks.

• Anonymous
  July 26, 2007
  DebugDiag will work fine, but you'll need to set a Registry key and then set a breakpoint. Add the following registry key.  This was added to the .NET Framework so that a breakpoint exception will be thrown when and OutOfMemory condition occurs.  This is documented in the following article: 820745 Failfast occurs when you experience an "out of memory" condition http://support.microsoft.com/?id=820745 HKEY_LOCAL_MACHINESOFTWAREMicrosoft.NETFramework DWord:  GCFailFastOnOOM Value:  2 Configure DebugDiag to capture the memory dump when the BreakPoint Exception is thrown and when the process stops.

1.  Open DebugDiag
2.  On the Rules tab, click Add Rule
3.  Select Crash and click Next
4.  Select "All IIS Processes" and click Next
5.  Click Advanced Exception Configuration
6.  Click Add Exception
7.  Select Breakpoint Exception, change Action Type to Full UserDump
8.  Click OK
9. Click Save and Close
10. Click Advanced Breakpoint Configuration
11. Click Add Breakpoint
12. Type KERNEL32!ExitProcess and change Action Type to Full UserDump
13. Click OK
14. Click Save and Close
15. Click Next through the rest of the wizard Inject LeakTrack.dll to capture native leak information:
16. Click the Processes tab
17. Right-click the ASPNET_WP.exe process (or w3wp.exe if running in IIS 6), select "Monitor for leaks" The memory dump will automatically get captured when the OutOfMemoryException is thrown. You can then call PSS to open a case. Tell the support engineer that you get that I have told you to get a dump and they can contact me to debug it. Jim