Who is this OutOfMemory guy and why does he make my process crash when I have plenty of memory left?

To answer this question, there are a few concepts we need to discuss.

Working on a 32-bit system, you can address 4 GB of memory, out of this 2 GB is typically reserved to the Operating System and 2 GB are allowed for each user mode process, such as w3wp.exe (asp.net) for example. This memory is called virtual memory and the 2 GB’s are 2GB independently of how much RAM you have added to the system. The amount of RAM simply decides how much paging and swapping you will do, i.e. how fast memory access will be.

When a process allocates memory it does so by first reserving virtual memory and then committing memory out of this chunk (this is the memory actually used). The committed memory is called private bytes.

The virtual address space is used for a number of different items in the process such as:

• Dll’s
• Native heaps (non .net heaps)
• Threads (each thread reserves 1 MB for the stack)
• .net heaps (for managed variables)
• .net loader heap (for assemblies and related structures)
• Virtual allocations made by com components

Virtual memory allocations are not necessarily (or very seldom) lined up nicely in memory. For example, dll’s have preferred loading addresses so gaps are left between them, and virtual allocations that have been de-allocated will also leave gaps. This means that even though you can address 2 GB of virtual memory you may not be able to use it all since when enough memory is used, the memory will look somewhat like a Swiss cheese and your plug might have a big enough hole to fit in.

This is what happens when you get an OutOfMemory exception.

I will likely talk more about .net memory management later, but for now I’m going to make it very brief since there are several very good blogs about this such as Maoni's CLR Performance blog https://blogs.msdn.com/maoni/ and https://blogs.msdn.com/yunjin .

In the .net framework the garbage collector which is our memory manager reserves virtual memory in heaps. Once these heaps are created and we create a new instance of a .net object this object is stored in these heap segments and memory is committed.

In the .net framework 1.1 (server version) the regular .net heaps are created in 64 MB segments. (variations occur if you have 8+ processors or have changed the settings manually but I will ignore this for now and talk about the common case)

These 64 MB segments need to be allocated in one big chunk, you can’t allocate 32 MB here and 32 MB there, so when you have filled up one of these 64 MB segments and a new segment needs to be created you will need to find an empty gap of 64 MB or more in the 2 GB memory space. If such a gap does not exist you will get an out of memory exception.

These OutOfMemory exceptions are generally not fatal but still bad as they leave the process in an unstable state. However, when the garbage collector needs to do a collection it requires space for internal structures in order to do this collection, and the more memory and objects that you use, the bigger these structures are. If the garbage collector does not have enough space for its collections you will get a fatal out of memory exception and the process will die.

As you may have gathered from the above the reserved memory (virtual bytes in performance monitor) will be much larger than the committed bytes (private bytes in performance monitor). Usually the difference will be somewhere around 500 MB when memory usage is high, and at about 1.2-1.4 GB or so virtual bytes it starts getting very hard to find these 64 MB holes which means that a normal .net process will start getting OutOfMemory exceptions at around 800 MB.

Again, this value will differ depending on your application, the dll’s that are loaded and memory allocation patterns of native com components etc.

Now you know why you get OutOfMemory exceptions even though you have a lot of RAM, the next task is to find out why you are using this much memory and I will spend some time on that in future blog posts.

Comments

• Anonymous
  February 02, 2006
  The comment has been removed

• Anonymous
  February 02, 2006
  Hi Mark,
  
  I would recommend that you follow the steps in:
  http://support.microsoft.com/?id=820745
  To set the GCFailFastOnOOM registry key to 2, and then gather a crash dump with adplus -crash.
  
  If you get OOM's at 400 MB, there are a few possibilities.
  
  1) your heap is severly fragmented
  2) you are allocating a huge object
  3) you get an OOM for some other reason than actually running out of virtual memory. Some error codes in COM+ for example translate to OOM.
  
  Once you get a dump you can determine which one you are running in to like this.
  
  1) You can take a look at the stack making the allocation (and the eventlog) to find out how big of an allocation you are making. (I have seen cases where for example the code allocates an array with the size based on a return value from some function, and that return value turned out to be completely bogus), if it's a very big allocation you run into case #1
  
  2) run !address, this will give you info about all reserved and used virtual segments, at the bottom of this list there is information about the largest contigous free space. If this is very low, say below 64 MB already at 400 MB private bytes, you have pretty bad fragmentation of the virtual memory space. This can be caused by a number of things for example a) a very large number of threads that are constantly killed and created (each thread takes up 0,5 - 1MB of stackspace depending on OS) and if they are constantly killed and created they will be spread out in hte vm space causing fragmentation b) a COM object making many small virtual allocs c) a large number of dlls, that are spread out a bit unevenly in the vm space. etc.
  
  3) for the 3rd case, you would likely be able to identify this on the stack of the OOM when it happens.
  
  
  I would also recommend that if this is an ASP.NET app, that you check that you dont have debug=true in any of your apps as this causes a certain amount of fragmentation which is of course bigger if there is a large number of dlls.
  Also check that your dlls are compiled in releasemode.
  
  You can do this in a dump as well by running !finddebugtrue and !finddebugmodules (from sos.dll)
  
  
  

• Anonymous
  June 07, 2006
  The comment has been removed

• Anonymous
  September 05, 2006
  The comment has been removed

• Anonymous
  May 23, 2007
  Hello! You have an outstanding good and well structured site. Thank you!

• Anonymous
  November 06, 2007
  In .NET 2 there is no GCFailFastOnOOM, only: HKEY_LOCAL_MACHINESOFTWAREMicrosoft.NETFrameworkGCBreakOnOOM 1

• Anonymous
  June 18, 2008
  Could you comment on garbage collection mechanisms on network performance monitoring agents and their effect on management of agent lifetime and application throughput ?

• Anonymous
  October 16, 2008
  Bonjour, ce second article est consacré à la fragmentation mémoire sous IIS6. Notez cependant qu'il peut

• Anonymous
  February 10, 2009
  Thanks. Very helpful article because things are clearly explained.

• Anonymous
  September 15, 2010
  Hi Tess, Thanks for the great information. My question is how do these numbers work out for 64-bit OS, specifically with IIS? Does IIS take full advantage of the 64-bit address space? At what point do you start seeing OOM exceptions with a 64-bit Asp.Net application? Thanks, Don

• Anonymous
  September 15, 2010
  DonM, On 64-bit your virtual address space is so large that you will pretty much never see an OOM (if you run a 64-bit process) for 32-bit procs it'll be 4 GB so OOMs would appear around maybe 2.8 GB private bytes. For 64 on 64 you would run into a lot of other issues, such as slowness for variuos reasons because the app used so much memory long before you could run into an OOM.