次の方法で共有

.NET Memory Leak reader email: Are you really “leaking” .net memory

  • [アーティクル]

I get several emails every week through the blog asking for help on various issues.  Unfortunately due to time constraints I can’t really look at them all on an individual bases but I thought this particular request might be interesting to share.

The problem description was “One of our w3wp.exe’s just keep on growing (slowly), and never releasing memory back to the pot.  The main offender appears to be System.String and System.Object[]”  and the email then had some !dumpheap –stat output (from windbg and sos)

0:000> .load sos
0:000> !dumpheap -stat
...
654060a8     5622       292344 System.Data.SqlClient.SqlBuffer
79109778     5505       308280 System.Reflection.RuntimeMethodInfo
79131d04     1598       329732 System.Boolean[]
65408b8c     5568       356352 System.Data.DataRow
654359c8      160       395136 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
65412bb4      160       395136 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
6540b178     5664       407808 System.Data.DataColumnPropertyDescriptor
65405fdc     5741       734848 System.Data.SqlClient._SqlMetaData
654088b4     5664       838272 System.Data.DataColumn
7a75a878    55225       883600 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
7912dd40      709       891172 System.Char[]
79102290   100163      1201956 System.Int32
7910be50   116208      1394496 System.Boolean
7912d9bc     1685      2078376 System.Collections.Hashtable+bucket[]
7912d7c0    13131      2205228 System.Int32[]
79131840     1402      2206136 System.DateTime[]
79131b20      793      2485548 System.Decimal[]
7912dae8     1316      3573540 System.Byte[]
7912d8f8    24039      5152404 System.Object[]
790fd8c4   272294     21045644 System.String
Total 781834 objects

The reason I thought this request was interesting is because it brings up an important point about troubleshooting memory issues.

Typically when I work with a customer on a support case and they mention that they have memory issues I usually start off by asking:

How did you determine that you have a memory leak or high memory usage?

This might sound like a simple question, i.e. you determine that you have a memory leak because memory keeps growing, or you determine that you have high memory usage because the process is using a lot of memory.  However, it is usually a lot more complex than that.  

For some applications it is normal to use up 1 GB of memory and for others 200 MB is an extremely high number.  To determine what is high or not for your application you need to establish some type of baseline of how the application works under normal circumstances.  

Also when it comes to high memory usage and potential memory leaks it is important to know what memory counters you are looking at.  Is it virtual bytes, private bytes, .net memory (#Bytes in all heaps) and for when and how long you are looking at these counters.  

For example, when working with .net applications, virtual bytes will go up in chunks since we allocate GC heap segments in chunks.  How big these chunks are depend on what framework version you are using and if it is running the server GC or workstation GC.   Also, as I have mentioned several times in previous blog posts, garbage collections do not occur at time intervals but rather when you allocate memory and reach the limits set for different generations.  Therefore it is normal for a process to not return memory if no allocations are made.

For some discussions on virtual byte vs. private bytes, and how the GC works, you can have a look at:

.NET Memory Usage – A restaurant analogy
How does the GC work
Generational GC – A post-it analogy
ASP.NET Memory investigation
32 vs 64 bit memory usage
Defining the “where” of a memory leak

In the email above it was never stated how much memory would grow, or what it would grow to, or even how much private bytes the process was using when the dump was taken.  I realize that it’s hard to remember to write everything in an email, or know what is important informaiton, but without this data it is very hard to say anything at all about what caused a potential memory increase.

Looking at the dumpheap –stat output above we can see that the process has roughly 40 MB of .net memory (by just adding up the major memory consumers).   This to me is not a very big number, but again, that completely depends on what the normal memory usage of the process is.   I often find that people stare themselves blind on the .net memory usage, but I think that before doing that, you need to use some critical thinking and determine if the 40 MB is even worth looking at.  If the process is using up 700 MB at this time for example, then the 40MB is just a drop in the ocean, and any statements made about it are irrelevant.

If the private bytes at this time is around 150 MB, then the question becomes, is that really a big number? And you need to consider whether you have taken a memory dump before it started “leaking”.  If the 150 MB is a high number for you, you need to look at what is using up the other 110 MB and you will probably find that a lot of it is going to assemblies etc.

Ideally if you have a perceived memory leak, you should take two memory dumps and compare the growth of the .net heap between them to see if the growth in the process is due to growth in .net memory.  Or an easier way of doing this would be to compare the #Bytes in all heaps and private bytes over time.

So the conclusion here is that the !dumpheap output can’t be used for analysis without the supporting data around why/how and when you saw the problem.

Another interesting fact that was left out of the email was whether this application had always had this issue or if this started happening recently.  If it started after the application had been live for some time we need to look at what changed. 

System.String and array objects

Finally, I would like to address the comment around System.String and System.Object[] being the major offenders.   I wrote an article in the beginning of this blog around examining !dumpheap –stat output, because it is very common that people focus in on the string and object[] objects when reviewing memory dumps.

The reason for this is that arrays and strings will always end up on the bottom of the !dumpheap –stat output since !dumpheap –stat will list objects by the total size of such objects.   This happens because a) pretty much everything uses strings and arrays so there will always be a large amount of strings and arrays in any dump and b) because of how the size is calculated.

If you take an object like a dataset, the dataset basically consists of a few different pointers to data tables, dataset name etc.  all in all the space used for the dataset object is around 80 bytes (just the pointers).  The actual datatables and its rows and columns etc. are not really a part of the dataset, they are just linked from the dataset.  Therefore if you have 10 datasets they would take up 800 total bytes, and if you have a 100 it would be close to 8kb,  independently of how many rows/columns, blobs etc. are in the columns and rows.   (note: actual dataset size varies with framework versions)

The size of strings and array objects however are variable.  The size of a string is the actual size of it’s contents, and the size of an array is the size of all the pointers to the elements.    Therefore the total size of all strings and arrays will for the most part hugely surpass the size of anything else in the process and thus they will always end up at the bottom.

I would say, if you look at a memory leak,  forget about the arrays and strings unless they end up on the LOH (!dumpheap –min 85000) in which case you might want to look at the ones that end up on the LOH separately.

In summary

If you are doing a memory leak investigation, first get facts about actual memory usage, baseline memory usage, how memory grows over time etc. to get a clear picture of if you are actually looking at a “leak” or normal behavior, and if you determine that it is in fact an issue in that the app is using a lot more .net memory than expected, look beyond the strings an arrays on the first pass through the data.

 

Have fun,

Tess

Comments

  • Anonymous
    February 27, 2009
    I get several emails every week through the blog asking for help on various issues.  Unfortunately

  • Anonymous
    March 01, 2009
    Tess, as always, very useful information.

  • Anonymous
    April 29, 2009
    great blogs, and you are pretty too :).

  • Anonymous
    May 11, 2009
    I have put together a quick and dirty debug diag script for troubleshooting .net memory leaks. (attached

  • Anonymous
    June 15, 2009
    Hi Tess, it would be great if you can add more tags for your posts such that the articles can be further categorized. It becomes easy that way to search for relevant posts.

  • Anonymous
    July 14, 2009
    I didn't noticed memory leaking before, till I was asked in an interview. :)