ASP.NET Memory Investigation

Artikkeli
08/13/2007

This is a bit of a continuation of ASP.NET Memory Issue: High memory usage in a 64bit w3wp.exe process so if you haven't checked it out you might want to just glance over it before reading this one to get the context of the problem and some notes on 64 bit debugging.

Before I go into the details I just want to mention that what I will talk about does not only apply to 64 bit debugging even though I am using a 64 bit dump, you can just as easily see the problems I will talk, about and use the same techniques to discover them in a 32 bit process/dump.

In my previous posts the dumps have normally been pretty clean, and a lot of times they have only been suffering from one single memory issue, to make it easier to illustrate how a small detail can cause a big havoc. As we know though, the world is seldom or never black and white and in many cases there are more than one reason for high memory usage in a process.

This particular dump has various high memory consumers (apart from the issue described in the previous post) so it gives me an opportunity to show a few different common high memory culprits.

In order to not give out any proprietary information I have changed the contents of strings and names of certain custom classes in this dump, so if you are a nitpicker and find inconsistencies in string lengths or similar, that is why:)

Debugging details of the memory investigation

I am going to concentrate on the .NET GC heaps in this case, and freely ignore any objects that are not rooted, and for now attribute them to the issue described in the previous post.

The GC Heaps account for 1,502,201,224 bytes out of the 1,6 GB used for the dump, and out of this approximately 800 MB-1GB is Gen 0 objects (see the previous post), so the interesting thing is to figure out where the rest is going, I.e. the memory that is really sticking around.

Looking at the bottommost ~20 lines of !dumpheap -stat, showing the individual objects that used up the most memory, we get...

 0:000> !dumpheap -stat
...
0x0000064253bda8f8     23,805     5,713,200 System.Xml.Schema.XmlSchemaElement
0x00000642bcf83908    126,788     6,085,824 System.Web.UI.WebControls.ListItem
0x00000642bcee8240     53,173     6,380,760 System.Web.UI.LiteralControl
0x00000642b77fcfb0     13,351     6,835,712 System.Data.DataTable
0x00000642bcee9a28    245,535     7,857,120 System.Web.UI.Pair
0x00000642bceabd08    263,432     8,429,824 System.Web.UI.StateBag
0x00000642788e5a88    172,973     9,714,456 System.Int32[]
0x00000642788405d8    124,338    10,941,744 System.Collections.Hashtable
0x00000642b7800d20    249,035    11,953,680 System.Data.DataRowView
0x0000064274e57ce8    300,268    12,010,720 System.Collections.Specialized.HybridDictionary
0x00000642bcea8920    137,011    12,056,968 System.Web.UI.Control+OccasionalFields
0x0000064274e57e70    278,134    13,350,432 System.Collections.Specialized.ListDictionary
0x0000064278827060    566,228    13,589,472 System.Int32
0x0000064253bcd750    466,140    14,916,480 System.Xml.Schema.BitSet
0x00000642788bd790    466,141    14,916,544 System.UInt32[]
0x00000642bceca168    514,257    16,456,224 System.Web.UI.StateItem
0x0000064253bcd090    444,728    17,789,120 System.Xml.XmlQualifiedName
0x000006427883e400    457,745    18,309,800 System.Collections.ArrayList
0x00000642bceb4f98    137,019    20,826,888 System.Web.UI.WebControls.TableCell
0x0000064274e57fe0    559,345    22,373,800 System.Collections.Specialized.ListDictionary+DictionaryNode
0x0000064253bc1c80    571,126    22,845,040 System.Xml.NameTable+Entry
0x00000642b78659d0     14,096    24,055,168 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
0x0000064278824860    784,204    25,094,528 System.Decimal
0x000006427883f770    565,270    27,132,960 System.Collections.ArrayList+ArrayListEnumeratorSimple
0x00000642b77fe2f0    124,259    28,828,088 System.Data.DataColumn
0x00000642788dbf50     30,668    29,910,520 System.Byte[]
0x00000642b7820d60     19,193    34,319,192 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
0x00000642b77fe7c0    377,440    36,234,240 System.Data.DataRow
0x00000642788db830     71,095    38,538,856 System.Char[]
0x00000642788fe918    135,194    50,900,832 System.Collections.Hashtable+bucket[]
0x00000642788e3b70     48,047   138,801,256 System.Decimal[]
0x00000642788d4758    897,213   172,527,160 System.Object[]
0x000006427881aaf8  3,013,208   197,958,896 System.String
0x0000000000146e90     12,107   220,141,008      Free
Total 15,904,890 objects, Total size: 1,502,201,224

Fragmented blocks larger than 0.5 MB:
            Addr     Size      Followed by
00000000934858a0   25.3MB 0000000094dc8d50 System.Threading._ThreadPoolWaitCallback
0000000094dc8dc0   11.4MB 000000009592b4e0 System.Threading._ThreadPoolWaitCallback
000000009592b550    2.3MB 0000000095b6be48 System.Threading._ThreadPoolWaitCallback
0000000095b6beb8    2.1MB 0000000095d7a828 System.Threading._ThreadPoolWaitCallback
0000000095d7a898    0.7MB 0000000095e350b8 System.Threading._ThreadPoolWaitCallback
0000000095e35128    2.7MB 00000000960e7968 System.Threading._ThreadPoolWaitCallback
...

I have grouped them together by color to get a bit of an overview of what we are looking at... and categorized them like this

XML Related Items
Data Related Items
UI + Web Controls Related Items
State and Cache related items
Free

In some cases, like in the case of Data, i have added UInt32[] for example because I know from experience that a lot of these arrays will be arrays of UInt32 objects in datasets. The same goes for HashTables and ListDictionaries in Cache and Session State. Of course not all of these will be directly related to Data or Cache but for rough categorization it works well to put them in those buckets.

Byte[], Object[], Char[] and String are all types of objects that you will always see at the bottom of the !dumpheap -stat output since they are used everywhere, and you can read some of my previous memory investigation posts to get more details about this.

64-bit specific memory information

Since pointers are quad-word rather than DWORDs in 64-bit processes, a lot of the objects here, such as the Object[] and some of the collections (excluding strings and Byte[] for example) are double the size they would be in a 32-bit process.

Free

A lot of Free objects (objects that are marked collected, but the space has not been reused or compacted) usually means one of two things.

1. The garbage collector hasn't compacted in a while (usually the case on the workstation build)

2. You have a lot of pinned object disabling the GC from compacting the heaps

In this case, the reason is #1, but not because we are running the workstation build, but because we are running 64-bit framework RTM and have very infrequent collections. See the previous post for a lengthier discussion and resolution.

Xml Related Items

When we have a lot of Xml Related items in the bottom part of !dumpheap -stat I usually look for a more top-level object type like XmlDocument since all other Xml related objects usually tend to be members or members of members of XmlDocuments. In other words, if you were to get rid of the XmlDocument, the other ones would go as well. The reason I look for a top-level object is because there are usually fewer of them so it is easier to work with them in order to find out why they stick around.

 0x0000064253bc0c10      5,048     1,534,592 System.Xml.XmlDocument

And then i dump out a few and do !gcroot on them to find out why they are rooted...

 0:000> !dumpheap -mt 0x0000064253bc0c10      
Using our cache to search the heap.
   Address         MT             Size  Gen
0x00000000800dfb98 0x0000064253bc0c10      304    2 System.Xml.XmlDocument 
0x00000000800e3578 0x0000064253bc0c10      304    2 System.Xml.XmlDocument 
0x00000000800e6f58 0x0000064253bc0c10      304    2 System.Xml.XmlDocument 
0x00000000800ea938 0x0000064253bc0c10      304    2 System.Xml.XmlDocument 
0x00000000800ee318 0x0000064253bc0c10      304    2 System.Xml.XmlDocument 
0x00000000800f1cf8 0x0000064253bc0c10      304    2 System.Xml.XmlDocument 
0x00000000800f56d8 0x0000064253bc0c10      304    2 System.Xml.XmlDocument 
0x00000000800f90b8 0x0000064253bc0c10      304    2 System.Xml.XmlDocument 
0x00000000800fb8a0 0x0000064253bc0c10      304    2 System.Xml.XmlDocument 
0x00000000800fe5a8 0x0000064253bc0c10      304    2 System.Xml.XmlDocument 
0x0000000080102c28 0x0000064253bc0c10      304    2 System.Xml.XmlDocument 
0x0000000080106f58 0x0000064253bc0c10      304    2 System.Xml.XmlDocument 
0x000000008010a938 0x0000064253bc0c10      304    2 System.Xml.XmlDocument 
...


0:000> !gcroot 00000000800dfb98
...
DOMAIN(000000000017E210):HANDLE(Strong):4b41308:Root:  00000000c0020990(System.Threading._TimerCallback)->
  00000000c0020908(System.Threading.TimerCallback)->
  00000000c001b530(System.Web.Caching.CacheExpires)->
  00000000c001b568(System.Object[])->
  00000000c001bef8(System.Web.Caching.ExpiresBucket)->
  00000001008fbf40(System.Web.Caching.ExpiresPage[])->
  00000001008fbff8(System.Web.Caching.ExpiresEntry[])->
  000000010001ebd8(System.Web.Caching.CacheEntry)->
  000000010001eb98(System.Web.SessionState.InProcSessionState)->
  00000000800dd340(System.Web.SessionState.SessionStateItemCollection)->
  00000000800dd3f0(System.Collections.Hashtable)->
  00000000802fe7e0(System.Collections.Hashtable+bucket[])->
  00000000800e09e8(System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry)->
  00000000800dfb98(System.Xml.XmlDocument) 
...

...so most of them are rooted in an InProcSessionState object which in turn is rooted in cache. (This is always the case for inproc session objects). So in other words, these XmlDocuments are stored in Session scope and more specifically in the session variable called Classes.

 0:000> !do 00000000800e09e8
Name: System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
MethodTable: 0000064274e6c8c8
EEClass: 0000064274efd388
Size: 32(0x20) bytes
GC Generation: 2
 (C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
              MT            Field           Offset                 Type VT             Attr            Value Name
000006427881aaf8  400116d        8        System.String  0 instance 00000001400dd3a8 Key
0000064278818fb0  400116e       10        System.Object  0 instance 00000000800dfb98 Value



0:000> !do 00000001400dd3a8 
Name: System.String
MethodTable: 000006427881aaf8
EEClass: 000006427892f7d8
Size: 46(0x2e) bytes
GC Generation: 2
 (C:\WINDOWS\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: Classes
Fields:
              MT            Field           Offset                 Type VT             Attr            Value Name
0000064278827060  4000096        8         System.Int32  1 instance               11 m_arrayLength
0000064278827060  4000097        c         System.Int32  1 instance               10 m_stringLength
00000642788216d8  4000098       10          System.Char  1 instance               50 m_firstChar
000006427881aaf8  4000099       20        System.String  0   shared           static Empty
                                 >> Domain:Value  0000000000120a60:00000642787c36e0 000000000017e210:00000642787c36e0 <<
00000642788db830  400009a       28        System.Char[]  0   shared           static WhitespaceChars
                                 >> Domain:Value  0000000000120a60:00000000bfff0550 000000000017e210:00000000bfff7390 <<

If you want to dig deeper into this you may be interested in the following two posts:

ASP.NET Memory - How much are you caching? + an example of .foreach

ASP.NET Memory Leak Case Study: Sessions Sessions Sessions…

A few of the Xml related items such as some of the XmlQualifiedName objects for example actually turn out to be rooted in DataSets but I just want to do a pretty quick and dirty memory investigation here to get a good overview of what is going on.

Data Related Items

For the data related items i usually use System.Data.DataSet as a top-level item, and if I would do that in this case (following the same procedure as for XmlDocument) I would find that some of them are stored in session scope and some in cache directly.

State Related Items

If we have a lot of state related items that could be an indication of a couple of things

Lots of caching
Lots of in-proc session state (stored in cache)
Lots of aspx or ascx pages around with related viewstate (System.Web.UI.Pair, System.Web.UI.StateBag)

I am going to leave #3 for a bit and discuss the other two, since we already know that we have a lot of DataSets and XmlDocuments in Session and cache.

If we take a look at the cache and it's size we find that it is ~550 MB...

 0x00000642bcece978          1            24 System.Web.Caching.Cache

0:000> !dumpheap -mt 0x00000642bcece978          
   Address         MT             Size  Gen
0x00000000c001b288 0x00000642bcece978       24    2 System.Web.Caching.Cache 
...

0:000> !objsize 0x00000000c001b288 
sizeof(00000000c001b288) =  557,714,296 (  0x213e0b78) bytes (System.Web.Caching.Cache)

This is a pretty big chunk of the memory so I think it is safe to say that other than the issue described in the previous post, this is where we should focus our efforts if we want to reduce the amount of memory used.

If we then go one step further to try to see how much this is session objects we can dump out the InProcSessionState objects and check the size of one of those...

 0x00000642bcedf988        601        38,464 System.Web.SessionState.InProcSessionState

0:000> !dumpheap -mt 0x00000642bcedf988
   Address         MT             Size  Gen
0x00000000800d9ab8 0x00000642bcedf988       64    2 System.Web.SessionState.InProcSessionState 
0x00000000800dba40 0x00000642bcedf988       64    2 System.Web.SessionState.InProcSessionState 
0x00000000800dbb58 0x00000642bcedf988       64    2 System.Web.SessionState.InProcSessionState 
0x00000000800dbcc0 0x00000642bcedf988       64    2 System.Web.SessionState.InProcSessionState 
0x00000000800dbdd8 0x00000642bcedf988       64    2 System.Web.SessionState.InProcSessionState 
0x00000000800dbef0 0x00000642bcedf988       64    2 System.Web.SessionState.InProcSessionState 
0x00000000800dc008 0x00000642bcedf988       64    2 System.Web.SessionState.InProcSessionState 
...

0:000> !objsize 0x00000000800dc008 
sizeof(0x00000000800dc008) =  557,714,296 (  0x213e0b78) bytes (System.Web.SessionState.InProcSessionState)

Ok, so we have 601 active sessions and the size of just one of those sessions is the same amount as the cache itself which just doesn't make sense, and in fact offline I had taken !objsize of a sample of the datasets and XmlDocuments and although it is a sizeable chunk they will not rack up to 550 MB together, so something is fishy.

What is actually happening is that we are storing something in session scope that eventually has a link to a HttpContext, which has a link to cache, causing the actual size of the cache to be included in !objsize for the individual session objects.

When you see something like this, you should pay attention, because although the "circular reference" will be broken by taking the object out of session state, the problem is that this normally happens when you have added something to session state which has a link to an aspx page or ascx page or similar. Very much like in the case of the EventHandlers issue or in the CacheItemRemovedCallback, but as you will soon see, this is slightly different.

UI + Web Controls Related Items

The final stage of this memory investigation is to look at the UI and Web Controls related items. We already know that our memory can roughly be divided into memory usage caused by the issue in this post, and memory used for session state. However what we don't know is causing us to have so much session state since the data and xml related items don't seem to match the total size.

If we pick the TableCells and work out who is using them, just like in the data and xml cases, we get...

 0x00000642bceb4f98 137,019 20,826,888 System.Web.UI.WebControls.TableCell

0:000> !dumpheap -mt 0x00000642bceb4f98
Using our cache to search the heap.
   Address         MT             Size  Gen
0x00000000806321b0 0x00000642bceb4f98      152    2 System.Web.UI.WebControls.TableCell 
0x0000000080632850 0x00000642bceb4f98      152    2 System.Web.UI.WebControls.TableCell 
0x0000000080632a38 0x00000642bceb4f98      152    2 System.Web.UI.WebControls.TableCell 
0x0000000080632b90 0x00000642bceb4f98      152    2 System.Web.UI.WebControls.TableCell 
0x0000000080632ce8 0x00000642bceb4f98      152    2 System.Web.UI.WebControls.TableCell 
0x0000000080632e40 0x00000642bceb4f98      152    2 System.Web.UI.WebControls.TableCell 
0x0000000080632f98 0x00000642bceb4f98      152    2 System.Web.UI.WebControls.TableCell 
0x00000000806331b0 0x00000642bceb4f98      152    2 System.Web.UI.WebControls.TableCell 
0x0000000080633308 0x00000642bceb4f98      152    2 System.Web.UI.WebControls.TableCell 
0x0000000080633460 0x00000642bceb4f98      152    2 System.Web.UI.WebControls.TableCell 
0x00000000806335b8 0x00000642bceb4f98      152    2 System.Web.UI.WebControls.TableCell 
0x0000000080633710 0x00000642bceb4f98      152    2 System.Web.UI.WebControls.TableCell 

0:000> !gcroot 0000000080632850
...
DOMAIN(000000000017E210):HANDLE(WeakSh):4b412f8:Root:  00000000c0020b80(System.Threading._TimerCallback)->
  00000000c0020af8(System.Threading.TimerCallback)->
  00000000c001df60(System.Web.Caching.CacheExpires)->
  00000000c001df98(System.Object[])->
  00000000c001e928(System.Web.Caching.ExpiresBucket)->
  00000000c0adbf78(System.Web.Caching.ExpiresPage[])->
  00000000c0adc030(System.Web.Caching.ExpiresEntry[])->
  0000000140122e48(System.Web.Caching.CacheEntry)->
  0000000140122e08(System.Web.SessionState.InProcSessionState)->
  00000000c01632a0(System.Web.SessionState.SessionStateItemCollection)-> 
  00000000c0163350(System.Collections.Hashtable)->
  00000001002feb20(System.Collections.Hashtable+bucket[])->
  00000000c0403398(System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry)->
  000000008062b008(MyControls.MyCustomGridControl)-> 
  0000000080611c68(System.Web.UI.WebControls.DataGrid)->
  0000000080611da0(System.Web.UI.Control+OccasionalFields)->
  0000000080629c20(System.Web.UI.ControlCollection)->
  0000000080632098(System.Object[])->
  0000000080631fe8(System.Web.UI.WebControls.ChildTable)->
  0000000080632660(System.Web.UI.Control+OccasionalFields)->
  00000000806326b8(System.Web.UI.WebControls.Table+RowControlCollection)->
  00000000806326f0(System.Object[])->
  0000000080632780(System.Web.UI.WebControls.DataGridItem)->
  00000000806329a8(System.Web.UI.Control+OccasionalFields)->
  0000000080632a00(System.Web.UI.WebControls.TableRow+CellControlCollection)->
  00000000806330f0(System.Object[])->
  0000000080632850(System.Web.UI.WebControls.TableCell)

So it appears that for some reason we are storing a web control (MyControls.MyCustomGridControl) in session state...

This may not look like a very bad idea, in fact I can completely see why this is done, since it is a fast and easy way to store the state of the data that this particular user was working with and use this throughout the session, but there are some major issues with this...

All the data related to the control will now be linked to session and can't be garbage collected before this object goes out of session, while what we really wanted was to just store the actual data and perhaps the rowfilters. And among this data is not only UI related items, like WebControls.DataGridItem etc. which will be relatively useless to us once the page that held the grid has finished executing... but there is also a link to the page it was loaded on in multiple ways.

1. In the parent field of the control
2. As the target for many of the events related to the grid (like in the EventHandlers memory investigation)

And since we have a link to the page, we also keep everything that belongs to it in session state, such as its viewstate, its other controls etc. which can definitely be a whole lot of data, and this data can not be collected until the session expires.

This also explain why we have a circular reference with the cache going on, since the page has a context which links to the cache...

So what should you do instead of storing controls in session scope? The best would probably be to store the data you need in a separate object and store that in session scope if it takes too long time or is otherwise impractical to gather when needed.

In summary

When it comes to memory investigations where most of the memory is .NET GC memory, the memory investigation is often a paint-by-numbers process where the steps go

Run !dumpheap -stat and look for items that stick out, or try to group/categorize the bottommost items in the list to figure out who are your main memory hoggers
!gcroot the objects to figure out why they are still around... (Note! Make sure you !gcroot a couple of them before making any big decisions based on the results)
Rinse and repeat until you have accounted for most of the memory.

The tricky part is of couse figuring out what sticks out, grouping them properly, and understanding the results of !gcroot:) but hopefully this blog helps you with some of that.

Until next time,

Tess

Comments

Anonymous
August 13, 2007
Tess - Can you post the offending code fragment? I'm assuming its something simple like: Session.Add("control", MyControls.MyCustomGridControl); And a valid work around (as you are suggesting) would be to do something like: Session.Add("Control_Sort", MyControls.MyCustomGridControl.SortExpression); Session.Add("Control_SelectedIndex", MyControls.MyCustomGridControl.SelectedIndex) etc.
Anonymous
August 13, 2007
Hi Christopher, Yes the offending code fragment would look something like your Session.Add... What you can actually store in session depends on if it has some links back to the control itself or not. For example a SortExpression or a SelectedIndex is usually a string or an integer so that would work well. Depending on how much state you would need to keep you could create a class containing the data from the control and sort expressions and selected index or whatever you need to store. The point being that the control itself has several links back to the page which makes it unsuitable for session storing.
Anonymous
August 14, 2007
Tess - Thanks - I noted your statement about a separate object for state, I just didn't want to write the full code in the comment ;-) Is there any winDB statement that would list all objects with a reference to a given object such as System.Web.SessionState.InProcSessionState? Sort of an opposite of gcroot?
Anonymous
August 14, 2007
There isnt really an opposite of gcroot and it is hard to invision one since there is usually a very small roots to each object but could be almost an infinite number of leaf nodes. It would perhaps be interesting to see them in a dumpheap -stat fashion, and i believe there are some commands in the works like that for specific types of objects like datasets, sessionstate etc. Until then, you will have to work with windbg scripting like .foreach
Anonymous
September 10, 2007
The comment has been removed
Anonymous
August 05, 2008
Is this a copy-paste error? In "state related items" you ask for the size of the same address/object 2 times. The result is the same, but the type of object is different. 1st try: 0:000> !objsize 0x00000000c001b288 sizeof(00000000c001b288) = 557,714,296 ( 0x213e0b78) bytes (System.Web.Caching.Cache) 2nd try: 0:000> !objsize 0x00000000c001b288 sizeof(0x00000000c001b288) = 557,714,296 ( 0x213e0b78) bytes (System.Web.SessionState.InProcSessionState)
Anonymous
August 05, 2008
Nice catch Jack, yes it must have been a copy-paste error... I changed the object address now
Anonymous
October 30, 2008
Where are you entering these commands? These wont work at the dos prompt level: C:>!dumpheap -stat '!dumpheap' is not recognized as an internal or external command, operable program or batch file. C:>
Anonymous
October 30, 2008
Hi Jon, Those are commands to enter in windbg.exe (part of debugging tools for windows). Look at the memory lab (on the left hand side) for better instructions on how to do this.
Anonymous
February 26, 2009
I get several emails every week through the blog asking for help on various issues.  Unfortunately
Anonymous
March 03, 2009
Tess, You are my hero! Another great post that helped me to resolve my customer's problem... THANK YOU!
Anonymous
May 11, 2009
I have put together a quick and dirty debug diag script for troubleshooting .net memory leaks. (attached
Anonymous
June 26, 2010
Hi Tess, Thanks for your labs - they were a great help. We are facing some serious memory increase in our application in production. A few weeks ago the w3wp processes never increased beyond 620 MB in task manager but now they go upto 750 MB-800MB. I took a dump using procdump. I ran !eeheap -gc and it said total size is 523 MB but at the time I took the dump the process size was 780 MB in task manager. What would this mean? Eventually the app domain recycles saying that the private bytes memory limit has been reached. I also ran the command - !dumpheap -type *ascx to find that there are many ascx pages in Generation 2 but when I run a gcroot it looks like nothing is holding onto these pages. Please advise as we are facing huge losses in production.
Anonymous
July 04, 2010
Hi Vandana, The difference between the memory used for the process and !eeheap -gc can be anything from memory used for the dlls loaded or other native memory. Given that it's a very small difference comparatively it is probably just the dlls. It's hard to say without looking at it why they wouldnt be rooted, I would continue looking at them to see if you find any of them rooted as it might just have been that you picked one that wasn't rooted. It could of course also be that you just recently had many requests so you have a lot of them in memory that are ready for collection. /Tess

Jaa

ASP.NET Memory Investigation

Comments

Lisäresursseja