ASP.NET Memory Issues - High Memory Usage with AjaxPro (fixed in current version)
I was helping a colleague out with an OOM (OutOfMemory) situation he was dealing with.
Problem description:
Their applications memory usage would grow over time until they finally ended up with an out of memory exception.
First debug:
They had gotten a memory dump when memory usage was really high 1.4 GB using debug diag and I opened it up in windbg.exe, loaded up sos (.loadby sos mscorwks) and ran !dumpheap -stat to get the content of the GC heaps.
0:028> !dumpheap -stat
Statistics:
MT Count TotalSize Class Name
...
7ae77c54 59,028 1,416,672 System.Drawing.Color
663bb5a0 16,530 1,454,640 System.Web.UI.WebControls.RequiredFieldValidator
66411ac8 67,503 1,620,072 System.Web.UI.WebControls.Unit
663bd0a0 20,581 1,646,480 System.Web.UI.WebControls.Label
663bc55c 17,218 1,721,800 System.Web.UI.WebControls.DataGridItem
663c3f04 148,559 1,782,708 System.Web.UI.WebControls.FontInfo
7912dd40 16,219 1,937,904 System.Char[]
7a75a878 126,071 2,017,136 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
6640a7cc 32,408 2,074,112 System.Web.UI.HtmlControls.HtmlGenericControl
65431bb4 3,483 2,185,332 System.Collections.Generic.Dictionary`2+Entry[[System.Data.DataRow, System.Data],[System.Data.DataRowView, System.Data]][]
6540b178 31,055 2,235,960 System.Data.DataColumnPropertyDescriptor
79101fe4 50,490 2,827,440 System.Collections.Hashtable
65421898 59,458 2,853,984 System.Data.Common.ObjectStorage
6540addc 121,403 2,913,672 System.Data.DataRowView
6540b09c 71,366 3,140,104 System.Data.Common.StringStorage
79131488 3,149 3,239,368 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.Resources.ResourceLocator, mscorlib]][]
664162d4 104,417 3,759,012 System.Web.UI.EmptyControlCollection
66401a78 98,714 4,343,416 System.Web.UI.WebControls.Style
663eb1d0 283,592 4,537,472 System.Web.UI.AttributeCollection
79102290 389,910 4,678,920 System.Int32
79131840 4,588 4,753,168 System.DateTime[]
663dd2b0 110,687 4,870,228 System.Web.UI.WebControls.TableItemStyle
65407d48 16,545 4,897,320 System.Data.DataTable
664140a4 89,233 5,353,980 System.Web.UI.LiteralControl
79104368 247,402 5,937,648 System.Collections.ArrayList
66412f04 227,678 6,374,984 System.Web.UI.WebControls.ListItem
7912d7c0 107,276 8,479,900 System.Int32[]
663c7308 297,688 10,716,768 System.Web.UI.ControlCollection
6641194c 700,898 11,214,368 System.Web.UI.StateBag
7912dae8 14,050 11,661,948 System.Byte[]
7a7580d0 764,922 15,298,440 System.Collections.Specialized.HybridDictionary
663d7328 219,634 18,449,256 System.Web.UI.WebControls.TableCell
7912d9bc 54,737 18,666,456 System.Collections.Hashtable+bucket[]
663c1de8 1,203,791 19,260,656 System.Web.UI.StateItem
7a75820c 690,058 19,321,624 System.Collections.Specialized.ListDictionary
6641f33c 444,421 19,554,524 System.Web.UI.Control+OccasionalFields
7a7582d8 1,198,303 23,966,060 System.Collections.Specialized.ListDictionary+DictionaryNode
79105a0c 1,048,528 25,164,672 System.Guid
654088b4 170,538 25,239,624 System.Data.DataColumn
654359c8 11,068 25,859,792 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
65412bb4 24,380 42,023,632 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
65408b8c 761,406 48,729,984 System.Data.DataRow
7912d8f8 579,883 115,071,292 System.Object[]
000dc6e8 247 241,601,256 Free
790fd8c4 3,524,118 373,698,368 System.String
Total 15,860,201 objects, Total size: 1,214,968,272
The output above, showing the most memory consuming objects, tells us that there are pretty much two types of objects that consume most of the memory. Data related items and UI related items.
Notes about finding lots of UI objects on the heap
If you have followed my blog, you have probably noticed that I have spoken about this particular pattern before, with many UI related items, and that it is usually due to storing user controls in session or cache, and/or using static controls that where you set up event handlers in the page class to handle events for these static controls.
Basically, anytime you store a control or a UI item in session scope, cache or static variables, you will also hold a reference to the page that it was created on, as well as any controls it has and any data that might be databound to any of the controls on the pages. In other words, until the control goes out of scope, i.e. is removed from cache or session state, these objects will not be available for garbage collection.
Here are a few posts that I have written on the topic:
ASP.NET Memory: Thou shalt not store UI objects in cache or session scope
ASP.NET Quiz Answers: Does Page.Cache leak memory?
.NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon
Next debugging actions
When you see a lot of UI objects on the heap like this, the next step is to figure out why they are sticking around. One of the things I will always do first in these cases is to look at aspx and ascx pages on the heap and !gcroot them to see where they are rooted, i.e. what is keeping them in memory.
0:028> !dumpheap -type *aspx
Using our cache to search the heap.
Address MT Size Gen
...
6d43ede4 10303dd4 408 2 ASP.default_aspx
6d59eb50 10303dd4 408 2 ASP.default_aspx
6d67cb28 10303dd4 408 2 ASP.default_aspx
6d97e558 10303dd4 408 2 ASP.default_aspx
6df13390 10303dd4 408 0 ASP.default_aspx
6e6b53f0 10303dd4 408 0 ASP.default_aspx
Statistics:
MT Count TotalSize Class Name
...
10303dd4 436 177,888 ASP.default_aspx
Total 1,230 objects, Total size: 573,544
In this case there were 1230 aspx pages on the heap, in reality there should be approximately one per currently executing request, so this tells us that they are definitely staying longer than they should since I couldn't find a single thread executing a request when I printed out all the callstacks with ~* e !clrstack.
I gcrooted one of the pages to see why it is sticking around and found that it was stored in a static HybridDictionary...
0:028> !gcroot 6d67cb28
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 11 OSTHread 5d8
Scan Thread 15 OSTHread 19c
Scan Thread 16 OSTHread e8c
Scan Thread 17 OSTHread c0c
Scan Thread 9 OSTHread 12f0
Scan Thread 18 OSTHread 15f0
Scan Thread 4 OSTHread 1138
Scan Thread 5 OSTHread 120c
Scan Thread 3 OSTHread 1664
Scan Thread 22 OSTHread 12e8
Scan Thread 25 OSTHread 1098
DOMAIN(000FA020):HANDLE(Pinned):22211f0:Root: 0a629ac8(System.Object[])->
02777208(System.Collections.Specialized.HybridDictionary)->
06946578(System.Collections.Hashtable)->
3eac4e0c(System.Collections.Hashtable+bucket[])->
6d67cb28(ASP.default_aspx)
And if I run !objsize on this Hybrid dictionary I find that it holds on to about 941 MB of data so this is certainly very interesting...
0:028> !objsize 02777208
sizeof(02777208) = 941,224,148 ( 0x3819f0d4) bytes (System.Collections.Specialized.HybridDictionary)
I have to add a small caveat here... if you !objsize something that contains an aspx page, your objsize will include the size of the cache since the page has an indirect reference to the cache.
0:028> !dumpheap -type System.Web.Caching.Cache
Using our cache to search the heap.
Address MT Size Gen
0662fe54 6639d878 12 2 System.Web.Caching.Cache
Statistics:
MT Count TotalSize Class Name
6639d878 1 12 System.Web.Caching.Cache
Total 1 objects, Total size: 12
0:028> !objsize 0662fe54
sizeof(0662fe54) = 10,770,512 ( 0xa45850) bytes (System.Web.Caching.Cache)
In this case though the cache is very small so most of the memory held up by this HybridDictionary is the pages itself.
Ok, so now we know that our issue is due to the fact that we have a lot of pages in memory, and they are sticking around in a static HybridDictionary. The next step is to find out what this hybrid dictionary is and who is populating it with pages, and why...
To do this I dump out the HashTable+bucket[] that contains the page and search for the page address (6d67cb28) and the result was the entry displayed below
0:028> !da -details 3eac4e0c
...
[132] 3eac5444
Name: System.Collections.Hashtable+bucket
MethodTable 791021d8
EEClass: 79102154
Size: 20(0x14) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
MT Field Offset Type VT Attr Value Name
790fd0f0 4000937 0 System.Object 0 instance 6d67cb28 key
790fd0f0 4000938 4 System.Object 0 instance 6d67d388 val
79102290 4000939 8 System.Int32 1 instance 27189591 hash_coll
...
The page seems to be stored as the Key of this entry, so someone is populating a HybridDictionary with a key/value pair, where key=<the page>...
The value in this case is a ListDictionary
0:028> !do 6d67d388
Name: System.Collections.Specialized.ListDictionary
MethodTable: 7a75820c
EEClass: 7a75819c
Size: 28(0x1c) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
MT Field Offset Type VT Attr Value Name
7a7582d8 4001157 4 ...ry+DictionaryNode 0 instance 6d67d3a4 head
79102290 4001158 10 System.Int32 1 instance 4 version
79102290 4001159 14 System.Int32 1 instance 4 count
79115ea8 400115a 8 ...ections.IComparer 0 instance 00000000 comparer
790fd0f0 400115b c System.Object 0 instance 00000000 _syncRoot
and if we print out the first entry of the listdictionary (the head node) we find that the value is Ajax.NET.Prototype...
0:028> !do 6d67d3a4
Name: System.Collections.Specialized.ListDictionary+DictionaryNode
MethodTable: 7a7582d8
EEClass: 7a7c4220
Size: 20(0x14) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
MT Field Offset Type VT Attr Value Name
790fd0f0 4001167 4 System.Object 0 instance 027773d8 key
790fd0f0 4001168 8 System.Object 0 instance 6d67d2e8 value
7a7582d8 4001169 c ...ry+DictionaryNode 0 instance 6d67d44c next
0:028> !do 027773d8
Name: System.String
MethodTable: 790fd8c4
EEClass: 790fd824
Size: 54(0x36) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: Ajax.NET.prototype
Fields:
MT Field Offset Type VT Attr Value Name
79102290 4000096 4 System.Int32 1 instance 19 m_arrayLength
79102290 4000097 8 System.Int32 1 instance 18 m_stringLength
790ff328 4000098 c System.Char 1 instance 41 m_firstChar
790fd8c4 4000099 10 System.String 0 shared static Empty
>> Domain:Value 000d5d68:790d884c 000fa020:790d884c <<
7912dd40 400009a 14 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 000d5d68:026203f4 000fa020:0262429c <<
To figure out where this comes from, I saved out all the modules to disc using !for_each_module
!savemodule ${@#Base} f:\blog\modules\${@#ModuleName}.dll
as described in https://blogs.msdn.com/tess/archive/2008/01/10/using-reflector-to-search-through-code-and-resolve-net-issues.aspx.
I loaded them all up in reflector, and did a string search for Ajax.NET.Prototype, which returned a method containing this string(AjaxPro.Utility.RegisterCommonAjax, in AjaxPro.dll)
Armed with this, I searched the internet for "AjaxPro memory leak HybridDictionary" and found https://www.ajaxpro.info/changes.txt which shows a list of changes in the AjaxPro library. Of particular interest is a fix made in version 6.4.27.1
Version 6.4.27.1 (beta) - Fixed null values to DBNull.Value for System.Data.DataTable. - Fixed memory leak with HybridDictionary for JavaScript include rendering.
And the solution here was to upgrade to the latest version of AjaxPro.
About AjaxPro
Normally I don't write about 3rd party products, especially issues with 3rd party products, but I know that AjaxPro is a nice AJAX library that is used by quite a few of our customers so I hope this post is of general interest, as this issue is resolved in a later version of AjaxPro. I also hope that the post can be of use for finding other similar issues outside of AjaxPro.
I should also mention that I am posting this with the permission of Michael Schwartz who developed AjaxPro, and I must say that I was extremely impressed with the openness in his response when I asked if it was ok to post about this. His comment was, "feel free to write about it, I love to hear critics if there are any... to improve developement and/or fix bugs"
AjaxPro is now open source and can be downloaded from codeplex here https://www.codeplex.com/AjaxPro/. If you are interested in this you should also visit Michaels blog at https://weblogs.asp.net/mschwarz/
Laters,
Tess
Comments
Anonymous
September 12, 2008
Hello, I'm using AjaxPro, too, and I am very happy with the library. I have another question: is it possible to get a dump of a running system without interacting/stopping the users?Anonymous
September 14, 2008
yes, you can take a memory dump with adplus (debugging tools for windows) in hang mode (adplus -pn w3wp.exe -hang) or a full dump with debug diag 1.1. It will stop the process momentarily but usually not long enough for the users to notice.Anonymous
September 15, 2008
Awesome! You've used some many tools that I've never heard of before. I do have some sites that tend to grow in memory usage more than others, and it's a complete mystery. I hope I'll be able to reproduce some of the steps you showed. Thanks.Anonymous
September 20, 2008
In some cases you may end up with the following exception when working with RadControls for ASP.NET AjaxAnonymous
September 27, 2013
i'm facing an issue with cache and div's, your blog explains why i should not use cache with generic html objects - thanks you! mike from mentallica.co.il