High Memory due to System.WeakReference
We recently saw an issue that was manifesting as very high memory utilization by a w3wp.exe process that was hosting an ASP.net application. To start troubleshooting, we gathered a memory dump of the process when its memory usage was very high.
A standard way of starting out the debugging of a high memory problem in a managed application is to run the !dumpheap –stat command. When we did so, we found millions of 16 byte System.WeakReference objects (I’ve truncated the output for brevity’s sake):
0:000> !dumpheap -stat
Loading the heap objects into our cache.
total 28,603,114 objects
Statistics:
MT Count TotalSize Class Name
0x648efc44 4,199 83,980 System.Configuration.ConfigurationValue
0x6639b220 4,368 87,360 System.Web.VirtualPath
0x6639e3fc 4,534 90,680 System.Web.Caching.CacheKey
0x663b4074 2,063 90,772 System.Web.HttpCookie
0x648ea704 2,003 96,144 System.Configuration.ConfigurationValues
0x79131840 99 104,612 System.DateTime[]
0x663b27f4 2,066 107,432 System.Web.HttpValueCollection
0x663aa5e8 2,072 107,744 System.Web.HttpModuleCollection
0x66435948 2,073 107,796 System.Collections.Generic.Dictionary`2
0x65409d70 1,772 113,408 System.Data.SimpleType
0x663ada24 4,126 115,528 System.Web.Hosting.RecyclableCharBuffer
0x663aa6f8 4,126 132,032 System.Web.HttpAsyncResult
0x7910d7e8 4,148 132,736 System.AsyncCallback
0x6641f33c 3,028 133,232 System.Web.UI.Control+OccasionalFields
0x65405fdc 1,102 141,056 System.Data.SqlClient._SqlMetaData
0x7a766474 8,912 142,592 System.ComponentModel.CollectionChangeEventArgs
0x79111038 2,748 153,888 System.Reflection.RuntimePropertyInfo
0x7910efbc 13,608 163,296 System.Runtime.Remoting.Messaging.CallContextSecurityData
0x663af054 2,063 165,040 System.Web.HttpWriter
0x653fe4d4 2,080 208,000 System.Data.SqlClient.SqlParameter
0x663b08c8 4,126 231,056 System.Web.HttpCookieCollection
0x02525e84 2,073 240,468 ASP._global_asax
0x79109778 4,395 246,120 System.Reflection.RuntimeMethodInfo
0x663af964 2,072 248,640 System.Web.SessionState.SessionStateModule
0x79102290 25,426 305,112 System.Int32
0x663aa0a0 2,063 354,836 System.Web.HttpRequest
0x663a9dac 2,063 387,844 System.Web.HttpContext
0x663a4adc 2,063 404,348 System.Web.HttpResponse
0x79108ce0 11,536 415,296 System.Collections.Hashtable+HashtableEnumerator
0x7911a2d0 13,610 489,960 System.Runtime.Remoting.Messaging.LogicalCallContext
0x79104de8 13,616 490,176 System.Threading.ExecutionContext
0x663b205c 31,080 497,280 System.Web.HttpApplication+SyncEventExecutionStep
0x663ad7c0 2,063 503,372 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6
0x7912d7c0 7,403 522,620 System.Int32[]
0x7a7566e8 26,959 539,180 System.ComponentModel.EventHandlerList+ListEntry
0x65412bb4 514 539,672 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
0x7a75a878 35,598 569,568 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
0x79104368 25,337 608,088 System.Collections.ArrayList
0x0252788c 2,027 770,260 ASP.default_aspx
0x654088b4 5,281 781,588 System.Data.DataColumn
0x79108afc 11,541 877,116 System.Threading.ExecutionContext+ExecutionContextRunData
0x654359c8 1,038 1,125,544 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
0x66437650 2,073 1,252,092 System.Collections.Generic.Dictionary`2+Entry
0x791044dc 53,803 1,721,696 System.EventHandler
0x79101fe4 31,106 1,741,936 System.Collections.Hashtable
0x000e2dc0 3,644 3,621,108 Free
0x7912d9bc 31,396 4,984,728 System.Collections.Hashtable+bucket[]
0x790fd8c4 95,552 9,502,548 System.String
0x7912dd40 8,376 12,904,904 System.Char[]
0x7912d8f8 53,376 336,430,944 System.Object[]
0x79104c38 27,851,257 445,620,112 System.WeakReference
Total 28,603,114 objects, Total size: 836,223,424
NOTE: For more information on the !dumpheap debugger command, have a look at the following blog post:
https://blogs.msdn.com/tess/archive/2005/11/25/496973.aspx
The next step was to dump out some of the System.WeakReference instances to try to dig in to why they’re there:
0:000> !dumpheap -mt 0x79104c38
Using our cache to search the heap.
Address MT Size Gen
0x025d7ab8 0x79104c38 16 2 System.WeakReference
0x025dc2d0 0x79104c38 16 2 System.WeakReference
0x025dc540 0x79104c38 16 2 System.WeakReference
0x025dd330 0x79104c38 16 2 System.WeakReference
0x025dd8d8 0x79104c38 16 2 System.WeakReference
0x025def80 0x79104c38 16 2 System.WeakReference
0x025def90 0x79104c38 16 2 System.WeakReference
0x025defa0 0x79104c38 16 2 System.WeakReference
0x025defb0 0x79104c38 16 2 System.WeakReference
0x025defc0 0x79104c38 16 2 System.WeakReference
0x025defd0 0x79104c38 16 2 System.WeakReference
0x025defe0 0x79104c38 16 2 System.WeakReference
0:000> !do 0x025defa0
Name: System.WeakReference
MethodTable: 79104c38
EEClass: 79104bd4
Size: 16(0x10) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
MT Field Offset Type VT Attr Value Name
791016bc 40005a9 4 System.IntPtr 1 instance 33389776 m_handle
7910be50 40005aa 8 System.Boolean 1 instance 0 m_IsLongReference
All of the objects we dumped out looked the same, and none of them appeared to be rooted. Looking around for other signs of problems, we found an components built in debug mode:
0:000> !sos.FindDebugModules
Loading all modules.
Searching for modules built in debug mode...
custom_component.DLL not built release
custom_component2.DLL not built release
custom_component3.DLL not built release
Having modules built in debug mode running on a Production server is never a good idea. And as it turned out in this case, the debug mode modules combined with the fact that these modules implement the __ENCLIST helper class for Visual Studio’s Edit and Continue feature.
When a component is built in debug mode, the Edit and Continue debugging feature of Visual Studio is enabled. The Edit and Continue debugging feature in Visual Studio 2005 and 2008 maintains a list of weak references to objects that have been created. If the object that has been created is a class that contains an event, these weak references are maintained for the duration of the program. This behavior increases memory usage. To run into the memory usage problem, the component must be built in DEBUG mode, and in must use the __ENCLIST help class that allows edit and continue.
In this scenario, the problem got resolved by rebuilding those components in Release mode instead of Debug mode.
Comments
Anonymous
March 12, 2009
.NET NP .NET Profiler – a tool is designed to assist in troubleshooting issues such as slow performanceAnonymous
March 12, 2009
.NETNP.NETProfiler–atoolisdesignedtoassistintroubleshootingissuessuchasslowperf...Anonymous
March 24, 2009
In VS2008: When we deploy our web application with right click: "Publish..." Why does VS not automatically build a Release and deploy it? If there is a reason for not building Release and deploying it, it would be nice to have a menu point: "Publish release build..." Thx Peter