Udostępnij za pośrednictwem

ASP.NET Memory Leak: Byte arrays rooted in System.Security.Policy.Evidence

  • Artykuł

Today I got a question from a reader (Chris) about a memory leak they are seeing in their application

When we do a '!dumpheap -min 85000 -type Byte[]' we can see 100s of byte array objects using up ~545MB of memory. A majority of the objects are the same size (either 4.5 or 9MB in size). Looking at the memory addresses, they all appear to be different copies of our assemblies. And when we do !GcRoot on those addresses they all either have no results returned, or show a rooted System.Security.Policy.Evidence object:

DOMAIN(000FDC18):HANDLE(Strong):-:Root:-(System.Security.Policy.Evidence)->
(System.Collections.ArrayList+SyncArrayList)->
(System.Collections.ArrayList)->
(System.Object[])->
(System.Security.Policy.Hash)->
(System.Byte[])

Through PerfMon we can see our Large Object Heap size grow and grow, and through !dumpheap it looks like it is getting fragmented.

We have not figured out how/why so many copies of our main assembly’s would be loaded by different rooted System.Security.Policy.Evidence object over and over again and never released. Our code never creates and new AppDomains and never manually loads any assembly. It receives a lot of web service requests, makes some database queries and/or makes its own web service requests and then returns the results.

Chris did a lot of ground work here and did everything exactly right

1. Looked at the .net memory (!eeheap -gc) to find out that that is where his memory issue is, since most of the memory is managed (.net)

2. Looked at the large objects (!dumpheap -min 85000) and identified that the byte[]s on the LOH were taking up most of the .net memory

3. Compared the byte[]s to find a pattern (they were all 4,5 MB byte arrays and 9 MB byte arrays)

4. Looked at the contents of the byte arrays (dc <addr of byte array>) and noticed that the byte arrays contained text that resembled some of his assemblies, and not only that, he noticed that they were the same assemblies so it looks like the same data is loaded over and over

5. Looked at the roots for the byte[]s and found that they were rooted in System.Security.Policy.Evidence

From here it gets a bit tricky to figure out the cause of the issue.  But it turns out that when you load an assembly that reference another assembly, a security.policy.hash is generated.  The hash it generates is about the same size as the referenced dlls, so that is where the 4.5 and 9 MB come from.  This is then cached in case another dll is referencing the same dll so the hash wont be regenerated.

That is all well and good, the question is why are we then generating so many of these policy hashes?

There was an issue in .net 2.0 RTM where, if memory pressure is high (i.e. if system memory is low) we would flush some items from cache which meant that we need to regenerate these policy.hashes.  Of course this in turn may cause more memory pressure so we flush again and create again etc. etc.

Solution

This issue is fixed in https://support.microsoft.com/default.aspx/kb/929963 FIX: The ASP.NET cache may flush some assemblies if the system memory is low, which is included in SP1 for .Net framework 2.0, so preferably one should install SP1 to fix this issue.

Have a good one,

Tess

Comments

  • Anonymous
    June 30, 2008
    Tess, Thanks for post from my email but our solution ended up being a little different. First, we were already running SP1. And second, Our 4.5MB assembly was hosting a lot of our code, and all of our [WebMethod]s. We did a little Reflecting and found out that the "Executing Assembly" will be copied into memory by the Evidence class as part of CAS on every web service request, and remain in memory for the length of the request. We moved our WebMethods out to a new assembly and changed our .asmx files to point to this new assembly. Our new WebMethods only assembly is only 300K and every method just calls the same old code in the big assembly and everything is working normally now. Hope this makes sense and is a valid solution. Thanks for the help, Chris

  • Anonymous
    August 01, 2008
    I've found that with sp1 (on our local machines and our servers)  whenever a web service request is made,  the application will store the same assembly 4 times in the manged heap as byte arrays.  This is very confusing.   To rule out any faults in the application, I created a new web app.  Added a single web service.  Used the "hello world" example that gets generated automatically when creating a new web service.  To make it easier for me to differentiate the byte arrays, I changed the "hello world" string to a string builder containing a long lorem ipsum.  I then just returned the toString() of the string builder.  Built the application, ran it.  Did an dump right before I ran the webservice(just used GET) and another right after.  WinDbg showed 4 identical byte arrays containing the webservice assembly.  I guess my question is, have you or anyone else seen this behavior before?  Any idea what could be causing it.   At first I thought it was an awfully big coincidence that there are Soap, Soap 1.2, Get, and Post(4 total).  After shutting off Soap and Soap 1.2 in the web config, it still stored 4 assemblies.  I'm out of ideas.  This has caused me many hours of headache on our production boxes as our web service assembly was well over 300kb.  Our application hosts many templated sites which all will cause 4 of these assemblies to be build.  So, 300kb x 4 x 1000 sites = out of memory. I have modified the structure and split out some of the related web services into separate assemblies.  This has dropped most of them down to below the LOH threshold.  The application is constantly under development and I could see a developer adding more code to some of those assemblies potentially bringing them back into the LOH.  I guess I'm just venting, but I would appreciate it if some of you could try to reproduce this.  Maybe it's just all our boxes, maybe it's a misconfiguration, maybe it's a framework issue that could just as easily be fixed. Anyways, thanks for your help.  John

  • Anonymous
    August 04, 2008
    interesting... I dot recognize the problem though, sorry

  • Anonymous
    February 10, 2009
    Hi John, hi Tess, I have the same issue here that I can see the content of the DLLs in bin in memory. Maybe Tess has some time to have a look on this problem. Regards, Michael

  • Anonymous
    March 13, 2009
    Are you trying to improve the memory usage of your .net application? I’ve spent some time recently trying

  • Anonymous
    March 17, 2011
    Apparently there is an additional issue involved here with duplicating byte[]s that is resolved in 4.0, for 3.5 the best way to work around it is to my knowledge to reduce the size of the dlls containing the web methods.