Out of memory issue when Silverlight Application talks to WCF service using System.Net.Socket via Duplex Channel

We worked on an interesting issue where WCF was hosted as windows service. Data was Pushed from WCF Server to Silverlight client via Sockets every 2 sec. Processing WCF-pushed data is the bulk of the task that this Silverlight Application does. The app just sits idle most of the time and processes pushed-data from WCF. App exhibits increased memory consumptions even if there is no user interaction. It starts at 200MB and reaches to 1.5GB in 10 hours. Rather than letting the good research go waste, we are publishing the details here.

First step was to profile the app to see what kind of memory growth we are seeing. We used a 3rd profiler and it was showing memory issues at the .NET layer. To be sure, we collected memory dump of the Silverlight application using Debug Diagnostics Tool. Sure enough it showed lots of Managed (.NET) Memory accumulation, as well as, Native ones. Typically managed layers calls into native layer when using communication APIs, so an increased in Managed memory also contributes to an increase in Native memory. So, for now, we are concentrating on Managed memory accumulation being the culprit here.

 

0:006> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal

<unknown> 500 37b662d8 ( 891.399 Mb) 55.78% 43.53%  <===== Managed memory

Heap 928 1cde6000 ( 461.898 Mb) 28.91% 22.55%  <===== Native memory

Free 518 1c1fa000 ( 449.977 Mb) 21.97%

Image 1017 9ef3d28 ( 158.952 Mb) 9.95% 7.76%

Stack 257 5400000 ( 84.000 Mb) 5.26% 4.10%

Other 36 163000 ( 1.387 Mb) 0.09% 0.07%

TEB 82 52000 ( 328.000 kb) 0.02% 0.02%

PEB 1 1000 ( 4.000 kb) 0.00% 0.00%

 

After poking around the dump, we found that sockets were not closed properly. So we coded DuplexClientBase.CloseAsync() and this resolved the Managed memory leak.

But we were not out of the wood yet!

If the application runs for a while, we see that we no longer have Managed memory accumulation, but the Native memory continues to rise leading to Out of Memory error. It is taking double the amount of time to get an OOM, but it gets it eventually. Hmmm...we got a Native leak to debug!

Collected another set of dumps (since we made code changes, earlier dumps no longer relevant). After poking around the dump, we found that, for each and every failed connection when the WCF server is down, a new set of proxy object is created and is not getting deleted. As the proxy object is alive, the corresponding internal objects like sockets, channels are getting leaked. As for the remedy, we need to make sure proxy objects get deleted properly. WCF channels are not getting closed at the client/Silverlight side and hence the new proxy objects are still alive. To resolve this memory leak we need to close the WCF channel before we create new WCF proxy objects. This warranted a code fix.

Below is the code snippet in C# for closing duplex WCF channels.

 ICommunicationObject o = (ICommunicationObject)c; 
 o.Closed += new EventHandler(o_Closed); 
 o.BeginClose(null, null); 
 

 And it worked! No more leaky proxies or sockets or channels.

 

The morale of the story: Close the WCF channel before creating new WCF proxy objects

 

Author: Enamul[MSFT] & Prashant[MSFT]. Rich Client Patterns & Practices.