.NET Finalizer Memory Leak: Debugging with sos.dll in Visual Studio
Normally I write about issues that only manifest themselves in production environment, issues that you can't really reproduce in a controlled dev environment every time you perform a certain action. In those cases you need to use tools like windbg to gather dumps and do post-mortem debugging.
Windbg works really well for those types of issues, but it has its shortcommings since it is not really a managed debugger so it is much harder to set breakpoints in .NET code or step through code, or even inspect objects in a visual way like you can in a managed debugger like Visual Studio.
Visual Studio on the other hand doesn't allow you to do post-mortem debugging the same way windbg does, and there is no easy way to view information about the domains loaded in the process or to view information about the objects on the .net heaps. (For some links on post-mortem debugging with Visual Studio, see Peters comment to this post)
My colleague had an issue that was pretty easily reproducible but he needed both worlds, i.e. stepping through code to a specific point, looking at the objects on the stack visually and at the same time he needed to view the contents on the heap, so he resorted to having two debuggers attached, visual studio debugging managed, and windbg debugging native with sos to view the managed heap. That is pretty nasty, and there is a much easier way to combine these two worlds... debugging with sos in Visual Studio.
To illustrate how it works i'm using a sample from Ingo Rammer.
If you want to follow along you can download the sample code here under the link Hardcore Production Debugging. The specific sample I am using is FinalizerProblem which is basically the winforms equivalent of my post on Unblock my Finalizer.
Problem description:
When I click on the button "Do Work" it creates a number of instances of the class MyBusinessObject. Even though I know that the objects should no longer be referenced after that they don't appear to go away even if I invoke a GC.Collect() with GC.WaitForPendingFinalizers(). Why are my objects not released?
Debugging the issue:
In this case we could easily attach windbg and use sos.dll per my post above to figure out that the reason these objects are sticking around is because of a blocked finalizer, but I will use Visual Studio in order to show you how to load up sos in it and run sos commands.
Step 1: Enable Native Debugging for the project
In order to load an extension like SOS.dll you have to be debugging in native mode, so before starting the debugger go into Project/Properties/Debug on the context menu for the project and check the box for Enable unmanaged code debugging.
Step 2: Debug and Break
Debug the problem as you normally would in Visual Studio until you have reproduced the issue (i.e. in this case click on "Do Work" to instanciate the objects, followed by "Run GC" to perform the garbage collection.
Break into the process (Debug menu/Break All)
Step 3: Load sos
In order to load sos.dll you have to open up the Immediate Window (Debug/Windows/Immediate or Ctrl+D, I) and type
.load C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll
This should yield the response
extension C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll loaded
Step 4: Debug with sos
Now we are ready to debug with sos.dll.
There are a few rules here...
1. You can not run native commands like kb, dc etc.
2. You can not run ~* e which means you cant run ~* e !clrstack to see the stacks on all threads
You can however run all the commands in sos.dll like !dumpdomain, !dumpheap etc.
To switch the thread context for thread specific commands like !clrstack and !dumpstackobjects you can open the threads window (debug/windows/threads) and doubleclick the thread you want to switch to. I will show an example of that later...
The first thing we want to do is find our objects on the heap
!dumpheap -type MyBusinessObject
PDB symbol for mscorwks.dll not loaded
Address MT Size
027437e4 01d6683c 12
02743830 01d6683c 12
0274387c 01d6683c 12
...
02747d6c 01d6683c 12
02747db8 01d6683c 12
02747e04 01d6683c 12
02747e50 01d6683c 12
02747e9c 01d6683c 12
02747ee8 01d6683c 12
total 30 objects
Statistics:
MT Count TotalSize Class Name
01d6683c 30 360 FinalizerProblem.MyBusinessObject
Total 30 objects
Then we can grab one of those objects and run !gcroot on it to find out why it is still around
!gcroot 02747d6c
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Error during command: warning! Extension is using a feature which Visual Studio does not implement.
Scan Thread 7092 OSTHread 1bb4
Scan Thread 6864 OSTHread 1ad0
Finalizer queue:Root:02747d6c(FinalizerProblem.MyBusinessObject)
In this case it is rooted in the finalizer queue which means it is waiting to be finalized and if we look at the finalizequeue we can see that we have 69 objects that are waiting to be finalized so the question that remains is why we aren't finalizing them...
!finalizequeue
SyncBlocks to be cleaned up: 0
MTA Interfaces to be released: 0
STA Interfaces to be released: 0
----------------------------------
generation 0 has 0 finalizable objects (002906d0->002906d0)
generation 1 has 36 finalizable objects (00290640->002906d0)
generation 2 has 0 finalizable objects (00290640->00290640)
Ready for finalization 69 objects (002906d0->002907e4)
Statistics:
MT Count TotalSize Class Name
7b47f8f8 1 20 System.Windows.Forms.ApplicationContext
...
7910b694 10 160 System.WeakReference
7b47ff4c 4 224 System.Windows.Forms.Control+ControlNativeWindow
01d6683c 22 264 FinalizerProblem.MyBusinessObject
01d65a54 1 332 FinalizerProblem.Form1
7b4827e8 2 336 System.Windows.Forms.Button
7ae78e7c 8 352 System.Drawing.BufferedGraphics
...
Total 105 objects
If we were debugging in windbg, the next natural step would have been to run !threads, figure out which one was the finalizer and look at what is is running.
Since we are in Visual Studio instead we can open the threads window, navigate to the thread that is performing Finalization and look at what it is doing. If you can't determine that by the function that is currently on the top of the user-code part of the callstack you can still identify it with !threads.
The cool thing here is that we jump straight into the code, where it is blocking and can use everything we are used to in Visual Studio like the watch and locals window or stepping in code for example, but without loading up sos.dll we would not have been able to track down why our MyBusinessObject instances were still around.
Final words:
I know many people are a bit adversed to starting with windbg since it means you need to learn a whole new toolset and perfectly honestly windbg is not as "beginner friendly" as for example visual studio is, then again, its meant to be used mostly for post-mortem debugging which is not exactly an everyday task for most people.
Hopefully though with the info above you can still begin to use sos.dll in the cozy and familiar visual studio debugging environment.
Speaking of making things a bit more user-friendly and visual, Ingo Rammer who wrote the demo has developed a tool called SOSAssist which is kind of a GUI interface to sos.dll.
Until next time,
Tess
Comments
Anonymous
October 19, 2007
Maybe I misundestood the context when you said "Visual Studio on the other hand doesn't allow you to do post-mortem debugging". The following are a sample couple of links that describe post-mortem (crash dump) debugging with Visual Studio: http://msdn2.microsoft.com/en-us/library/d5zhxt22.aspx http://www.codeproject.com/debug/postmortemdebug_standalone1.aspAnonymous
October 19, 2007
The comment has been removedAnonymous
October 19, 2007
NoVa Code Camp 2007.02 [Via: Jeff Schoolcraft ] EarthClassMail.com - Moving from LAMP to .NET 3.5 [Via:...Anonymous
October 21, 2007
Thanks mcgurk, That would have to be the weirdest compliment I've ever gotten:)Anonymous
October 21, 2007
Hi Peter, No you didnt missunderstand, I just phrased it wrong... You can do limited dump analysis in visual studio, and to be honest it works pretty good, but given how hard it is to get to the native parts and run commands like ~* etc. it's almost a can't. I'll edit the post to reflect this... thanks for the comment...Anonymous
October 26, 2007
SOSAssist saved my life. Thanks for sharing it!Anonymous
November 02, 2007
The comment has been removedAnonymous
November 20, 2008
I still have to figure out a way to host the video from the break-out session but until then, here areAnonymous
November 29, 2010
This article saved my life (job)! Thank you.Anonymous
July 10, 2012
The comment has been removed