MSCOM OPS March Debug Madness…3rd Session Q & A Dignosing Memory Leaks in ASP.NET Applications
Wednesday’s topic was “Diagnosing Memory Leaks in ASP.NET Applications” presented by Jim Dobbin, a Sr. Systems Engineer on the MSCOM OPS Debugging Team. As this week progressed we continued to be impressed by the level of the questions asked. Here are your questions and Jim’s answers.
1. Asked: Is this built into Visual Studio or where do you enter these commands?
Answered: This is debugging with the Debugging tools for Windows and the SOS extension for managed code. Completely outside of the Visual studio environment.
2. Asked: Is debugdiag not a useful tool in this situation? Or in other words when do you use windbg and not debugdiag?
Answered: Debugdiag is a more automated tool that does some general analysis, which can be very useful. We prefer to use windbg/cdb as we have some debugger extensions that enable us to more quickly isolate the problem. In some cases, we will use automation scripts with windbg/cdb to track down problems somewhat like debugdiag would. I would say that debugdiag knows quite a bit about IIS internals, but not as much about asp.net/clr internals.
3. Asked: Is there a debugger quick reference guide for the commands?
Answered: There is always the debugger help docs and within SOS you can use the !help command to get the reference guide for that extension.There is probably a quick reference guide out there somewhere on the internet, but I'm not aware of one we have. We will be posting all our debug logs to our blog after the week is over. We'll try to throw together a quick reference with that post.
4. Asked: I got here late. Is there a url to listen to this in entirety at a later time?
Answered: Yes, to view this webcast again you can go to www.microsoft.com/webcasts this will be available to view or download in 24 hours
5. Asked: What search terms are useful when searching for Microsoft KB articles on Microsoft applications like CRM that have memory leak problems? I have been getting out of non-paged pool problems on my system since installing CRM but I cannot find and reports of the problem searching the support articles?
Answered: usually the application name and "memory leak" is sufficient. In the case of non-paged pool, CRM may be consuming user mode objects that have kernel mode resources, such as handles. Poolmon.exe would be a good tool to help track down this leak. Just do a search on support.microsoft.com for poolmon and you should find the appropriate documentation. https://support.microsoft.com/kb/177415 How to Use Memory Pool Monitor (Poolmon.exe) to Troubleshoot Kernel Mode Memory Leaks
6. Asked: What is the name of the app he is using?
Answered: The windows debugger tools, specifically windbg from that package.
7. Asked: How do you know what the source code .aspx file is?
Answered: Typically, !gcroot or !clrstack will call out the name of the aspx filename if it's involved in the leak. It will be something like ASP.filename_aspx. If all you have is a reference to the assembly like Web_App_xxxxx.dll, you can do a findstr on the temporary asp.net assemblies directory under c:\windows\microsoft.net\framework\v2.05727\, searching for the assembly name in all *.compiled files. This will give you the aspx page that was compiled into that assembly.
8. Asked: Any suggestions on debugging a live site? In some cases, it may not be possible to take a machine out of rotation. In other words, is it possible to get a "dump" I could copy down to a dev box and examine?
Answered: You could use the debuggers and ADPlus or other tools like Debugdiag to get a dump for offline examination. But this is usually still intrusive while the debuggers runs and saves the dump. For memory investigations like this, we often use this method and investigate offline.
9. Asked: I own John Robbins book on Debugging .NET. All there other good references about debugging? I'm able do to debug fairly well using Visual Studio IDE. Any material for novices using WinDbg in managed code?
Answered: Take a look Production Debugging .Net Applications: https://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/DBGrm.asp
It has some great debugging walkthroughs. We are going to try and post a quick reference for sos as well as some other commands to our blog after the week is over.
10. Asked: I am looking for the debugger. Where can I find it?
Answered: https://www.microsoft.com/whdc/devtools/debugging/default.mspx
11. Asked: Hi there... what perfmon counters can I add to diagnose an asp.net app memory leak?
Answered: I would go ahead and view the webcast from Monday: Microsoft.com Operations Introduces Real World Debugging: Determining When You Have a Problem and Beginning the Initial Debugging (Level 300). Jeff Johnson did a great job on covering that. Basically, you'll want to pay attention to the .Net CLR Memory counters, specifically Gen0, 1, & 2 heap size counters, as well as Large Object Heap size.
12. Asked: GREAT Presentation... It's nice to gain some more insight into debugging tools and practices. Do you have, or will a upcoming webcast cover debugging JavaScript in IE? Your team BLOG address?
Answered: Go ahead and post to our blog and we'll see what info we can provide - https://blogs.technet.com/mscom/default.aspx
13. Asked: I used to program in assembly back in the old days of the Z80, but since then I have not seen anything in x86. Any resources for a novice x86 assembly reader?
Answered: All kinds of resources for x86 and x64 for that matter. A great book is The Art of Assembly Language by Randy Hyde. You can also get the reference books from Intel and AMD through print or online. There's a minimal charge for those reference books.
14. Asked: Do identical applications recompiled for 64Bit tend to use more or less than a 32 bit compile?
Answered: It really comes down to how many pointers the application uses, such as whether the application uses a lot of string objects or custom objects. Something that's really intensive with string objects could potentially have > 30% size overhead, but typically it's not that big of deal.
15. Asked: What is the recommended "one-stop-shop" for debugging information? There is so much out there and hard to mine all this information out. There also seems to be a lack of continuity at MS with tools. For example there was no mention of the psscor extension and just a brief reference to DebugDiag. Many tools to do the same thing I guess but for us in the real world, it's hard to know what the best tools are.
Answered: Unfortunately, no one stop shop that I am aware of. MSDN has good number of articles on debugging and blogs.msdn.com has a lot of debugging info being posted. Regarding PSSCOR, we used to use that extensively when we were running on Framework 1.1 (Everett) and there is no PSSCOR for 2.0 (Whidbey) yet, although the SOS extension in 2.0 does incorporate many of its commands.
16. Asked: Can you give a quick walk through on how to make a dump I can copy off the server...and then how I could use the win dugger to query it?
Answered: The quick run through – Attach a debugger to the process, you should get an initial break in. run the “.dump /ma c:\mydumpfile.dmp “command , then “.detach” when the dump command says it successfully has written the file. This is still intrusive debugging until you .detach. Copy the file and open it in WinDBG with the File menu Open Crash Dump. You could also do this with ADPlus that comes with the debugger package.Check out the debugging tools - https://www.microsoft.com/whdc/devtools/debugging/default.mspx and check out the debugger help – Debugger Operations section, Crash Dump files section and the extra tools section for ADPlus info.
17. Asked: Were can I find tip like the ones that were told here "EAX generally contains the return of a method" or "CMP [ECX], ECX" is a check for a AV?
Answered: These really depend on the type of calling conventions that are being used. These can be found through x86/x64 assembly reference books. Some of this is just through experience.
18. Asked: What is the best tool for debugging multithreaded app?
Answered: It really depends on what you're trying to troubleshoot. Certainly windbg/cdb handles multithreaded debugging with ease.
19. Asked: I thought in ASP.NET memory leaks were a thing of the past?
Answered: For the most part they are , it’s not easy to leak memory in ASP.Net. I had to work at it for my demos.Objects/Memory that fall out of scope will get cleaned up if there are no further references left to them. So a leak like condition can still occur if objects still contain references to them. The methods demonstrated with SOS are how you find the references that still exist should run into this.
20. Asked: Is it possible to fix the problems, if possible, at the debugging stage?
Answered: It’s not easily done within the debugger. It’s great for finding bugs but not fixing them. Although you can modify memory within the debugger,it’s not typically recommended as modifying memory outside of the application's normal program flow can cause unknown behavior. That being said, we have on occasion used the debugger to change certain variable to modify the application during runtime.
21. Asked: Are you able to do everything that WinDbg does in Visual Studio?
Answered: Generally yes, and I think more than what VS can do with the different extensions available. Windbg has a richer command line experience for using debugger extensions, etc.
22. Comment: Great series. Keep on with the good work!
23. Asked: I've used CLR Profiler in order to see how the objects are allocated and the gens disposed, do you recommend it?
Answered: We do recommend the CLR profiler but find it is generally too much of a perf hit to attempt it in production. It is great for pre-production and test.
24. Asked: Any suggestions for novices (with no knowledge/experience of assembly lang.), as to where to start?
Answered: A great book is The Art of Assembly Language by Randy Hyde. You can also get the reference books from Intel and AMD through print or online.
25. Asked: Right now the federaldeveloper folks are giving a webcast on VSTE about automating testing with the software testing edition of VSTE. Do you think it's worth trying to figure out how to automate memory leaks with VSTE? Seems to me most memory leaks get figured out by a single tenacious programmer who cares and tracks them down manually? GREAT presentation - thanks again
Answered: I think that’s something the VS team is always looking at improving. Memory leaks are still pretty rare in managed code. The difficulty I see is the how to get you the coverage for all scenarios. I went over just a few that I have run into in the past today and I am sure there are more I haven't seen yet.
26. Comment: Again, many thanks. This is one of the most useful series I've attended.