Tools for Your Debugging Toolbox
This article was just updated to include an internal Microsoft tool that is now public.
There are many free tools used to troubleshoot and debug software. Below I present a list of the tools that my peers and I use most of the time. Though most of the tools below are free Microsoft tools, not all are very well known.
If you’re a Microsoft Premier customer and think it’s a good idea for you and your team to learn from a Developer PFE about when and how these tools should be used (with demos), contact your TAM (Technical Account Manager) and tell him/her you’re interested on the PFE Developer Toolbox Chalk Talk. This Chalk Talk can be delivered on site or remotely and takes about 6 hours.
Also, if you want to see some interesting videos of a Developer PFE debugging an application take a look at these videos.
Note: This article includes most tools from this blog post plus some more.
TOOLS
- Performance Monitor - PAL
- Process Monitor
- Process Explorer
- MPSReport
- SPSReport
- SPDisposeCheck
- Dependency Walker
- SQL Nexus
- LogParser
- Indihiang
- PowerShell
- Application Verifier
- Logger/LogViewer
ADVANCED TOOLS
- XPerf – Windows Performance Analyzer
- PerfView – Low Level Profiler for .NET applications
- DebugDiag – Debug Diagnostic
- ProcDump – Process Dump
- WinDbg – Windows Debugging Tools
- WinDbg Scripts – Automate the Debugging
- Netmon – Microsoft Network Monitor
- Fiddler – HTTP Debugger Proxy
- NP .NET Profiler - Lightweight profiler designed to assist in troubleshooting issues such as slow performance, memory related issues, and first chance exceptions in .NET applications
BRIEF SUMMARY OF EACH TOOL
Performance Monitor
- Use to get information about the application’s health.
- Use to see if and when the suspicious symptom happens.
- Save a log file. It can be analyzed later.
- Part of the Windows Operating System.
PAL (Performance Analysis of Logs)
- The PAL tool reads in a Performance Monitor counter log (any known format) and analyzes it using complex, but known thresholds (provided).
- The tool generates an HTML based report which graphically charts important performance counters and throws alerts when thresholds are exceeded.
- The thresholds are originally based on thresholds defined by the Microsoft product teams and members of Microsoft support, but continue to be expanded by this ongoing project.
Download:
Process Monitor
- Capture of thread stacks for each operation make it possible in many cases to identify the root cause of an operation
- Reliable capture of process details, including image path, command line, user and session ID
- Filters can be set for any data field, including fields not configured as columns
- Advanced logging architecture scales to tens of millions of captured events and gigabytes of log data
- Process tree tool shows relationship of all processes referenced in a trace
- Boot time logging of all operations
Download:
https://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
Process Explorer
- Easy way to see information from processes.
- What is each thread doing? Call stack is available.
- How is the CPU usage? You can see the CPU usage, Kernel and User Mode.
- You can see which program has a particular file or directory opened.
- You can search for a specific handle or DLL among the processes running.
Download:
https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
MPSReport
- Check DLL’s versions, hotfixes, software updates.
- Compare if two machines have the same drivers, registry settings and softwares.
Download:
https://support.microsoft.com/kb/823393/
SPSReport
- Think of MPSReport for SharePoint.
- The SPS Reporting Tool is utilized to gather detailed information regarding a systems current configuration.
Download:
https://www.codeplex.com/spsreport
SPDisposeCheck
- The SPDisposeCheck utility will assist you dig through your custom SharePoint MSIL assemblies looking for areas in your code that may require “closer examination” and might lead to Dispose() related memory leaks.
- A manual code review is still required to cast out ‘false positives’ that the tool may produce in the output report
Download:
Best Practices:
https://msdn.microsoft.com/en-us/library/aa973248.aspx
Dependency Walker
- Scans any 32-bit or 64-bit Windows module (exe, dll, ocx, sys, etc.) and builds a hierarchical tree diagram of all dependent modules
- For each module found, it lists all the functions that are exported by that module.
- Detailed information about each file including a full path to the file, base address, version numbers, machine type, debug information, and more.
- Useful for troubleshooting system errors related to loading and executing modules.
Download:
https://www.dependencywalker.com/
SQL Nexus
- You don’t need to be a DBA to use this tool. J
- You can quickly and easily load SQL Trace files; T-SQL script output, including SQL DMV queries; and Performance Monitor logs into a SQL Server database for analysis.
- Excellent tool for isolating problems on the SQL Server side.
Download:
https://www.codeplex.com/sqlnexus
Screenshots:
https://sqlnexus.codeplex.com/wikipage?title=SqlNexusReports&referringTitle=Home
CLRProfiler
- Focused on managed heap
- Who allocates what
- What objects survive
- What is on the heap
- Who is holding on to objects
- Instrumented application writes log
- Separate tool to analyze log offline
- Intrusive tool
- By default, every allocation, every call is logged
- Expect 10 – 100 x slowdown
- Logging can be turned off selectively for speedup
- Not a tool to measure where time is spent
Download:
How To: Use CLR Profiler
https://msdn.microsoft.com/en-us/library/ms979205.aspx
LogParser
- Log parser is a powerful, versatile tool that provides universal query access to text-based data such as log files, XML files and CSV files, as well as key data sources on the Windows® operating system such as the Event Log, the Registry, the file system, and Active Directory
Download:
Forum:
https://forums.iis.net/default.aspx?GroupID=51
Forensic Log parsing:
https://www.securityfocus.com/infocus/1712
Log Parser scripts:
https://blogs.iis.net/rakkimk/archive/2008/11/18/logparser-useful-logparser-scripts.aspx
https://blogs.msdn.com/carloc/archive/2008/02/06/logparser-scripts-for-various-occasions.aspx
https://blogs.technet.com/clinth/archive/2010/01/07/w3c-iis-log-analysis-using-log-parser.aspx
Visual LogParser tools:
https://www.codeplex.com/visuallogparser
Indihiang
- Based on LogParser
- Great tool to analyze IIS logs
- Generate great charts
- User interface easy to interact
- Downside: slower than running regular LogParser scripts
Download:
https://indihiang.codeplex.com/
PowerShell
- Why do you need to learn another scripting language?
- Less code than JScript and VBScript to accomplish the same task.
- Total integration with .NET Framework.
- Great for administrators and developers.
- In PFE we use PowerShell as the preferred programming language used to create tools!
- Alternative to LogParser tool.
Download:
PowerShell is part of Windows 7 and newer versions.
For other Windows versions you can download from:
PowerShell blog:
https://blogs.msdn.com/powershell/
Application Verifier (for Native coded applications)
- When the application is using APIs correctly:
- Unsafe TerminateThread APIs.
- Correct use of Thread Local Storage (TLS) APIs.
- Correct use of virtual space manipulations (for example, VirtualAlloc, MapViewOfFile).
- Whether the application is hiding access violations using structured exception handling.
- Whether the application is attempting to use invalid handles.
- Whether there are memory corruptions or issues in the heap.
- Whether the application runs out of memory under low resources.
- Whether the correct usage of critical sections is occurring.
- Whether an application running in an administrative environment will run well in an environment with less privilege.
- Whether there are potential problems when the application is running as a limited user.
- Whether there are uninitialized variables in future function calls in a thread's context.
Download:
References:
https://msdn.microsoft.com/en-us/library/aa480483.aspx
https://technet.microsoft.com/en-us/library/bb457063.aspx
https://blogs.technet.com/askperf/archive/2009/05/22/two-minute-drill-application-verifier.aspx
Logger/LogViewer
- Logger.exe logs every API call done by the target application.
- LogViewer.exe displays the API calls logged by Logger.exe
- With LogViewer.exe you can specific APIs that were called, filtering the output.
Download:
Logger/LogViewer are part of the Debugging Tools For Windows:
https://www.microsoft.com/whdc/devtools/debugging/default.mspx
Command reference for Logexts.dll extension:
https://msdn.microsoft.com/en-us/library/ff552064(v=VS.85).aspx
Calling Logexts.dll from WinDbg:
https://blogs.msdn.com/debuggingtoolbox/archive/2007/04/14/windbg-script-tracing-api-calls.aspx
ADVANCED TOOLS
XPerf
- A very efficient tracing infrastructure provided by Windows
- Enables high volume of tracing with minimal performance degradation
- Can be used in User Mode and Kernel Mode
- Provides many different graphical views of trace data including:
- CPU Sampling
- CPU and Disk utilization by process and thread
- Interrupt service routine and deferred procedure call using
- Hard faults
- Disk I/O Detail
- Call stacks
Download:
PerfView
- Tool for quickly and easily collecting and viewing time and memory performance data.
- Like XPERF, it is based on ETW (Event Tracing for Windows)
- 2 modes of execution:
- Optimizing Time.
- Optimizing Memory.
- Access to call stacks and Garbage Collector information.
- Low Level compared to other Profilers.
Download:
http://www.microsoft.com/en-us/download/details.aspx?id=28567
Tutorial:
https://channel9.msdn.com/Series/PerfView-Tutorial
DebugDiag
- You can create rules in order to collect dumps under specific conditions.
- Crash rule – used for exceptions.
- Performance rule – used for performance problems in any application, based on any Performance Monitor counter (like Procdump but with more options and UI). Also used for hangs or performance problems in IIS, based on internal ETW events.
- Memory and Handle Leak rule – for memory leaks coming from native code.
- Manual Dump collection– used in cases when the rules don’t apply, like performance.
- Automated Analysis feature – DebugDiag can debug the dump for you and present a report with the findings! The dump files don't need to be collected with DebugDiag.
Download:
How to use the Debug Diagnostics tool to troubleshoot a process that has stopped responding in IIS
https://support.microsoft.com/kb/919792/en-us
How to use the IIS Debug Diagnostics tool to troubleshoot a memory leak in an IIS process
https://support.microsoft.com/kb/919790/en-us
How to use the IIS Debug Diagnostics Tool to troubleshoot an IIS process that stops unexpectedly
https://support.microsoft.com/kb/919789/en-us
How to use the Debug Diagnostics Tool to troubleshoot high CPU usage by a process in IIS
https://support.microsoft.com/kb/919791/en-us
A client application may intermittently receive an error message when a client application tries to create a COM+ component
https://support.microsoft.com/kb/911359/en-us
ProcDump
- Enables you to collect dump files when a specific application is consuming high CPU.
- You choose the CPU threshold to trigger the dump.
- Excellent for intermitent high CPU scenarios.
Download:
https://technet.microsoft.com/en-us/sysinternals/dd996900.aspx
WinDbg
- Free and powerful Microsoft debugger.
- More powerful than Visual Studio.
- Enables user mode debugging and kernel debugging.
- Post-Mortem debugging (dump analysis) and live debugging.
- Downside: More difficult to use than Visual Studio but worth learning.
Download:
https://www.microsoft.com/whdc/devtools/debugging/default.mspx
Psscor2.dll – debugger extension:
Sosex.dll – debugger extension:
https://www.stevestechspot.com/
WinDbg Scripts
- Scripts are used to automate the debugging session.
- WinDbg Scripts are created with the WinDbg script language which is similar to C.
- WinDbg Scripts are great for small scripts.
Download:
https://blogs.msdn.com/debuggingtoolbox/
Netmon
- Use it when you suspect the bottleneck is network related.
- Collects logs from network activity.
- Easy way to visualize HTTP, TCP/IP and other types of network communication.
Download:
https://blogs.technet.com/netmon/
Parsers:
https://nmparsers.codeplex.com/
Fiddler
- Fiddler is a Web Debugging Proxy which logs all HTTP(S) traffic between your computer and the Internet.
- If you think the network might be the bottleneck for the poor performance of your web application you can use Fiddler or Netmon.
Download:
https://www.fiddler2.com/fiddler2/
Instructional videos:
https://www.fiddler2.com/Fiddler/help
NP .NET Profiler
- Easy to use .NET Profiler, easier to use than Visual Studio.
- Used for issues such as slow performance, memory related issues, and first-chance exceptions in .NET applications.
- It can troubleshoot the following types of .NET applications:
- ASP.NET Web Applications
- .NET Windows Applications (WCF, WPF and WF )
- .NET Console Applications
- .NET Window Services
- .NET COM+ Components
- Azure Service
Download:
https://www.microsoft.com/en-us/download/details.aspx?id=35370
If you have a free tool you’d like to share feel free to do that via Comments below.
Comments
Anonymous
October 04, 2012
Re Xperf / Windbg: have you seen this article: randomascii.wordpress.com/.../xperf-symbol-loading-pitfalls - re performance regressions in dbghelp.dll when used with Xperf?Anonymous
October 05, 2012
Hi Barry, I was not aware of that (thanks for sharing it! :)) and never had this situation happening to me. Here is why: a) When collecting a trace/dump file (whether it's XPerf, DebugDiag, UMDH, PerfView, etc...) you don't need symbols. b) When analyzing a trace/dump file then you'll need to setup the symbols. What I do is to use the same symbols folder/path I use for WinDbg/DebugDiag when analyzing the trace, so most, if not all symbols that I need, are already local in my machine. Thanks, RobertoAnonymous
October 07, 2012
Thanks.Anonymous
October 08, 2012
Rafarah, you misunderstood the problem which my blog post (pointed to by Barry Kelly) describes. Even if the symbols are on your machine then xperfview may not be able to load them -- if dbghelp.dll is not in its path. Even if xperfview is able to load symbols it may be painfully slow -- recent versions of dbghelp.dll can run up to 150x slower than older versions (2.5 hours to transcode a single PDB file versus one minute) when used by xperfview.Anonymous
October 08, 2012
Bruce, thanks for clarifying that. Since I haven't experience this issue (maybe because of the version of XPerf I've been using) I don't have much to say about it, so I suggest to reach these guys: blogs.technet.com/.../askpfeplat They have many posts about XPerf and may have more information about this problem.Anonymous
October 09, 2012
Really useful. thanks for sharing.Anonymous
March 01, 2013
I could not live without the API Monitor of Rohitab (www.rohitab.com/downloads) It logs WINAPI access in real time for your applications. Curiously, almost no one know it, but the only tools from Microsoft that had such quality are the sysinternals one. Ok was not MS initially... :p Nicolas,Anonymous
June 25, 2013
For those not having access to SQL Server Profiler, there's an express edition version that is free and open source. The original project on Google Code seems to have been removed by the developer after they got bought or something, but someone shared an archive of the source on GitHub, no binaries this time around but the source is available. It's nowhere as feature rich as the official MS version, but it was handy for SQL trace profiling as a free tool: github.com/.../sqlexpressprofiler