Troubleshooting Null Reference Exceptions in live ASP.NET applications
This post is more developer centric than for administrators but as the industry moves towards DevOps both worlds are merging and is becoming seamless hence the information in this blog could be helpful to both sides.
In my day to day work, I do a lot of troubleshooting on the IIS Admin side but while delving into the scheme of things I realized that more often than not the issues lie in application code.
Most developers are able to troubleshoot code issues on their development boxes. But what if the issue is only happening on production servers and not on development machines. This will need live debugging, which is the idea behind this blog post.
For the purpose of this blog, I have created a Sample application which generates a Null pointer reference exception. But the debugging method can be used to troubleshoot any type of exception.
Here is the snip of the login form -
Once I click on the Submit button I get the below Server Error -
Note - This type of detailed errors will generally be seen when you are browsing the web page locally on the web server and not on remote clients (default settings). To change this you can follow - https://technet.microsoft.com/en-us/library/bb684665.aspx.
At the same time, you would generally see an event in the application system event log –
<snip>
Event code: 3005
Event message: An unhandled exception has occurred.
Application information:
Application domain: /LM/W3SVC/4/ROOT-1-130775233748599802
Trust level: Full
Application Virtual Path: /
Application Path: E:\SampleWebApp1\
Machine name: DESALG-LAPTOP
Process information:
Process ID: 18744
Process name: w3wp.exe
Account name: IIS APPPOOL\SampleWebApp1
Exception information:
Exception type: NullReferenceException
Exception message: Object reference not set to an instance of an object.
at SampleWebApp1.WebForm1.Button1_Click(Object sender, EventArgs e)
at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
Request information:
Request URL: https://localhost:8080/WebForm1.aspx
Request path: /WebForm1.aspx
User host address: ::1
User: FAREAST\desalg
Is authenticated: True
Authentication Type: Negotiate
Thread account name: IIS APPPOOL\SampleWebApp1
<snip>
From the stack trace, you can see that it tells us the function (SampleWebApp1.WebForm1.Button1_Click) in which the exception occurred but it does not tell us the line of code where the exception occurred.
Assuming this code is deployed on the web server, here is a way how you can debug this live using windbg.
You can download and install WinDBG from here - https://msdn.microsoft.com/en-us/windows/hardware/hh852365
Note – You can follow the link https://msdn.microsoft.com/en-us/library/dd465337(v=vs.110).aspx to deploy the attached project to this blog on the web server.
If you deployed this with the default settings then hitting the link - https://localhost:8080/WebForm1.aspx would open the web page and trigger the worker process on the server.
Step 1 – Setup the Source File path (Location of the application source files) – File > Source File Path …
Step 2 - Setup the Symbol File path (Location of the application pdb files) – File > Symbol File Path …
Step 3 – File > Attach to the process – Select the worker process
Once you click ‘Ok’, the worker process will get attached to WinDBG and the control will be with debugger.
Step 4 – Run the below command to load the appropriate version of SOS.dll in WinDBG which is required for the purpose of debugging.
0:035> .cordll -ve -u -l
Automatically loaded SOS Extension
CLRDLL: Loaded DLL C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscordacwks.dll
CLR DLL status: Loaded DLL C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscordacwks.dll
*** WARNING: Unable to verify checksum for SampleWebApp1.dll
0:035> .Chain –Gives the extensions loadedby windbg
<snipped>
Extension DLL chain:
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos: image 4.0.30319.34209, API 1.0.0, built Sat Apr 12 07:33:15 2014
[path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\SOS.dll]
<snipped>
Step 5 – Type ‘g’ in the command line of WinDBG to pass the control back to the application.
Step 6 – Fill in the login form and Click on Submit to generate the exception. The control will be sent back to WinDBG when the exception is hit.
0:035> g
(4910.4a7c): CLR exception - code e0434352 (first chance)
CLR exception type: System.Web.HttpUnhandledException
""
(4910.1dd4): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
SampleWebApp1!SampleWebApp1.WebForm1.Button1_Click(System.Object, System.EventArgs)+0x8:
00007ff7`c71318e8 8b00 mov eax,dword ptr [rax] ds:00000000`00000008=????????
If your source code alignment is properly done, windbg does a pretty neat job of highlighting the line of code that has caused the exception. This should be good enough information for the developer of the code to take forward the debugging.
For advanced folks, little bit more debugging and you can pull more details about the exception like the call stack and the type of exception.
0:040> !clrstack -- Gives the callstack of the thread which threw the exception
SampleWebApp1.WebForm1.Button1_Click(System.Object, System.EventArgs)
System.Web.UI.WebControls.Button.RaisePostBackEvent(System.String)
System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
System.Web.UI.Page.ProcessRequest()
System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)
[InlinedCallFrame: 0000003e724eeac8] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
[InlinedCallFrame: 0000003e724eeac8] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)
[ContextTransitionFrame: 0000003e724eefc8]
0:040> t –Step into the code
(4910.1dd4): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
KERNELBASE!RaiseException+0x6c:
00007ff8`337f8b9c 488b8c24c0000000 mov rcx,qword ptr [rsp+0C0h] ss:0000003e`724ed450=0000ed06eea53973
0:040> t
(4910.1dd4): CLR exception - code e0434352 (first chance)
clr!IEE+0x71a7:
00007ff8`268a9c03 483b4728 cmp rax,qword ptr [rdi+28h] ds:00007ff8`096b77d8=00000000000cb27b
0:040> !pe –Prints exception on the callstack and will work if the SOS dll is loaded
Exception object: 0000003ca7048cb8
Exception type: System.Web.HttpUnhandledException
Message: <none>
InnerException: System.NullReferenceException, Use !PrintException 0000003ca7048990 to see more.
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80004005
There are nested exceptions on this thread. Run with -nested for details
References - https://blogs.msdn.com/b/kaevans/archive/2011/04/11/intro-to-windbg-for-net-developers.aspx
Disclaimer: The information in this weblog is provided "AS IS" with no warranties, and confers no rights. This weblog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion. Inappropriate comments will be deleted at the authors discretion. All code samples are provided "AS IS" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular.