ISA Server Firewall Service crashed…but why?
1. Introduction
When question that I always receive when working with Firewall Service crashing is: why is it crashing? When the answer is: due a third party application…then the next question is: how is that? I thought each process was running independently and one couldn’t crash the other, right? That’s correct; however you need to remember how things work on ISA core architecture. Let’s step back and review the following diagram:
Figure 1 – ISA Server architecture (from ISA Server 2006 Firewall Core Document)
Notice that Firewall Service (wspsrv.exe) runs in User Mode while Firewall Engine (fweng.sys) runs in kernel mode. While is true that each process has its own address space, security token, etc, it is also true that each process is composed by threads, where each thread can be executing a different set of instructions and interacting with different components. ISA Server 2006 allows third party application to build their proprietary Web Filter (ISA Server supports ISAPI filter development) and by doing so it will somehow interfere in the way that Web Proxy Filter acts by default.
2. Digging in
If you use Process Explorer (or ProcMon) to open the properties of wspsrv.exe process you will see that there are many threads in execution as shown Figure 2:
Figure 2 – Threads running in the context of wspsrv.exe process.
If you select one of those threads and click Stack you will see the stack content and the modules in use. A stack is a region of the memory that is used to temporarily store data; it is added and removed in a last-in-first-out base. When you choose the thread and click on the stack you can see what it is in execution on that thread on that moment. Having this foundation understanding let’s take a look in the following diagram to understand how wspsrv.exe process can be affected by a third party filter:
Figure 3 – Firewall service process and the threads that belongs to it.
As you can see in this diagram there are some threads within the wspsrv.exe process and I’m using the stack of two of them as example. First stack from thread 1988 just have Microsoft modules and for the purpose of this example let’s focus on the stack that belongs to the thread 1920 which has a third party module (MyWebFilter.dll) loaded into it.
If this module for some reason execute an operation that cause an unhandled exception we might compromised the whole thread and possible crash the process. If you do not have a debugger attached to the process you will not get a dump for the wspsrv.exe, the only thing that will happen is that Firewall Service will crash (process quits from the memory) and an event is registered in the event viewer saying that the Firewall Service crashed. If you want to catch this type of crash you need a debugger attached to the process, to do that you can use an article that I wrote some time back about that, check it out here.
3. Access Violation
For the purpose of this example let's assume that this fake third party filter module did cause Firewall Service to crash and since I did have DebugDiag attached to wspsrv.exe process I was able to catch the second chance crash. In this case here it is the result for this crash by a partial output from !analyze –v command:
0:040> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
FAULTING_IP:
ntdll!KiUserExceptionDispatcher+e
7c82857e 0ac0 or al,al
EXCEPTION_RECORD: 102cf8cc -- (.exr 0x102cf8cc)
ExceptionAddress: 10161a50 (MyWebFilter.dll+0x00001a50)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000000
Parameter[1]: 10192068
Attempt to read from address 10192068
DEFAULT_BUCKET_ID: STATUS_STACKOVERFLOW
PROCESS_NAME: wspsrv.exe
ERROR_CODE: (NTSTATUS) 0xc00000fd - A new guard page for the stack cannot be created.
READ_ADDRESS: 1016caac
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
IP_MODULE_UNLOADED:
MyFilter+1a50
10161a50 ?? ???
CONTEXT: 102cf8e8 -- (.cxr 0x102cf8e8)
eax=102cfe44 ebx=00000000 ecx=10192048 edx=f9b10046 esi=10192048 edi=102cfe38
eip=10161a50 esp=102cfbb4 ebp=102cfbdc iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010206
MyWebFilter.dll+0x1a50:
10161a50 ?? ???
Resetting default scope
RECURRING_STACK: From frames 0x7 to 0xa
LAST_CONTROL_TRANSFER: from 102cfe38 to 10161a50
IP_ON_STACK:
+102cfe38
102cfe38 5c pop esp
FRAME_ONE_INVALID: 1
STACK_TEXT:
WARNING: Frame IP not in any known module. Following frames may be wrong.
102cfbb0 102cfe38 00000006 00000000 10192048 MyWebFilter.dll+0x1a50
102cfc2c 776bf813 77796898 00327277 00000000 0x102cfe38
102cfc30 77796898 00327277 00000000 00000000 ole32!COleStaticMutexSem::Release+0x1a
102cfe6c 7c83ac6c 00000000 0efd5400 0efd54a8 ole32!gComLock+0x18
102cfec8 7c83ca92 62251dc0 00000000 0efd5400 ntdll!RtlpWaitOrTimerCallout+0x74
102cfeec 7c83a857 0efd54a8 7c88b080 0ef88528 ntdll!RtlpAsyncWaitCallbackCompletion+0x37
102cff44 7c83aa3b 7c83ca5b 0efd54a8 00000000 ntdll!RtlpWorkerCallout+0x71
102cff64 7c83aab2 00000000 0efd54a8 0ef88528 ntdll!RtlpExecuteWorkerRequest+0x4f
102cff78 7c839f90 7c83a9fa 00000000 0efd54a8 ntdll!RtlpApcCallout+0x11
102cffb8 77e6482f 00000000 00000000 00000000 ntdll!RtlpWorkerThread+0x61
102cffec 00000000 7c839f2b 00000000 00000000 kernel32!BaseThreadStart+0x34
Let see our registers:
0:040> r
eax=00000000 ebx=00000000 ecx=1016caac edx=7c828786 esi=00000000 edi=00000000
eip=1016caac esp=102911b0 ebp=102911d0 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
MyWebFilter.dll+0xcaac:
1016caac ?? ???
Now let’s look at the EIP register (which points to where in the program the processor was currently executing the code):
0:040> r eip
eip=1016caac
Let’s dump it:
0:040> dd eip
1016caac ???????? ???????? ???????? ????????
1016cabc ???????? ???????? ???????? ????????
1016cacc ???????? ???????? ???????? ????????
1016cadc ???????? ???????? ???????? ????????
1016caec ???????? ???????? ???????? ????????
1016cafc ???????? ???????? ???????? ????????
1016cb0c ???????? ???????? ???????? ????????
1016cb1c ???????? ???????? ???????? ????????
Well, it doesn’t looks good since it is pointing to a bunch of question mark (either invalid or not accessible memory). Let’s see what memory address EIP was pointing to:
0:040> !address eip
10160000 : 10160000 - 00040000
Type 00000000
Protect 00000001 PAGE_NOACCESS
State 00010000 MEM_FREE
Usage RegionUsageFree
What this PAGE_NOACCESS means? Let’s see the definition from MSDN:
“Pages in the region become guard pages. Any attempt to read from or write to a guard page causes the operating system to raise the STATUS_GUARD_PAGE exception and turn off the guard page status. Guard pages thus act as a one-shot access alarm. The PAGE_GUARD flag is a page protection modifier. An application uses it with one of the other page protection flags, with one exception: it cannot be used with PAGE_NOACCESS. When an access attempt leads the operating system to turn off guard page status, the underlying page protection takes over. If a guard page exception occurs during a system service, the service typically returns a failure status indicator.”
From: https://msdn.microsoft.com/en-us/library/aa450977.aspx
One strong hypothesis here (since we don’t have the code for the third party application to debug) is that this module tried to access an invalid memory address and therefore corrupted the stack causing the access violation. This was enough to cause the whole process (wspsrv.exe) to crash.