Kernel dump analysis - Bugcheck 0xA (IRQL_NOT_LESS_OR_EQUAL)
Yet another kernel memory dump to be analyzed - The bugcheck this time is the 0xA - IRQL_NOT_LESS_OR_EQUAL. To better understand what this message means we would need a little background on Windows Internals but basically when executing anything at a interrupt request level (IRQL) = 2 or higher (in normal circumstances instructions get executed at IRQL = 0) we can't page fault. On the situation of this specific dump, we would crash even on IRQL = 0 since we're trying to write to the memory address 0x0 which is in user mode address space and is reserved (the first useable address in the user mode address space is 0x00010000)
Let's jump to the analysis.
Bugcheck info:
0: kd> .bugcheck
Bugcheck code 0000000A
Arguments 00000000 00000002 00000001 8087bb19
| | | |à This is the failing instruction address
| | |à 0x1 means WRITE, 0x0 means READ, so it crashed when trying to WRITE to 0x0
| |à This is the IRQL at which problem happened (IRQL=2)
|à This is the memory address where the problem happened
So it crashed basically because it failed to access a memory address (which at a lower IRQLs would result in a page fault) on the IRQL = 2. At IRQL = 2 and above page faults are not allowed to happen.
Current thread stack:
0: kd> kvnL
# ChildEBP RetAddr Args to Child
00 b7d6ebbc 8087bb19 badb0d00 00000000 00000000 nt!_KiTrap0E+0x2a7 (FPO: [0,0] TrapFrame @ b7d6ebbc)
01 b7d6ec40 80972b03 f7ca7008 80a56be4 00000000 nt!ExDeleteResourceLite+0x1f (FPO: [Non-Fpo]) (CONV: stdcall)
02 b7d6ec5c 80932ca2 ef4d0860 ef4d0848 00000000 nt!SepTokenDeleteMethod+0x9d (FPO: [Non-Fpo]) (CONV: stdcall)
03 b7d6ec74 8086c1a5 ef4d0860 00000000 8b478df0 nt!ObpRemoveObjectRoutine+0xde (FPO: [Non-Fpo]) (CONV: stdcall)
04 b7d6ec94 b7765ada f7ca3444 b7764007 f7ca8460 nt!ObfDereferenceObject+0x67 (FPO: [Non-Fpo]) (CONV: fastcall)
WARNING: Stack unwind information not available. Following frames may be wrong.
05 b7d6ecb0 b776037a f7cbf004 b7d6ece0 f82cde28 naiavf5x+0xcada
06 b7d6ecf0 b775b520 f82cde28 f82cde28 8b572548 naiavf5x+0x737a
07 b7d6ed04 8081dcdf 8b478990 f82cde28 f82cde38 naiavf5x+0x2520
08 b7d6ed18 808f8be4 00000000 00000000 f7ca8448 nt!IofCallDriver+0x45 (FPO: [Non-Fpo]) (CONV: fastcall)
09 b7d6ed50 80932ca2 00ca8460 f7ca8460 00000001 nt!IopDeleteFile+0x13a (FPO: [Non-Fpo]) (CONV: stdcall)
0a b7d6ed68 80932cec f7ca8460 00000001 8ad47770 nt!ObpRemoveObjectRoutine+0xde (FPO: [Non-Fpo]) (CONV: stdcall)
0b b7d6ed80 8087f92f 00000000 00000000 8ad47770 nt!ObpProcessRemoveObjectQueue+0x36 (FPO: [1,0,0]) (CONV: stdcall)
0c b7d6edac 80948bd0 00000000 00000000 00000000 nt!ExpWorkerThread+0xeb (FPO: [Non-Fpo]) (CONV: stdcall)
0d b7d6eddc 8088d4e2 8087f844 80000000 00000000 nt!PspSystemThreadStartup+0x2e (FPO: [Non-Fpo]) (CONV: stdcall)
0e 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16
The current frame being executed is a post-exception frame, I mean, it's just handling the situation. The problem really happened here:
0: kd> .trap b7d6ebbc
ErrCode = 00000002
eax=00000000 ebx=ef4d0848 ecx=00000000 edx=00000000 esi=f7ca7008 edi=00000000
eip=8087bb19 esp=b7d6ec30 ebp=b7d6ec40 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
nt!ExDeleteResourceLite+0x1f:
8087bb19 8901 mov dword ptr [ecx],eax ds:0023:00000000=????????
0: kd> kvnL
*** Stack trace for last set context - .thread/.cxr resets it
# ChildEBP RetAddr Args to Child
00 b7d6ec40 80972b03 f7ca7008 80a56be4 00000000 nt!ExDeleteResourceLite+0x1f (FPO: [Non-Fpo]) (CONV: stdcall)
01 b7d6ec5c 80932ca2 ef4d0860 ef4d0848 00000000 nt!SepTokenDeleteMethod+0x9d (FPO: [Non-Fpo]) (CONV: stdcall)
02 b7d6ec74 8086c1a5 ef4d0860 00000000 8b478df0 nt!ObpRemoveObjectRoutine+0xde (FPO: [Non-Fpo]) (CONV: stdcall)
03 b7d6ec94 b7765ada f7ca3444 b7764007 f7ca8460 nt!ObfDereferenceObject+0x67 (FPO: [Non-Fpo]) (CONV: fastcall)
WARNING: Stack unwind information not available. Following frames may be wrong.
04 b7d6ecb0 b776037a f7cbf004 b7d6ece0 f82cde28 naiavf5x+0xcada
05 b7d6ecf0 b775b520 f82cde28 f82cde28 8b572548 naiavf5x+0x737a
06 b7d6ed04 8081dcdf 8b478990 f82cde28 f82cde38 naiavf5x+0x2520
07 b7d6ed18 808f8be4 00000000 00000000 f7ca8448 nt!IofCallDriver+0x45 (FPO: [Non-Fpo]) (CONV: fastcall)
08 b7d6ed50 80932ca2 00ca8460 f7ca8460 00000001 nt!IopDeleteFile+0x13a (FPO: [Non-Fpo]) (CONV: stdcall)
09 b7d6ed68 80932cec f7ca8460 00000001 8ad47770 nt!ObpRemoveObjectRoutine+0xde (FPO: [Non-Fpo]) (CONV: stdcall)
0a b7d6ed80 8087f92f 00000000 00000000 8ad47770 nt!ObpProcessRemoveObjectQueue+0x36 (FPO: [1,0,0]) (CONV: stdcall)
0b b7d6edac 80948bd0 00000000 00000000 00000000 nt!ExpWorkerThread+0xeb (FPO: [Non-Fpo]) (CONV: stdcall)
0c b7d6eddc 8088d4e2 8087f844 80000000 00000000 nt!PspSystemThreadStartup+0x2e (FPO: [Non-Fpo]) (CONV: stdcall)
0d 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16
So it's crashing when trying to write to the address pointed by ECX - mov dword ptr [ecx], eax - This is the assembly code of the function where it crashed from the start to the crash instruction :
0: kd> u nt!ExDeleteResourceLite nt!ExDeleteResourceLite+0x1f + 1
nt!ExDeleteResourceLite [d:\nt\base\ntos\ex\resource.c @ 2250]:
8087bafa 8bff mov edi,edi
8087bafc 55 push ebp
8087bafd 8bec mov ebp,esp
8087baff 83ec0c sub esp,0Ch
8087bb02 56 push esi
8087bb03 8d55f4 lea edx,[ebp-0Ch]
8087bb06 b9c0418b80 mov ecx,offset nt!ExpResourceSpinLock (808b41c0)
8087bb0b ff1508118080 call dword ptr [nt!_imp_KeAcquireInStackQueuedSpinLock (80801108)]
8087bb11 8b7508 mov esi,dword ptr [ebp+8] ßESI is obtaining its value from the content of EBP+8
8087bb14 8b4e04 mov ecx,dword ptr [esi+4] ß ECX is obtaining its value from the content of ESI+4
8087bb17 8b06 mov eax,dword ptr [esi]
8087bb19 8901 mov dword ptr [ecx],eax ß This is where it crashed
ECX is getting its value from the address pointed by ESI + 4, and ESI is getting its value from the address pointed by EBP + 8 which is a pointer to the first parameter passed from the previous function on the stack. Confirming:
0: kd> dd esi+4 l1
f7ca700c 00000000
0: kd> dd poi(ebp+8) l1
f7ca7008 00000000
Next step is look to the previous function on the stack. The assembly for the next function up to the point it calls the current one is this:
0: kd> u nt!SepTokenDeleteMethod nt!SepTokenDeleteMethod+0x9d
nt!SepTokenDeleteMethod [d:\nt\base\ntos\se\token.c @ 2652]:
80972a66 8bff mov edi,edi
80972a68 55 push ebp
80972a69 8bec mov ebp,esp
80972a6b 51 push ecx
80972a6c 51 push ecx
80972a6d 56 push esi
80972a6e 8b7508 mov esi,dword ptr [ebp+8] ß ESI is coming from the content EBP + 8 (first parameter passed from the previous function)
80972a71 f6868800000020 test byte ptr [esi+88h],20h
80972a78 57 push edi
80972a79 7538 jne nt!SepTokenDeleteMethod+0x4d (80972ab3)
80972a7b 8bbe94000000 mov edi,dword ptr [esi+94h]
80972a81 53 push ebx
80972a82 8d4f0c lea ecx,[edi+0Ch]
80972a85 eb0f jmp nt!SepTokenDeleteMethod+0x30 (80972a96)
80972a87 8d42ff lea eax,[edx-1]
80972a8a 8bd8 mov ebx,eax
80972a8c 8bc2 mov eax,edx
80972a8e f00fb119 lock cmpxchg dword ptr [ecx],ebx
80972a92 3bc2 cmp eax,edx
80972a94 741c je nt!SepTokenDeleteMethod+0x4c (80972ab2)
80972a96 8b11 mov edx,dword ptr [ecx]
80972a98 83fa01 cmp edx,1
80972a9b 75ea jne nt!SepTokenDeleteMethod+0x21 (80972a87)
80972a9d 8b4704 mov eax,dword ptr [edi+4]
80972aa0 8945f8 mov dword ptr [ebp-8],eax
80972aa3 8b4708 mov eax,dword ptr [edi+8]
80972aa6 8945fc mov dword ptr [ebp-4],eax
80972aa9 8d45f8 lea eax,[ebp-8]
80972aac 50 push eax
80972aad e890ecffff call nt!SepDeReferenceLogonSession (80971742)
80972ab2 5b pop ebx
80972ab3 8d4638 lea eax,[esi+38h]
80972ab6 8b08 mov ecx,dword ptr [eax]
80972ab8 0b4804 or ecx,dword ptr [eax+4]
80972abb 6a00 push 0
80972abd 5f pop edi
80972abe 7407 je nt!SepTokenDeleteMethod+0x61 (80972ac7)
80972ac0 57 push edi
80972ac1 50 push eax
80972ac2 e8b5140000 call nt!SepModifyTokenPolicyCounter (80973f7c)
80972ac7 8b4678 mov eax,dword ptr [esi+78h]
80972aca 3bc7 cmp eax,edi
80972acc 7407 je nt!SepTokenDeleteMethod+0x6f (80972ad5)
80972ace 57 push edi
80972acf 50 push eax
80972ad0 e86ff8f1ff call nt!ExFreePoolWithTag (80892344)
80972ad5 8b868c000000 mov eax,dword ptr [esi+8Ch]
80972adb 3bc7 cmp eax,edi
80972add 7406 je nt!SepTokenDeleteMethod+0x7f (80972ae5)
80972adf 50 push eax
80972ae0 e85766ffff call nt!SeFreeCapturedObjectTypeList (8096913c)
80972ae5 8b8690000000 mov eax,dword ptr [esi+90h]
80972aeb 3bc7 cmp eax,edi
80972aed 7407 je nt!SepTokenDeleteMethod+0x90 (80972af6)
80972aef 57 push edi
80972af0 50 push eax
80972af1 e84ef8f1ff call nt!ExFreePoolWithTag (80892344)
80972af6 8b4630 mov eax,dword ptr [esi+30h] ß EAX is coming from the content of ESI + 0x30
80972af9 3bc7 cmp eax,edi
80972afb 740f je nt!SepTokenDeleteMethod+0xa6 (80972b0c)
80972afd 50 push eax ß EAX is pushed on the stack as the first parameter
80972afe e8f78ff0ff call nt!ExDeleteResourceLite (8087bafa)
So once again we need to look at the previous function since this one is also receiving the bad value as parameter. By revisiting the stack we see the first parameter it passes is also received as below:
0: kd> kbL5
ChildEBP RetAddr Args to Child
b7d6ec40 80972b03 f7ca7008 80a56be4 00000000 nt!ExDeleteResourceLite+0x1f
b7d6ec5c 80932ca2 ef4d0860 ef4d0848 00000000 nt!SepTokenDeleteMethod+0x9d
b7d6ec74 8086c1a5 ef4d0860 00000000 8b478df0 nt!ObpRemoveObjectRoutine+0xde
b7d6ec94 b7765ada f7ca3444 b7764007 f7ca8460 nt!ObfDereferenceObject+0x67
WARNING: Stack unwind information not available. Following frames may be wrong.
b7d6ecb0 b776037a f7cbf004 b7d6ece0 f82cde28 naiavf5x+0xcada
As we can go directly to the function nt!ObfDereferenceObject+0x67 and try to see where its setting up that parameter from. The assembly code from this function to the point it calls the next one is this:
0: kd> u nt!ObfDereferenceObject nt!ObfDereferenceObject+0x67 ß This is a fastcall function so the first two parameters are passed through the registers ECX and EDX instead of using the stack (EBP as a reference)
nt!ObfDereferenceObject [d:\nt\base\ntos\ob\obref.c @ 2441]:
8086c13e 8bff mov edi,edi
8086c140 55 push ebp
8086c141 8bec mov ebp,esp
8086c143 51 push ecx
8086c144 803de0088a8000 cmp byte ptr [nt!ObpTraceEnabled (808a08e0)],0
8086c14b 53 push ebx
8086c14c 56 push esi
8086c14d 57 push edi
8086c14e 894dfc mov dword ptr [ebp-4],ecx ß The location pointed by EBP-0x4 which is local variable is receiving the value of ECX which is the first parameter passed to this function
8086c151 8d71e8 lea esi,[ecx-18h]
8086c154 7408 je nt!ObfDereferenceObject+0x20 (8086c15e)
8086c156 6a00 push 0
8086c158 56 push esi
8086c159 e8c8fcffff call nt!ObpPushStackInfo (8086be26)
8086c15e 83cbff or ebx,0FFFFFFFFh
8086c161 f00fc11e lock xadd dword ptr [esi],ebx
8086c165 4b dec ebx
8086c166 7547 jne nt!ObfDereferenceObject+0x71 (8086c1af)
8086c168 8b3d90108080 mov edi,dword ptr [nt!_imp__KeGetCurrentIrql (80801090)]
8086c16e ffd7 call edi
8086c170 64a124010000 mov eax,dword ptr fs:[00000124h]
8086c176 6683787200 cmp word ptr [eax+72h],0
8086c17b 752c jne nt!ObfDereferenceObject+0x6b (8086c1a9)
8086c17d ffd7 call edi
8086c17f 3c01 cmp al,1
8086c181 7326 jae nt!ObfDereferenceObject+0x6b (8086c1a9)
8086c183 803de0088a8000 cmp byte ptr [nt!ObpTraceEnabled (808a08e0)],0
8086c18a 740f je nt!ObfDereferenceObject+0x5d (8086c19b)
8086c18c 833dd8078a8000 cmp dword ptr [nt!ObpTraceNoDeregister (808a07d8)],0
8086c193 7506 jne nt!ObfDereferenceObject+0x5d (8086c19b)
8086c195 56 push esi
8086c196 e805fcffff call nt!ObpDeregisterObject (8086bda0)
8086c19b 6a00 push 0
8086c19d ff75fc push dword ptr [ebp-4] ß Here it is pushing the content of EBP-4 which we know is coming from ECX (first parameter) as a first parameter to the next function
8086c1a0 e81f6a0c00 call nt!ObpRemoveObjectRoutine (80932bc4) ß this is the function call to the next function
So we know the bad value is coming from the previous function on the stack which is naiavf5x+0xcada. By looking at the assembly for that we have this:
0: kd> u b7765ac1-10 b7765ad5+2
naiavf5x+0xcab1:
b7765ab1 ff1550a476b7 call dword ptr [naiavf5x+0x11450 (b776a450)]
b7765ab7 5f pop edi
b7765ab8 8bc6 mov eax,esi
b7765aba 5e pop esi
b7765abb 5b pop ebx
b7765abc 5d pop ebp
b7765abd c20800 ret 8
b7765ac0 56 push esi
b7765ac1 8bf1 mov esi,ecx ß ESI is coming from ECX
b7765ac3 8d86a8000000 lea eax,[esi+0A8h]
b7765ac9 50 push eax
b7765aca e8818affff call naiavf5x+0x5550 (b775e550)
b7765acf 85c0 test eax,eax
b7765ad1 7516 jne naiavf5x+0xcae9 (b7765ae9)
b7765ad3 8bce mov ecx,esi ß ECX which will be the first parameter to the next function is coming from ESI
b7765ad5 e8b6f7ffff call naiavf5x+0xc290 (b7765290)
CONCLUSION:
So at this point we can’t move further since we have no symbols for the NAIAVF5X.SYS driver – which is a file system filter driver from McAfee. The conclusion would be that this driver is bad until the McAfee analyze this same dump and prove us wrong J. Here is some addition info about the driver:
0: kd> lmvm naiavf5x
start end module name
b7759000 b776d440 naiavf5x (no symbols)
Loaded symbol image file: naiavf5x.sys
Image path: \SystemRoot\system32\drivers\naiavf5x.sys
Image name: naiavf5x.sys
Timestamp: Mon Aug 04 20:08:00 2003 (3F2F0370) ß By the way, this is old stuff. They might have already corrected this on newer versions of the driver and I wouldn’t be surprised to find some documentation about this on McAfee’s web site.
CheckSum: 00020D5B
ImageSize: 00014440
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0
By the way, for this dump, “!analyze –v” does the job and points to the right guilty driver.
Until the next dump analysis, I mean, next post :)
Comments
- Anonymous
January 01, 2003
Hello Marcelo, I am trying to apply your debugging steps to a few 0xA dumps we have had over the last few weeks, all were 'probably caused' by TDI.sys. (so I'm guessing something network related) The last function called before the stack was: hal!KeAcquireSpinLockRaiseToSynch+0xe Any tips on further analyzing this problem? I really find this stuff interesting but I'm guessing I would have to aquire some Assembler knowledge to be able to debug these kind of issues :S
ChildEBP RetAddr Args to Child
00 b7104a04 80a5a3de badb0d00 00000000 80a5da73 nt!KiTrap0E+0x2a7 (FPO: [0,0] TrapFrame @ b7104a04) 01 b7104a74 b4bb1b6c b7104ae0 8ab6ae48 00000000 hal!KeAcquireSpinLockRaiseToSynch+0xe (FPO: [0,0,0]) 02 b7104a8c b4bb750d 89351002 00000000 00000029 tcpip!TCPDataRequestComplete+0x40 (FPO: [Non-Fpo]) 03 b7104aac b4bb74ad b7104ae0 00003700 88e3e1a8 tcpip!CompleteSends+0x31 (FPO: [Non-Fpo]) 04 b7104aec b4bb6cbc 00000029 3b806af8 8c51e700 tcpip!TcpFastReceive+0x390 (FPO: [Non-Fpo]) 05 b7104bc8 b4bb32bf 8cb63f48 0100007f 0100007f tcpip!TCPRcv+0x723 (FPO: [Non-Fpo]) 06 b7104c28 b4bb14de 00000024 8cb63f48 b4bb6a6d tcpip!DeliverToUser+0x189 (FPO: [Non-Fpo]) 07 b7104cb8 b4bbd163 8cb63f48 b7104d30 00000014 tcpip!IPRcvPacket+0x686 (FPO: [Non-Fpo]) 08 b7104d64 f7658064 b4bf2ee0 8cb63f48 8cf50660 tcpip!LoopXmitRtn+0x195 (FPO: [Non-Fpo]) 09 b7104d80 8088043d 8cb63f48 00000000 8cf50660 TDI!CTEpEventHandler+0x32 (FPO: [Non-Fpo]) 0a b7104dac 80949b7c b4bf2ee0 00000000 00000000 nt!ExpWorkerThread+0xeb (FPO: [Non-Fpo]) 0b b7104ddc 8088e062 80880352 00000001 00000000 nt!PspSystemThreadStartup+0x2e (FPO: [Non-Fpo]) 0c 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16 3: kd> .trap b7104a04 ErrCode = 00000002 eax=00000002 ebx=00000000 ecx=00000026 edx=00000000 esi=89351040 edi=00000002 eip=80a5a3de esp=b7104a78 ebp=b7104a8c iopl=0 nv up ei pl nz na po nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010202 hal!KeAcquireSpinLockRaiseToSynch+0xe: 80a5a3de f00fba2900 lock bts dword ptr [ecx],0 ds:0023:00000026=???????? Sincerely, Enrico Klein
- Anonymous
June 01, 2011
I had the exact same problem on <a href="www.articlewritingclicks.com/.../a> on a McAffee Internet Security version.I switched to another antiv and it never came back.I wonder if they know about it?