Debugging a Debugger to Debug a Dump
Recently I came across an instance where my debugger did not do what I wanted. Rarely do computers disobey me, but this one was unusually stubborn. There was no other option; I had to bend the debugger to my will.
There are many ways to make a computer program do what you want. If you have the source code you can rewrite and recompile the program. If you have a hex editor you can edit the code of the binary. A shim can be used to modify a program at runtime. In this instance I was in a hurry and I was ok with a temporary solution, so I used a debugger to change the execution of the debugger while it ran.
Debug a debugger? Can you do such a thing? Of course you can.
In this example a memory dump was captured of a system and I was asked to determine if the system had run out of desktop heap. Usually the !dskheap command can be used to determine how much heap has been used by each desktop. Unfortunately, this command failed me.
23: kd> !dskheap
Error Reading TotalFreeSize from nt!_HEAP @ fffffa8019c65c00
Failed to GetHeapInfo for desktop @fffffa8019c65c00
EnumDsktps failed on Winsta: 19c4f090FillWinstaArray failed
The error indicates that the command couldn’t read from the _HEAP structure at fffffa8019c65c00 for desktop fffffa8019c65c00. Further investigation found that reason I got this error is that the heap for the desktop in question is not valid memory. Because the memory is described by a prototype PTE I assume that the heap has not been initialized (Note: See Windows Internals’ Memory Management chapter for more information about proto PTEs).
23: kd> dt win32k!tagDESKTOP fffffa8019c65c00
+0x000 dwSessionId : 0
+0x008 pDeskInfo : 0xfffff900`c05e0a70 tagDESKTOPINFO
+0x010 pDispInfo : 0xfffff900`c0581e50 tagDISPLAYINFO
+0x018 rpdeskNext : 0xfffffa80`19c6ef20 tagDESKTOP
+0x020 rpwinstaParent : 0xfffffa80`19c4f090 tagWINDOWSTATION
+0x028 dwDTFlags : 0x110
+0x030 dwDesktopId : 0x19c65c00`00000003
+0x038 spmenuSys : (null)
+0x040 spmenuDialogSys : (null)
+0x048 spmenuHScroll : (null)
+0x050 spmenuVScroll : (null)
+0x058 spwndForeground : (null)
+0x060 spwndTray : (null)
+0x068 spwndMessage : 0xfffff900`c05e0d90 tagWND
+0x070 spwndTooltip : 0xfffff900`c05e0fa0 tagWND
+0x078 hsectionDesktop : 0xfffff8a0`00ef08e0 Void
+0x080 pheapDesktop : 0xfffff900`c05e0000 tagWIN32HEAP
+0x088 ulHeapSize : 0x18000
+0x090 cciConsole : _CONSOLE_CARET_INFO
+0x0a8 PtiList : _LIST_ENTRY [ 0xfffffa80`19c65ca8 - 0xfffffa80`19c65ca8 ]
+0x0b8 spwndTrack : (null)
+0x0c0 htEx : 0n0
+0x0c4 rcMouseHover : tagRECT
+0x0d4 dwMouseHoverTime : 0
+0x0d8 pMagInputTransform : (null)
23: kd> dd 0xfffff900`c05e0000
fffff900`c05e0000 ???????? ???????? ???????? ????????
fffff900`c05e0010 ???????? ???????? ???????? ????????
fffff900`c05e0020 ???????? ???????? ???????? ????????
fffff900`c05e0030 ???????? ???????? ???????? ????????
fffff900`c05e0040 ???????? ???????? ???????? ????????
fffff900`c05e0050 ???????? ???????? ???????? ????????
fffff900`c05e0060 ???????? ???????? ???????? ????????
fffff900`c05e0070 ???????? ???????? ???????? ????????
23: kd> !pte fffff900`c05e0000
VA fffff900c05e0000
PXE at FFFFF6FB7DBEDF90 PPE at FFFFF6FB7DBF2018 PDE at FFFFF6FB7E403010 PTE at FFFFF6FC80602F00
contains 000000076245C863 contains 0000000762569863 contains 000000045FA17863 contains F8A000F4F9780400
pfn 76245c ---DA--KWEV pfn 762569 ---DA--KWEV pfn45fa17 ---DA--KWEV not valid
Proto: FFFFF8A000F4F978
There are many desktops in this session and I wanted to know about the usage of the other desktops, but the !dskheap command stopped after just one error. I needed to force it to continue after this error, so I launched a debugger to debug my debugger. There is a command to do this, just run .dbgdbg.
23: kd> .dbgdbg
Debugger spawned, connect with
"-remotenpipe:icfenable,pipe=cdb_pipe,server=NINJA007"
For clarity I will call the original debugger where I ran !dskheap debugger1, and the new debugger spawned by .dbgdbg debugger2 .
Before switching to debugger2 I need to know what I am going to debug. The error message gives a hint about where to set a breakpoint, I am looking for a failure from GetHeapInfo.
23: kd> !dskheap
Error Reading TotalFreeSize from nt!_HEAP @ fffffa8019c65c00
Failed to GetHeapInfo for desktop @fffffa8019c65c00
EnumDsktps failed on Winsta: 19c4f090FillWinstaArray failed
I need to know which module GetHeapInfo is in, the .extmatch match command indicates which module contains the !dskheap command.
23: kd> .extmatch dskheap
!kdexts.dskheap
Switching to debugger2 I set a breakpoint on kdexts!GetHeapInfo. Use Ctrl+C to trigger a debug break in cdb (this is the same as a Ctrl+Break in windbg).
0:004> bp kdexts!GetHeapInfo
0:004> g
Switch back to debugger1 and run the !dskheap command.
23: kd> !dskheap
In debugger2 I have hit the breakpoint.
Breakpoint 0 hit
kdexts!GetHeapInfo:
000007f9`4237b9b0 4055 push rbp
The error says GetHeapInfo failed, so I am interested in what this function returns. To see what GetHeapInfo returns I go up one level in the stack and set a new breakpoint on the code just after it returns. This new breakpoint will also dump the return value of GetHeapInfo (return values are always in the rax register).
0:000> gu
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> r rax
rax=0000000000000000
0:000> bc *
0:000> bp 000007f9`4237b483 "r rax"
0:000> g
The next time the breakpoint hit the return value was 1, which in this instance means GetHeapInfo failed. This is where I exerted my control over the computer: I forced the return value to 0.
rax=0000000000000001
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> r rax=0
I ran through the other breakpoints and changed rax as necessary.
0:000> g
rax=0000000000000000
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> g
rax=0000000000000000
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> g
rax=0000000000000000
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> g
rax=0000000000000000
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> g
rax=0000000000000001
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> r rax=0
0:000> g
rax=0000000000000000
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> g
Everything was going well, until the computer defied me again. The !dskheap output computes the percentage of heap usage by dividing the bytes used by the size of the heap. Unfortunately, the size of the heap was left at 0 for the two heaps where I changed the return value. It is well known that only Chuck Norris can divide by zero; to prevent a roundhouse kick to your computer the processor generates an exception.
(2d0.928): Integer divide-by-zero - code c0000094 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
kdexts!DisplayInfo+0x2ee:
000007f9`4237b90e 49f7f3 div rax,r11
0:000> r r11
r11=0000000000000000
0:000> r rax
rax=0000000000000000
0:000> g
Fortunately debugger1 handles the divide by zero exception and it is easy to run !dskheap again.
23: kd> !dskheap
Back in debugger2 I set a new breakpoint on the div instruction that outputs the divisor. When the divisor (r11) is 0 I changed it to a non-zero value to avoid the wrath of Mr. Norris.
rax=0000000000000000
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> bp 000007f9`4237b90e
0:000> bp 000007f9`4237b90e "r r11"
breakpoint 1 redefined
0:000> g
rax=0000000000000001
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> r rax=0
0:000> g
rax=0000000000000000
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> g
rax=0000000000000000
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> g
rax=0000000000000000
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> g
rax=0000000000000000
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> g
rax=0000000000000001
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> r rax=0
0:000> g
rax=0000000000000000
kdexts!EnumDsktps+0x197:
000007f9`4237b483 4885c0 test rax,rax
0:000> g
r11=0000000000033333
kdexts!DisplayInfo+0x2ee:
000007f9`4237b90e 49f7f3 div rax,r11
0:000> g
r11=0000000000000000
kdexts!DisplayInfo+0x2ee:
000007f9`4237b90e 49f7f3 div rax,r11
0:000> r r11=1
0:000> g
r11=00000000000007ae
kdexts!DisplayInfo+0x2ee:
000007f9`4237b90e 49f7f3 div rax,r11
0:000> g
r11=0000000000013333
kdexts!DisplayInfo+0x2ee:
000007f9`4237b90e 49f7f3 div rax,r11
0:000> g
r11=0000000000013333
kdexts!DisplayInfo+0x2ee:
000007f9`4237b90e 49f7f3 div rax,r11
0:000> g
r11=0000000000013333
kdexts!DisplayInfo+0x2ee:
000007f9`4237b90e 49f7f3 div rax,r11
0:000> g
r11=0000000000000000
kdexts!DisplayInfo+0x2ee:
000007f9`4237b90e 49f7f3 div rax,r11
0:000> r r11=1
0:000> g
r11=0000000000013333
kdexts!DisplayInfo+0x2ee:
000007f9`4237b90e 49f7f3 div rax,r11
0:000> g
Finally, back in debugger1 I now have complete output for !dskheap. After a few strategic modifications of the program’s execution I got it to output the data I wanted. As it turns out we aren’t out of desktop heap after all.
23: kd> !dskheap
Error Reading TotalFreeSize from nt!_HEAP @ fffffa8019c65c00
Error Reading TotalFreeSize from nt!_HEAP @ fffffa801a53ea30
Winstation\Desktop Heap Size(KB) Used Rate(%)
------------------------------------------------------------
WinSta0\Default 20480 0%
WinSta0\Disconnect 0 0%
WinSta0\Winlogon 192 2%
Service-0x0-3e7$\Default 7680 1%
Service-0x0-3e4$\Default 7680 0%
Service-0x0-3e5$\Default 7680 0%
Service-0x0-26f46a$\Default 0 0%
Service-0x0-2706f2$\Default 7680 0%
------------------------------------------------------
Total Desktop: ( 51392 KB - 8 desktops)
Session ID: 0
============================================================
Comments
- Anonymous
February 28, 2013
This post is pure gold - useful and funny at the same time!