Dr Watson knows best..
I recently worked on a case which involved a crash ( it was the MMC ) which didn’t have a lot of info attached to it. There was no crash dump file, and no live debug to go on. All that I had was a Dr Watson log.
Who’s that you say? See here for some history of the good doctor.
If you run drwtsn32.exe ( not in Vista ) you will see some configuration options
One is a path to the log file, and the other is a path to where it will store the dump.
If you look in your machine, I’ll bet you have more than a few entries in your Watson log as well J
Here is what this one looked like… ( a few things changed.. bonus points to someone with enough time to waste to tell me the real module )
Application exception occurred:
App: C:\WINDOWS\system32\mmc.exe (pid=3444)
When: 10/30/2007 @ 15:35:22.324
Exception number: c0000005 (access violation)
*----> System Information <----*
Computer Name: MyComputer
User Name: spatdsg
Terminal Session Id: 1
Number of Processors: 4
Processor Type: x86 Family 15 Model 4 Stepping 10
Windows Version: 5.2
Current Build: 3790
Service Pack: 1
Current Type: Multiprocessor Free
Registered Organization: redmond
Registered Owner: Spatdsg
*----> Task List <----*
0 System Process
4 System
452 smss.exe
560 csrss.exe
932 svchost.exe
1008 svchost.exe
1068 svchost.exe
1088 svchost.exe
<snip>
---> Module List <----*
000000000400000 - 0000000000409000: C:\WINDOWS\system32\Normaliz.dll
0000000000b20000 - 0000000000b37000: C:\WINDOWS\system32\odbcint.dll
0000000001000000 - 00000000010d2000: C:\WINDOWS\system32\mmc.exe
00000000013d0000 - 00000000013dc000: C:\Program Files\Exchsrvr\bin\pttrace.dll
0000000076dc0000 - 0000000076de7000: C:\WINDOWS\system32\Foo.dll
<snip>
Now, when I get these I generally open them in notepad, start from the bottom of the file and work my way up - looking for the word “FAULT”
The file keeps a record of oldest to newest crashes ( newest being at the bottom of the file ) .
Once I locate the most recent crash I examine the stack which crashed.
eax=00000000 ebx=00000080 ecx=7c83005c edx=00000000 esi=77e6c24b edi=40000000
eip=76ddcb3c esp=0007e850 ebp=0007e868 iopl=0 nv up ei pl zr na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
C:\WINDOWS\system32\Foo.dll -
function: Foo!SchemaIsClassAContainer
76ddcb1e 08ff or bh,bh
76ddcb20 d6 ???
76ddcb21 83f8ff cmp eax,0xffffffff
76ddcb24 7556 jnz Foo!SchemaIsClassAContainer+0xce3 (76ddcb7c)
76ddcb26 ff157c10dc76 call dword ptr [Foo+0x107c (76dc107c)]
76ddcb2c 83f803 cmp eax,0x3
76ddcb2f 7544 jnz Foo!SchemaIsClassAContainer+0xcdc (76ddcb75)
76ddcb31 6a5c push 0x5c
76ddcb33 ff7508 push dword ptr [ebp+0x8]
76ddcb36 ff154411dc76 call dword ptr [Foo+0x1144 (76dc1144)]
FAULT ->76ddcb3c 66832000 and word ptr [eax],0x0 ds:0023:00000000=???
76ddcb41 59 pop ecx
76ddcb42 6a00 push 0x0
76ddcb44 ff7508 push dword ptr [ebp+0x8]
76ddcb47 8945fc mov [ebp-0x4],eax
76ddcb4a ff15cc10dc76 call dword ptr [Foo+0x10cc (76dc10cc)]
76ddcb50 85c0 test eax,eax
76ddcb52 741b jz Foo!SchemaIsClassAContainer+0xcd6 (76ddcb6f)
76ddcb54 8b45fc mov eax,[ebp-0x4]
76ddcb57 66c7005c00 mov word ptr [eax],0x5c
As you can see - there are no symbols present for this crash , so we can’t trust the function it thinks it was in – which is “Foo!SchemaIsClassAContainer”. You can however, trust the module but not the function name
So .. where and why did we really crash?
We call some funtion blah() which appears to return a pointer .
76ddcb36 ff154411dc76 call dword ptr [Foo+0x1144 (76dc1144)]
Then we dereference this pointer and try to set that to zero via a bitwise AND.
However – the pointer located in EAX is… zero.
76ddcb3c 66832000 and word ptr [eax],0x0 ds:0023:00000000=????
Obviously we aren’t checking for NULL on some return.
How to find this source code and what real function we’re calling? If I had the exact same version, and it loaded in the same spot, I could just do an ‘ln’ or start looking at the address 76ddcb3c. But, I don’t and I can’t.
I don’t have the dump, just this text file.
There are some hardcoded, and hopefully somewhat unique, interesting opcodes we see in the Watson log.
I pasted them below:
76ddcb2c 83f803 cmp eax,0x3
And this sequence:
76ddcb31 6a5c push 0x5c
76ddcb33 ff7508 push dword ptr [ebp+0x8]
So let’s try and look for some opcode pattern like this:
5c 6a 75 ff
When I start the MMC and load this dll - my DLL loads at:
0000000076dc0000 - 0000000076de7000: C:\WINDOWS\system32\Foo.dll
I get that information from the “lm” command.
Then I determine how much room we need to search through.
0:000> ? 0000000076de7000- 0000000076dc0000
Evaluate expression: 159744 = 00027000
Then I can search via the ‘s’ command:
WORD..
0:000> s -w 76dc0000 L0x27000 5c6a
76de1f04 5c6a 75ff ff08 5815 dc10 6676 2083 5900 j\.u...X..vf. .Y
Or:
DWORD..
0:003> s -d 76dc0000 L0x27000 75ff5c6a
76de1f04 75ff5c6a 5815ff08 6676dc10 59002083 j\.u...X..vf. .Y
Wow amazing that we only have one match ... looks like a nice match.
0:000> u 76de1f04
Foo!StoreSchemaInfoToFileW+0x37:
76de1f04 6a5c push 5Ch
76de1f06 ff7508 push dword ptr [ebp+8]
76de1f09 ff155810dc76 call dword ptr [Foo!_imp__wcsrchr (76dc1058)] ; there was no match here – returned NULL
76de1f0f 66832000 and word ptr [eax],0
76de1f13 59 pop ecx
76de1f14 59 pop ecx
76de1f15 6a00 push 0
76de1f17 ff7508 push dword ptr [ebp+8]
76de1f1a 8945fc mov dword ptr [ebp-4],eax
76de1f1d ff155412dc76 call dword ptr [Foo!_imp__CreateDirectoryW (76dc1254)]
76de1f23 85c0 test eax,eax
76de1f25 741b je Foo!StoreSchemaInfoToFileW+0x75 (76de1f42)
76de1f27 8b45fc mov eax,dword ptr [ebp-4]
76de1f2a 66c7005c00 mov word ptr [eax],5Ch
76de1f2f 33c0 xor eax,eax
76de1f31 50 push eax
So now we know the real function – StoreSchemaInfoToFileW and can go examine it, perhaps change it to check the return.
I've used this same method a number of times - no symbols ( some of our reskit tools ) or times like this where there is just the watson log.
Maybe it will help someone figure out something similar .. or maybe there was an easier way for me to do this?
spatdsg
Comments
Anonymous
November 07, 2007
I know that one of the Bugslayer columns had a program where you gave it a dll or exe to load and a crash offset and a load address and it figured out the symbol for you. I've used it before, can't seem to remember the name now though.Anonymous
November 07, 2007
You can save yourself all this trouble by creating and using map files.Anonymous
November 08, 2007
yes - but the whole point of the post is what to do when you lack symbols or map files.. and only have a few pieces of assembly to go from.