Some Multi-threaded terminal server applications crashes in Win 2008 R2
In Win2008 R2 some Multi-threaded terminal server application may crashes with access violation in the test eax, eax cpu instruction with following symptoms. This issue is very intermittent.
1.
2. You may find following 2 threads,
- Executing a test instruction and causing AV.
- Trying to change the protection of exactly same code memory page which the first thread is causing access violation ( Not waiting ).
- If DEP for the process is turned off application is working fine.
Following are the analysis details,
The stored exception information can be accessed via .ecxr.
(428.b50): Access violation - code c0000005 (first/second chance not available)
eax=00000000 ebx=7740f85c ecx=74e92dd9 edx=00000000 esi=00000000 edi=00000001
eip=741c17cd esp=0018a4dc ebp=0018a508 iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
mswsock!SockWaitForSingleObject+0x3a:
741c17cd 85c0 test eax,eax
0:000> k100
ChildEBP RetAddr
0018a508 741c678c mswsock!SockWaitForSingleObject+0x3a
0018a5f4 75f04a20 mswsock!WSPSelect+0x3a6
*** ERROR: Symbol file could not be found. Defaulted to export symbols for DvZediRimac002.dll -
0018a674 4b6f447d ws2_32!select+0x494
WARNING: Stack unwind information not available. Following frames may be wrong.
0018e598 4b6e5a80 ThirdParty!ThirdParty +0xac
0018e6ac 774bfd6e ThirdParty!ThirdParty +0x2fc
0018e698 00000000 ntdll!RtlpValidateHeap+0x20
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
Debugger CompCtrlDb Connection::Open failed 80004005
Debugger CompCtrlDb Connection::Open failed 80004005
Debugger CompCtrlDb Connection::Open failed 80004005
Debugger Dbgportaldb Connection::Open failed 80040e4d
Database Dbgportaldb not connected
FAULTING_IP:
mswsock!SockWaitForSingleObject+3a
741c17cd 85c0 test eax,eax
EXCEPTION_RECORD: ffffffff -- (.exr ffffffffffffffff)
ExceptionAddress: 741c17cd (mswsock!SockWaitForSingleObject+0x0000003a)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000008
Parameter[1]: 741c17cd
Attempt to execute non-executable address 741c17cd
PROCESS_NAME: KAREWE.exe
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
WRITE_ADDRESS: 741c17cd
BUGCHECK_STR: ACCESS_VIOLATION
LAST_CONTROL_TRANSFER: from 741c678c to 741c17cd
STACK_TEXT:
0018a508 741c678c 0000025c 00000264 00000001 mswsock!SockWaitForSingleObject+0x3a
0018a5f4 75f04a20 00000000 0018a6a0 00000000 mswsock!WSPSelect+0x3a6
0018a674 4b6f447d 00000000 0018a6a0 00000000 ws2_32!select+0x494
WARNING: Stack unwind information not available. Following frames may be wrong.
0018e598 4b6e5a80 0018e6fc 00000200 00000001 ThirdParty!ThirdParty +0xac
0018e6ac 774bfd6e 00000000 00000044 00310150 ThirdParty!ThirdParty +0x2fc
0018e698 00000000 02c30ac8 23040027 0000000d ntdll!RtlpValidateHeap+0x20
STACK_COMMAND: ~0s; .ecxr ; kb
PRIMARY_PROBLEM_CLASS: SOFTWARE_NX_FAULT_FALSE_POSITIVE
DEFAULT_BUCKET_ID: SOFTWARE_NX_FAULT_FALSE_POSITIVE
FAULTING_THREAD: 00000b50
FOLLOWUP_IP:
mswsock!SockWaitForSingleObject+3a
741c17cd 85c0 test eax,eax
SYMBOL_STACK_INDEX: 0
FOLLOWUP_NAME: wsstress
MODULE_NAME: mswsock
DEBUG_FLR_IMAGE_TIMESTAMP: 4a5bda77
SYMBOL_NAME: mswsock!SockWaitForSingleObject+3a
IMAGE_NAME: mswsock.dll
FAILURE_BUCKET_ID: ACCESS_VIOLATION_mswsock!SockWaitForSingleObject+3a
BUCKET_ID: ACCESS_VIOLATION_mswsock!SockWaitForSingleObject+3a
Followup: wsstress
---------
· The application is failing at
741c17cd 85c0 test eax,eax
due to access violation.
· I checked if any dll is tampered with
0:000> !for_each_module !chkimg @#ModuleName
found that the dll under question(mswsock) is not tampered.
- Above instruction (test) don't access any memory location other than where EIP points.
- So for the above instruction to generate an AV, the EIP should point to an address in a non executable page of memory.
- Again from !address command we can see that this a code region and it is from a mapped dll which should ideally an executable page.
0:000> !address eip
TEB 7efdd000 in range 7efdb000 7efde000
TEB 7efda000 in range 7efd8000 7efdb000
ProcessParametrs 003213f0 in range 00320000 0038f000
Environment 00320810 in range 00320000 0038f000
741c0000 : 741c1000 - 00001000
Type 01000000 MEM_IMAGE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageImage
FullPath C:\Windows\System32\mswsock.dll
On further debugging I found the following as well, which should be true in at least some cases of execution.
Only two threads in the application,
Ø Executing the test instruction and causing AV.
Ø Trying to change the protection of exactly same code memory page which the first thread is causing access violation.
0:000> ~
. 0 Id: 428.b50 Suspend: 1 Teb: 7efdd000 Unfrozen
1 Id: 428.900 Suspend: 1 Teb: 7efda000 Unfrozen
0:000> ~1s
eax=00000000 ebx=80000000 ecx=00000000 edx=00000000 esi=741c1098 edi=741c0000
eip=7740ffea esp=030afacc ebp=030afb08 iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
ntdll!NtProtectVirtualMemory+0x12:
7740ffea 83c404 add esp,4
0:001> k100
ChildEBP RetAddr
030afacc 7416aa2c ntdll!NtProtectVirtualMemory+0x12
030afb08 7416ac22 TSAPPCMP!TsRedirectRegisteredImage+0xcf
030afb58 7416af58 TSAPPCMP!TsWalkProcessDlls+0xf5
030afb8c 4b740f0c TSAPPCMP!TLoadLibraryA+0x3a
WARNING: Stack unwind information not available. Following frames may be wrong.
030afbe8 4b73f737 ThirdParty!ThirdParty +0x38a9
030afc84 774236fa ThirdParty!ThirdParty +0x20d4
030afd78 4b6b83c3 ntdll!RtlpFreeHeap+0xb7a
030afde0 774236fa ThirdParty!ThirdParty +0x9b
030afddc 50000063 ntdll!RtlpFreeHeap+0xb7a
030afec4 774236fa 0x50000063
030affd4 77429d45 ntdll!RtlpFreeHeap+0xb7a
030affec 00000000 ntdll!_RtlUserThreadStart+0x1b
Looking at that stack based on following link and trying to sleuth the parameters, ( NtProtectVirtualMemory has FPO and is written in assembly so normal methods don't work )
NTSYSAPI
NTSTATUS
NTAPI
NtProtectVirtualMemory(
IN HANDLE ProcessHandle,
IN OUT PVOID *BaseAddress,
IN OUT PULONG NumberOfBytesToProtect,
IN ULONG NewAccessProtection,
OUT PULONG OldAccessProtection );
Picking up the function arguments from stack
0:001> dds esp
030afacc 7740ffea ntdll!NtProtectVirtualMemory+0x12
030afad0 7416aa2c TSAPPCMP!TsRedirectRegisteredImage+0xcf
030afad4 ffffffff
030afad8 030afafc
030afadc 030afb00
030afae0 00000004
030afae4 030afaf8
030afae8 774f020c ntdll!PebLdr+0xc
030afaec 0038af48
030afaf0 0038be08
030afaf4 00000004
030afaf8 00000020
030afafc 741c1098 mswsock!_imp__NtOpenKey
030afb00 00000004
030afb04 741c0000 mswsock!_imp__OutputDebugStringA <PERF> (mswsock+0x0)
030afb08 030afb58
030afb0c 7416ac22 TSAPPCMP!TsWalkProcessDlls+0xf5
030afb10 741e4818 mswsock!DNSAPI_NULL_THUNK_DATA_DLB+0x8
030afb14 00000000
030afb18 00000001
030afb1c 3c144767
030afb20 00000000
030afb24 4b7a203c DvZediRimac002!CWatchedProxyObj::`vftable'+0x30280
030afb28 00000000
030afb2c 001a0018
030afb30 74162684 TSAPPCMP!`string'
030afb34 030afb54
030afb38 741c00d8 mswsock!_imp__OutputDebugStringA <PERF> (mswsock+0xd8)
030afb3c 0038be08
030afb40 030afb1c
030afb44 00000000
030afb48 030afb7c
Unassembleing the call to NtProtectVirtualMemory.
0:001> ub 7416aa2c L10
TSAPPCMP!TsRedirectRegisteredImage+0xaa [d:\w7rtm\termsrv\tsappcmp\register.c @ 1368]:
7416aa07 59 pop ecx
7416aa08 59 pop ecx
7416aa09 e9c2000000 jmp TSAPPCMP!TsRedirectRegisteredImage+0x173 (7416aad0)
7416aa0e 6a04 push 4
7416aa10 58 pop eax
7416aa11 8d4df0 lea ecx,[ebp-10h]
7416aa14 51 push ecx
7416aa15 50 push eax
7416aa16 8945f8 mov dword ptr [ebp-8],eax
7416aa19 8d45f8 lea eax,[ebp-8]
7416aa1c 50 push eax
7416aa1d 8d45f4 lea eax,[ebp-0Ch]
7416aa20 50 push eax
7416aa21 6aff push 0FFFFFFFFh
7416aa23 8975f4 mov dword ptr [ebp-0Ch],esi
7416aa26 ff151c131674 call dword ptr [TSAPPCMP!_imp__NtProtectVirtualMemory (7416131c)]
What address is trying to change the protection.
0:001> dc 030afafc
030afafc 741c1098 00000004 741c0000 030afb58 ...t.......tX...
030afb0c 7416ac22 741e4818 00000000 00000001 "..t.H.t........
030afb1c 3c144767 00000000 4b7a203c 00000000 gG.<....< zK....
030afb2c 001a0018 74162684 030afb54 741c00d8 .....&.tT......t
030afb3c 0038be08 030afb1c 00000000 030afb7c ..8.........|...
030afb4c 7416e80a 4b084bdf 00000000 030afb8c ...t.K.K........
030afb5c 7416af58 3c1447b3 00000000 4b7a203c X..t.G.<....< zK
030afb6c 00000000 611a0000 030afb60 7499fa2e .......a`......t
Location being changed by NtProtectVirtualMemory.
0:001> !address 741c1098
TEB 7efdd000 in range 7efdb000 7efde000
TEB 7efda000 in range 7efd8000 7efdb000
ProcessParametrs 003213f0 in range 00320000 0038f000
Environment 00320810 in range 00320000 0038f000
741c0000 : 741c1000 - 00001000
Type 01000000 MEM_IMAGE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageImage
FullPath C:\Windows\System32\mswsock.dll
Failing EIP in the other thread.
0:001> !address 741c17cd
TEB 7efdd000 in range 7efdb000 7efde000
TEB 7efda000 in range 7efd8000 7efdb000
ProcessParametrs 003213f0 in range 00320000 0038f000
Environment 00320810 in range 00320000 0038f000
741c0000 : 741c1000 - 00001000
Type 01000000 MEM_IMAGE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageImage
FullPath C:\Windows\System32\mswsock.dll
Conclusion
This is an issue in TSAPPCMP.dll
Until the fix is release if you happen to run in to this issue following is the solution.
Set the following registry value:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server]
to
"IAT"=dword:00000001