Debugging: When Recursive NDRs Bring the FSC to It's Knees And Causes Queuing
A while back (circa July 2015), we had alerting for queuing for a Dedicated customer. When we went and looked, the messages were just sitting in the Submission queue.
We dumped the EdgeTransport process and I cracked it open. When I looked at the dump, only 4 threads were at or above 30 frames and those threads had the same stack[s] (redacted for readability):
# Child-SP RetAddr Call Site
00 00000000`3587d298 000007fe`ff2e74f6 ntdll!ZwAlpcSendWaitReceivePort+0xa
01 00000000`3587d2a0 000007fe`ff2e1c32 rpcrt4!LRPC_CCALL::SendReceive+0x156
02 00000000`3587d360 000007fe`fdf228c0 rpcrt4!I_RpcSendReceive+0x42
03 00000000`3587d390 000007fe`fdf2282f ole32!ThreadSendReceive+0x40
04 00000000`3587d3e0 000007fe`fdf2265b ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xa3
05 00000000`3587d480 000007fe`fddddaaa ole32!CRpcChannelBuffer::SendReceive2+0x11b
06 00000000`3587d640 000007fe`fdddda0c ole32!CAptRpcChnl::SendReceive+0x52
07 00000000`3587d710 000007fe`fdf2205d ole32!CCtxComChnl::SendReceive+0x68
08 00000000`3587d7c0 000007fe`ff38fd61 ole32!NdrExtpProxySendReceive+0x45
09 00000000`3587d7f0 000007fe`fdf1f82f rpcrt4!NdrpClientCall2+0x9ea
0a 00000000`3587df60 000007fe`fdddd8a2 ole32!ObjectStublessClient+0x1ad
0b 00000000`3587e2f0 000007fe`f839a8b7 ole32!ObjectStubless+0x42
0c 00000000`3587e340 000007ff`0042901d mscorwks!DoCLRToCOMCall+0x177
0d 00000000`3587e3f0 000007ff`00424efc DomainBoundILStubClass.IL_STUB(System.Object, UInt32, UInt32 ByRef, UInt32 ByRef, System.Array ByRef)+0x15d
0e 00000000`3587e540 000007ff`001af3ec FSEAgent_8_0_685!Antigen.Scanner.Scan()+0x45c
0f 00000000`3587e640 000007fe`f74f2bbb FSEAgent_8_0_685!Antigen.Scanner.ScannerThread()+0x13c
10 00000000`3587e6e0 000007fe`f758aadd mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x9b
11 00000000`3587e730 000007fe`f839c6a2 mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()+0x4d
12 00000000`3587e780 000007fe`f825ff03 mscorwks!CallDescrWorker+0x82
13 00000000`3587e7d0 000007fe`f877a291 mscorwks!CallDescrWorkerWithHandler+0xd3
14 00000000`3587e870 000007fe`f81c37b9 mscorwks!MethodDesc::CallDescr+0x2b1
15 00000000`3587eab0 000007fe`f831d298 mscorwks!ThreadNative::KickOffThread_Worker+0x191
16 00000000`3587edd0 000007fe`f8230945 mscorwks!ManagedThreadBase_DispatchInner+0x2c
17 00000000`3587ee20 000007fe`f81fe52d mscorwks!ManagedThreadBase_DispatchMiddle+0x9d
18 00000000`3587eef0 000007fe`f835a2b5 mscorwks!ManagedThreadBase_DispatchOuter+0x31
19 00000000`3587ef30 000007fe`f832825b mscorwks!ManagedThreadBase_FullTransitionWithAD+0x35
1a 00000000`3587ef90 000007fe`f8320134 mscorwks!ThreadNative::KickOffThread+0xd3
1b 00000000`3587f070 00000000`779959dd mscorwks!Thread::intermediateThreadProc+0x78
1c 00000000`3587fc40 00000000`77aca651 kernel32!BaseThreadInitThunk+0xd
1d 00000000`3587fc70 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
All of those aforementioned threads were passing to COM:
!findstack mscorwks!DoCLRToCOMCall
Thread 064, 1 frame(s) match
* 12 000000003587e3e0 000007ff0042901d mscorwks!DoCLRToCOMCall+0x177
Thread 066, 1 frame(s) match
* 12 0000000051cfe240 000007ff0042901d mscorwks!DoCLRToCOMCall+0x177
Thread 068, 1 frame(s) match
* 12 0000000051dfddb0 000007ff0042901d mscorwks!DoCLRToCOMCall+0x177
Thread 070, 1 frame(s) match
* 12 000000005205e250 000007ff0042901d mscorwks!DoCLRToCOMCall+0x177
And the messages that were being processed were NDRs that have been around the block or two (sorry for the wall of text):
0:064> !mdt 00000000e074c8b0
00000000e074c8b0 (System.String) Length=12515, String="Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undelivered Mail Returned to Sender"
0:070> !mdt 0000000100776d88
0000000100776d88 (System.String) Length=11719, String="Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Delayed Mail (still being retried)"
The message details (in redacted form) were:
0:064> !mdt 0000000081097f30
0000000081097f30 (Antigen.AntigenMessage)
m_folder:00000000810a01f0 (System.String) Length=36, String="C:\forefront\data\\Archive\Internal\"
m_fileObject:0x250e0 (System.UInt32)
m_bcc:00000000e0754c50 (System.String) Length=43, String="<recipient>;"
m_flags:0x0 (System.Int32)
m_smtpSenderAddress:00000000907eb738 (System.String) Length=21, String=""Sender""
m_smtpExternalIP:NULL (System.String)
m_onlyTreatIfPurge:0x0 (System.UInt32)
m_isSystemMessage:0x1 (System.UInt32)
m_smtpExternalHostName:NULL (System.String)
m_MessageDateTimeReceived:00000000e0754990 (System.String) Length=20, String="2015-07-21T08:41:46Z"
m_MessageDateTimeSent:00000000e07549d8 (System.String) Length=20, String="2015-07-21T08:26:23Z"
m_EHSBypassVirusScan:0x0 (System.UInt32)
m_PremiumEngineSet:0x0 (System.UInt32)
m_IsTransportDecrypted:0x0 (System.UInt32)
m_ReplayHeaders:00000000810a0428 (System.String) Length=172, String="X-Receiver: <recipient> ORCPT=rfc822;<recipient>
X-Sender: <sender>
X-CreatedBy: MSExchange12"
When I checked message tracking, this was a partial indicator of what introduced the problem:
$msgTrk = @()
$hubz = Get-TransportServer
foreach($hub in $hubz)
{
$msgTrk += Get-MessageTrackingLog -Server $hub.Name -Recipients <recipient> -ResultSize Unlimited
}
$msgTrk.Count
1491474
In the words of Strongbad, "That is not a small number! That's a very big number! "
This was a limitation in the Forefront hook in Transport that caused it to consume a lot of resources on the heap, which was exacerbated by a mail-storm from the customer's on-premises.