Checking a dump file for WCF throttles
The ServiceModel performance counters can tell you how many concurrent calls, instances, and sessions you have. In .Net 4.0, more counters were added to tell you how close you are to the throttles for each of those categories. So if you're trying to performance tune an application, those performance counters would be the way to go. But if you're trying to look at something after-the-fact, you may only have a dump file. Here are some tricks for examining the memory of that dump file to find out if you're hitting your throttles.
The first thing you need is the dump file. This may have been captured by someone else or you may have to get one yourself. I will not cover how to get one of those. There are plenty of resources on how to do that. You can also attach to a live process, it doesn't have to be dump file. The second thing to obtain is windbg. Quickest way to get this is to go windbg.org. There you will find download links for windbg, the path for the Microsoft public symbol server, and many helpful links including a handy cheat sheet for crash dump analysis.
Assuming windbg is setup, we just need to open the crash dump file:
windbg.exe -z MyCrashDump.dmp
You'll see the following familiar windbg screen:
If windbg isn't familiar, that's not a problem for the purposes of this blog. The left side is the scratch pad and you can throw whatever text you want in there. It's more convenient than having notepad open but it does take some real estate. The right side is divided into three sections. For the work here, we only need to use the middle section for entering commands. The other two can be closed if you want.
The first thing to do is setup the symbol path. Windbg.org has the Microsoft public symbol server path, so we'll just set that by typing in the following command:
.sympath srv*E:\SymCache*https://msdl.microsoft.com/download/symbols
I use E:\SymCache as my symbol cache. Be sure to adjust this to wherever you want your symbol cache to be. With that done, reload the symbols by typing:
.reload
Next, load the SOS module which will help you debug managed code. If the process was using .Net 3.5 or earlier, type:
.loadby sos mscorwks
If the process was using .Net 4.0 (or later), type:
.loadby sos clr
At any time after this, you can type !help to see a list of SOS commands. If you've ever read Tess's blog, then you're probably aware of the !analyze command. We won't be using that here since we're after specific WCF information.
Next, let's find our WCF throttles by searching for FlowThrottle in the heap:
!dumpheap -type FlowThrottle -stat
Be careful that the capitalization is correct as this is a case-sensitive search. This should produce output similar to the following:
Statistics: MT Count TotalSize Class Name000007feee769c00 3 192 System.ServiceModel.Dispatcher.FlowThrottleTotal 3 objects
There should be three throttles per service. What this gives us is the MT or MethodTable of the FlowThrottle class. If you used a value for !dumpheap -type like Throttle or Queue, you might get a lot of types in the heap that have that in their class name. The MT will point you to a specific class. In this case, we can use MT to show just the objects were interested in. So, we'll dump the heap again but have it show only the FlowThrottle objects:
0:000> !dumpheap -mt 000007feee769c00 ------------------------------Heap 0 Address MT Size00000002af96ea50 000007feee769c00 64 00000002af96eb80 000007feee769c00 64 00000002afe2c710 000007feee769c00 64 total 3 objectsStatistics: MT Count TotalSize Class Name000007feee769c00 3 192 System.ServiceModel.Dispatcher.FlowThrottleTotal 3 objects
We could try to dump each individual object one-at-a-time by grabbing the address and using the !do (shorthand for !dumpobj) command:
0:000> !do 00000002af96ea50 Name: System.ServiceModel.Dispatcher.FlowThrottleMethodTable: 000007feee769c00EEClass: 000007feee12c950Size: 64(0x40) bytes (C:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)Fields: MT Field Offset Type VT Attr Value Name000007fef7865f00 4003569 30 System.Int32 1 instance 512 capacity000007fef7865f00 400356a 34 System.Int32 1 instance 96 count000007fef785e580 400356b 8 System.Object 0 instance 00000002af96ea90 mutex000007fef78a3540 400356c 10 ...ding.WaitCallback 0 instance 00000002af96ea10 release0000000000000000 400356d 18 0 instance 00000002af96eaa8 waiters000007fef785ec90 400356e 20 System.String 0 instance 00000002af96e930 propertyName000007fef785ec90 400356f 28 System.String 0 instance 00000002af96e9d0 configName
Here I can see that this throttle has a capacity of 512 and a count of 96. So we're not hitting this throttle. But I'd rather just print out all these objects at one time. To do this, we'll first look at the output if we add -short to our !dumpheap command:
0:000> !dumpheap -mt 000007feee769c00 -short00000002af96ea50 00000002af96eb80 00000002afe2c710 ------------------------------
This dumps out the object addresses only. Unfortunately it also sticks that line of dashes at the end but we can live with that. We can use a for loop in windbg to dump each object's contents in one shot:
0:000> .foreach (myobj {!dumpheap -mt 000007feee769c00 -short}) {!do ${myobj}} Name: System.ServiceModel.Dispatcher.FlowThrottleMethodTable: 000007feee769c00EEClass: 000007feee12c950Size: 64(0x40) bytes (C:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)Fields: MT Field Offset Type VT Attr Value Name000007fef7865f00 4003569 30 System.Int32 1 instance 512 capacity000007fef7865f00 400356a 34 System.Int32 1 instance 96 count000007fef785e580 400356b 8 System.Object 0 instance 00000002af96ea90 mutex000007fef78a3540 400356c 10 ...ding.WaitCallback 0 instance 00000002af96ea10 release0000000000000000 400356d 18 0 instance 00000002af96eaa8 waiters000007fef785ec90 400356e 20 System.String 0 instance 00000002af96e930 propertyName000007fef785ec90 400356f 28 System.String 0 instance 00000002af96e9d0 configNameName: System.ServiceModel.Dispatcher.FlowThrottleMethodTable: 000007feee769c00EEClass: 000007feee12c950Size: 64(0x40) bytes (C:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)Fields: MT Field Offset Type VT Attr Value Name000007fef7865f00 4003569 30 System.Int32 1 instance 3200 capacity000007fef7865f00 400356a 34 System.Int32 1 instance 0 count000007fef785e580 400356b 8 System.Object 0 instance 00000002af96ebc0 mutex000007fef78a3540 400356c 10 ...ding.WaitCallback 0 instance 00000002af96eb40 release0000000000000000 400356d 18 0 instance 00000002af96ebd8 waiters000007fef785ec90 400356e 20 System.String 0 instance 00000002af96e970 propertyName000007fef785ec90 400356f 28 System.String 0 instance 00000002af96eaf8 configNameName: System.ServiceModel.Dispatcher.FlowThrottleMethodTable: 000007feee769c00EEClass: 000007feee12c950Size: 64(0x40) bytes (C:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)Fields: MT Field Offset Type VT Attr Value Name000007fef7865f00 4003569 30 System.Int32 1 instance 3712 capacity000007fef7865f00 400356a 34 System.Int32 1 instance 96 count000007fef785e580 400356b 8 System.Object 0 instance 00000002afe2c750 mutex000007fef78a3540 400356c 10 ...ding.WaitCallback 0 instance 00000002afe2c6d0 release0000000000000000 400356d 18 0 instance 00000002afe2c768 waiters000007fef785ec90 400356e 20 System.String 0 instance 00000002afe2c688 propertyName000007fef785ec90 400356f 28 System.String 0 instance 00000002af98a030 configNameUnknown option: ------------------------------
So now we can see each object. But I don't know which throttle is which. There is a field called propertyName that would tell us the throttle's name. Let's dump the last one:
0:000> !do 00000002afe2c688 Name: System.StringMethodTable: 000007fef785ec90EEClass: 000007fef746b038Size: 70(0x46) bytes (C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)String: MaxConcurrentInstancesFields: MT Field Offset Type VT Attr Value Name000007fef7865f00 4000096 8 System.Int32 1 instance 23 m_arrayLength000007fef7865f00 4000097 c System.Int32 1 instance 22 m_stringLength000007fef78606d8 4000098 10 System.Char 1 instance 4d m_firstChar000007fef785ec90 4000099 20 System.String 0 shared static Empty >> Domain:Value 00000000014f4860:000000029f940370 000000000382feb0:000000029f940370 <<000007fef7860588 400009a 28 System.Char[] 0 shared static WhitespaceChars >> Domain:Value 00000000014f4860:000000029f940ac0 000000000382feb0:000000029f9489e8 <<
We can see that the last throttle is for max concurrent instances. But there's a lot of extra garbage around here. And if we can loop through each object and call !do on it, what else can we do? Here's one helpful command I came up with that puts the property name, count, and capacity of each throttle in one package:
0:000> .foreach (myobj {!dumpheap -mt 000007feee769c00 -short}) {!do -nofields poi(${myobj}+20);dd ${myobj}+34 L1;dd ${myobj}+30 L1}
Name: System.String
MethodTable: 000007fef785ec90
EEClass: 000007fef746b038
Size: 62(0x3e) bytes
(C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: MaxConcurrentCalls
00000002`af96ea84 00000060
00000002`af96ea80 00000200
Name: System.String
MethodTable: 000007fef785ec90
EEClass: 000007fef746b038
Size: 68(0x44) bytes
(C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: MaxConcurrentSessions
00000002`af96ebb4 00000000
00000002`af96ebb0 00000c80
Name: System.String
MethodTable: 000007fef785ec90
EEClass: 000007fef746b038
Size: 70(0x46) bytes
(C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: MaxConcurrentInstances
00000002`afe2c744 00000060
00000002`afe2c740 00000e80
Invalid parameter -nofields poi(------------------------------+20)
00000000`00000034 ????????
00000000`00000030 ????????
Notice the annoying dashed line at the end of the !dumpheap still gives us grief. Eventually you just stop seeing it. Also, note that the numbers are printed in hex. Not quite as nice as the !do output, but it serves the purpose. Breaking down the .foreach statement above, you can see that instead of doing !do ${myobj} , I run three separate commands per loop iteration (separated by semicolons). In each of these commands I add ( ${myobj}+offset) the offset for the field, which I got from the !do of one of the objects. The first command uses poi because we need that to get SOS operations to work. POI stands for point of indirection and is kind of like dereferencing a pointer. The other two commands print DWORD values (4-bytes) and don't need POI. These are native debugger commands and not part of an extension library so they don't have the leading ' ! '.
Comments
Anonymous
February 01, 2011
Good one. I would probably write the same script something like this .foreach (myobj {!dumpheap -mt 000007feee769c00 -short}) {.printf "%mu ", poi(${myobj}+20)+c;.printf "%p", poi(${myobj}+34);.printf "%p n", poi(${myobj}+30)} Which would give the output like this MaxConcurrentInstances 0000000000000017 0000001700000200 MaxConcurrentSessions 000000000000000a 0000000a00000200 MaxConcurrentInstances 000000000000002b 0000002b00000200 This will just avoid all the noise.Anonymous
February 01, 2011
Good tip Naveen! I think you want .printf "%mu ", poi(${myobj}+20)+10 though. +10 goes to the first char. Thanks!Anonymous
February 01, 2011
No I was doing it on x64 that's the reason for .printf "%mu ", poi(${myobj}+20)+c I have a platform independent dumpstring script as part of scripts j $ptrsize = 8 'aS !ds .printf "%mu n", c+';'aS !ds .printf "%mu n", 10+' So I could use something like this without being worried about x86/x64 0:000> !ds 00000000023620b8 MaxConcurrentInstancesAnonymous
February 01, 2011
No I was doing it on x64 that's the reason for .printf "%mu ", poi(${myobj}+20)+c I have a platform independent dumpstring script as part of scripts j $ptrsize = 8 'aS !ds .printf "%mu n", c+';'aS !ds .printf "%mu n", 10+' So I could use something like this without being worried about x86/x64 0:000> !ds 00000000023620b8 MaxConcurrentInstancesAnonymous
February 01, 2011
Cool! I'll be able to use this in some of my debugging work.