Debugging error System.InvalidOperationException Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool.
Disclaimer: Most of the content for this post come from an amazing blog post by Remi Lemarchand, which was the only source I could find on how to troubleshoot this issue, I am posting it here with some small modifications to fit a different scenario where we were not clear on which pool was having the leaking connections
We started with several operations failing with this error
CLR exception type: System.InvalidOperationException
"Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached."
The first step was to collect a memory dump, and start analyzing it, first we need to find all connection pools
0:166> !Name2EE System.Data.dll System.Data.ProviderBase.DbConnectionPool
Module: 00007ffbc78f4520
Assembly: System.Data.dll
Token: 0000000002000229
MethodTable: 00007ffbc8603f28
EEClass: 00007ffbc85e7a88
Name: System.Data.ProviderBase.DbConnectionPool
Then dump information about all connection pools using the MT we got from the previous step
0:166> .foreach (foo {!DumpHeap -short -MT 00007ffbc8603f28}) {.echo foo; !do foo}
0000003c7218fd38
Name: System.Data.ProviderBase.DbConnectionPool
MethodTable: 00007ffbc8603f28
EEClass: 00007ffbc85e7a88
Size: 168(0xa8) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffbc6d9f6f0 4000e73 80 System.Int32 1 instance 170000 _cleanupWait
00007ffbc860bbb0 4000e74 8 ...ctionPoolIdentity 0 instance 0000003cf1fa4508 _identity
00007ffbc8602b90 4000e75 10 ...ConnectionFactory 0 instance 0000003f71fcb8b0 _connectionFactory
00007ffbc86025c8 4000e76 18 ...nnectionPoolGroup 0 instance 0000003c7218f780 _connectionPoolGroup
00007ffbc860b500 4000e77 20 ...nPoolGroupOptions 0 instance 0000003c7218f758 _connectionPoolGroupOptions
00007ffbc862e0f8 4000e78 28 ...nPoolProviderInfo 0 instance 0000000000000000 _connectionPoolProviderInfo
00007ffbc8603dd0 4000e79 84 System.Int32 1 instance 1 _state
00007ffbc862e188 4000e7a 30 ...al, System.Data]] 0 instance 0000003c7218fde0 _stackOld
00007ffbc862e188 4000e7b 38 ...al, System.Data]] 0 instance 0000003c7218fe00 _stackNew
00007ffbc862fb38 4000e7c 40 ...on, System.Data]] 0 instance 0000003c7218fe20 _pendingOpens
00007ffbc6d9f6f0 4000e7d 88 System.Int32 1 instance 0 _pendingOpensWaiting
00007ffbc6f0a188 4000e7e 48 ...ding.WaitCallback 0 instance 0000003c72190510 _poolCreateRequest
00007ffbc6d9f6f0 4000e7f 8c System.Int32 1 instance 0 _waitCount
00007ffbc86504c0 4000e80 50 ...l+PoolWaitHandles 0 instance 0000003c7218ffe8 _waitHandles
00007ffbc6de5290 4000e81 58 System.Exception 0 instance 0000003ff2c89558 _resError
00007ffbc6d9c620 4000e82 9c System.Boolean 1 instance 0 _errorOccurred
00007ffbc6d9f6f0 4000e83 90 System.Int32 1 instance 5000 _errorWait
00007ffbc7010048 4000e84 60 ...m.Threading.Timer 0 instance 0000000000000000 _errorTimer
00007ffbc7010048 4000e85 68 ...m.Threading.Timer 0 instance 0000003c72190590 _cleanupTimer
00007ffbc8650c98 4000e86 70 ...tedConnectionPool 0 instance 0000003c72190498 _transactedConnectionPool
00007ffbc8650580 4000e87 78 ...al, System.Data]] 0 instance 0000003c72190130 _objectList
00007ffbc6d9f6f0 4000e88 94 System.Int32 1 instance 4 _totalObjects
00007ffbc6d9f6f0 4000e8a 98 System.Int32 1 instance 3 _objectID
00007ffbc8304be8 4000e72 b98 System.Random 0 static 0000003cf1fa73c0 _random
00007ffbc6d9f6f0 4000e89 a00 System.Int32 1 static 5117 _objectTypeCount
The previous one does not seem interesting, since it has only 4 connections in the pool and the wait count is zero. Moving on to the next pool
0000003c7360fec8
Name: System.Data.ProviderBase.DbConnectionPool
MethodTable: 00007ffbc8603f28
EEClass: 00007ffbc85e7a88
Size: 168(0xa8) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffbc6d9f6f0 4000e73 80 System.Int32 1 instance 140000 _cleanupWait
00007ffbc860bbb0 4000e74 8 ...ctionPoolIdentity 0 instance 0000003cf1fa4508 _identity
00007ffbc8602b90 4000e75 10 ...ConnectionFactory 0 instance 0000003f71fcb8b0 _connectionFactory
00007ffbc86025c8 4000e76 18 ...nnectionPoolGroup 0 instance 0000003f7215b290 _connectionPoolGroup
00007ffbc860b500 4000e77 20 ...nPoolGroupOptions 0 instance 0000003f7215b268 _connectionPoolGroupOptions
00007ffbc862e0f8 4000e78 28 ...nPoolProviderInfo 0 instance 0000000000000000 _connectionPoolProviderInfo
00007ffbc8603dd0 4000e79 84 System.Int32 1 instance 1 _state
00007ffbc862e188 4000e7a 30 ...al, System.Data]] 0 instance 0000003c7360ff70 _stackOld
00007ffbc862e188 4000e7b 38 ...al, System.Data]] 0 instance 0000003c7360ff90 _stackNew
00007ffbc862fb38 4000e7c 40 ...on, System.Data]] 0 instance 0000003c7360ffb0 _pendingOpens
00007ffbc6d9f6f0 4000e7d 88 System.Int32 1 instance 0 _pendingOpensWaiting
00007ffbc6f0a188 4000e7e 48 ...ding.WaitCallback 0 instance 0000003c736106a0 _poolCreateRequest
00007ffbc6d9f6f0 4000e7f 8c System.Int32 1 instance 2 _waitCount
00007ffbc86504c0 4000e80 50 ...l+PoolWaitHandles 0 instance 0000003c73610178 _waitHandles
00007ffbc6de5290 4000e81 58 System.Exception 0 instance 0000000000000000 _resError
00007ffbc6d9c620 4000e82 9c System.Boolean 1 instance 0 _errorOccurred
00007ffbc6d9f6f0 4000e83 90 System.Int32 1 instance 5000 _errorWait
00007ffbc7010048 4000e84 60 ...m.Threading.Timer 0 instance 0000000000000000 _errorTimer
00007ffbc7010048 4000e85 68 ...m.Threading.Timer 0 instance 0000003c73610720 _cleanupTimer
00007ffbc8650c98 4000e86 70 ...tedConnectionPool 0 instance 0000003c73610628 _transactedConnectionPool
00007ffbc8650580 4000e87 78 ...al, System.Data]] 0 instance 0000003c736102c0 _objectList
00007ffbc6d9f6f0 4000e88 94 System.Int32 1 instance 100 _totalObjects
00007ffbc6d9f6f0 4000e8a 98 System.Int32 1 instance 3541 _objectID
00007ffbc8304be8 4000e72 b98 System.Random 0 static 0000003cf1fa73c0 _random
00007ffbc6d9f6f0 4000e89 a00 System.Int32 1 static 5117 _objectTypeCount
This one definitely seems to be it, the number of connections in the pool is 100 which is our limit, and there are 2 objects waiting, so moving on to analyzing with more details based on Remi's instructions
0:166> !do 0000003c7360fec8
Name: System.Data.ProviderBase.DbConnectionPool
MethodTable: 00007ffbc8603f28
EEClass: 00007ffbc85e7a88
Size: 168(0xa8) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffbc6d9f6f0 4000e73 80 System.Int32 1 instance 140000 _cleanupWait
00007ffbc860bbb0 4000e74 8 ...ctionPoolIdentity 0 instance 0000003cf1fa4508 _identity
00007ffbc8602b90 4000e75 10 ...ConnectionFactory 0 instance 0000003f71fcb8b0 _connectionFactory
00007ffbc86025c8 4000e76 18 ...nnectionPoolGroup 0 instance 0000003f7215b290 _connectionPoolGroup
00007ffbc860b500 4000e77 20 ...nPoolGroupOptions 0 instance 0000003f7215b268 _connectionPoolGroupOptions
00007ffbc862e0f8 4000e78 28 ...nPoolProviderInfo 0 instance 0000000000000000 _connectionPoolProviderInfo
00007ffbc8603dd0 4000e79 84 System.Int32 1 instance 1 _state
00007ffbc862e188 4000e7a 30 ...al, System.Data]] 0 instance 0000003c7360ff70 _stackOld
00007ffbc862e188 4000e7b 38 ...al, System.Data]] 0 instance 0000003c7360ff90 _stackNew
00007ffbc862fb38 4000e7c 40 ...on, System.Data]] 0 instance 0000003c7360ffb0 _pendingOpens
00007ffbc6d9f6f0 4000e7d 88 System.Int32 1 instance 0 _pendingOpensWaiting
00007ffbc6f0a188 4000e7e 48 ...ding.WaitCallback 0 instance 0000003c736106a0 _poolCreateRequest
00007ffbc6d9f6f0 4000e7f 8c System.Int32 1 instance 2 _waitCount
00007ffbc86504c0 4000e80 50 ...l+PoolWaitHandles 0 instance 0000003c73610178 _waitHandles
00007ffbc6de5290 4000e81 58 System.Exception 0 instance 0000000000000000 _resError
00007ffbc6d9c620 4000e82 9c System.Boolean 1 instance 0 _errorOccurred
00007ffbc6d9f6f0 4000e83 90 System.Int32 1 instance 5000 _errorWait
00007ffbc7010048 4000e84 60 ...m.Threading.Timer 0 instance 0000000000000000 _errorTimer
00007ffbc7010048 4000e85 68 ...m.Threading.Timer 0 instance 0000003c73610720 _cleanupTimer
00007ffbc8650c98 4000e86 70 ...tedConnectionPool 0 instance 0000003c73610628 _transactedConnectionPool
00007ffbc8650580 4000e87 78 ...al, System.Data]] 0 instance 0000003c736102c0 _objectList <-- This is the list of connections in this connection pool
00007ffbc6d9f6f0 4000e88 94 System.Int32 1 instance 100 _totalObjects
00007ffbc6d9f6f0 4000e8a 98 System.Int32 1 instance 3541 _objectID
00007ffbc8304be8 4000e72 b98 System.Random 0 static 0000003cf1fa73c0 _random
00007ffbc6d9f6f0 4000e89 a00 System.Int32 1 static 5117 _objectTypeCount
Now we can inspect the objectlist
0:166> !do 0000003c736102c0
Name: System.Collections.Generic.List`1[[System.Data.ProviderBase.DbConnectionInternal, System.Data]]
MethodTable: 00007ffbc8650580
EEClass: 00007ffbc6e423b8
Size: 40(0x28) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffbc6d9ba28 4000cd1 8 System.Object[] 0 instance 0000003c736102e8 _items <-- Find the array of items
00007ffbc6d9f6f0 4000cd2 18 System.Int32 1 instance 100 _size
00007ffbc6d9f6f0 4000cd3 1c System.Int32 1 instance 2126 _version
00007ffbc6d96080 4000cd4 10 System.Object 0 instance 0000000000000000 _syncRoot
00007ffbc6d9ba28 4000cd5 8 System.Object[] 0 static <no information>
Then Dump the contents of the array
0:118> !da 0000003c736102e8
Name: System.Data.ProviderBase.DbConnectionInternal[]
MethodTable: 00007ffbc6d9ba28
EEClass: 00007ffbc6d9b980
Size: 832(0x340) bytes
Array: Rank 1, Number of elements 100, Type CLASS
Element Methodtable: 00007ffbc8602030
[0] 0000003ff605c8b8
[1] 0000003f7660b270
[2] 0000003ef6d2e860
[3] 0000003df7e341b0
[4] 0000003ef73b3280
[5] 0000003ef73b5260
[6] 0000003f77ad5c20
[7] 0000003e76deffb8
[8] 0000003cf7894fc0
...
Now You can start inspecting the different items in the array, to see if they all are being used by the same query using Remi's instructions
0:118> !do 0000003ff605c8b8
Name: System.Data.SqlClient.SqlInternalConnectionTds
MethodTable: 00007ffbc86289d8
EEClass: 00007ffbc861d270
Size: 304(0x130) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffbc6d9f6f0 40008bb 38 System.Int32 1 instance 18391 _objectID
00007ffbc6d9c620 40008be 44 System.Boolean 1 instance 0 _allowSetConnectionString
00007ffbc6d9c620 40008bf 45 System.Boolean 1 instance 1 _hidePassword
00007ffbc8601ed8 40008c0 3c System.Int32 1 instance 1 _state
00007ffbc6fca230 40008c1 8 System.WeakReference 0 instance 0000003ff605ca58 _owningObject
00007ffbc8603f28 40008c2 10 ....DbConnectionPool 0 instance 0000003c7360fec8 _connectionPool
00007ffbc8605050 40008c3 18 ...ctionPoolCounters 0 instance 0000003f71fcb8f0 _performanceCounters
00007ffbc86816a0 40008c4 20 ...ferenceCollection 0 instance 0000003ff6061b38 _referenceCollection
00007ffbc6d9f6f0 40008c5 40 System.Int32 1 instance 0 _pooledCount
00007ffbc6d9c620 40008c6 46 System.Boolean 1 instance 0 _connectionIsDoomed
00007ffbc6d9c620 40008c7 47 System.Boolean 1 instance 0 _cannotBePooled
00007ffbc6d9c620 40008c8 48 System.Boolean 1 instance 0 _isInStasis
00007ffbc6fce4f0 40008c9 50 System.DateTime 1 instance 0000003ff605c908 _createTime
00007ffbc862e010 40008ca 28 ...tions.Transaction 0 instance 0000000000000000 _enlistedTransaction
00007ffbc862e010 40008cb 30 ...tions.Transaction 0 instance 0000000000000000 _enlistedTransactionOriginal
00007ffbc6d9f6f0 40008ba 700 System.Int32 1 static 18584 _objectTypeCount
00007ffbc860b308 40008bc 628 ...teChangeEventArgs 0 static 0000003c71fb8d78 StateChangeClosed
00007ffbc860b308 40008bd 630 ...teChangeEventArgs 0 static 0000003c71fb8d90 StateChangeOpen
00007ffbc86019e8 400127f 58 ...lConnectionString 0 instance 0000003f7215acf8 _connectionOptions
00007ffbc6d9c620 4001280 88 System.Boolean 1 instance 0 _isEnlistedInTransaction
00007ffbc6e30eb0 4001281 60 System.Byte[] 0 instance 0000000000000000 _promotedDTCToken
00007ffbc6e30eb0 4001282 68 System.Byte[] 0 instance 0000000000000000 _whereAbouts
00007ffbc6de46f0 4001283 70 System.String 0 instance 0000003ff605da20 <CurrentDatabase>k__BackingField
00007ffbc6de46f0 4001284 78 System.String 0 instance 0000003f7215ae18 <CurrentDataSource>k__BackingField
00007ffbc884a318 4001285 80 ...egatedTransaction 0 instance 0000000000000000 <DelegatedTransaction>k__BackingField
00007ffbc860d1b0 40012a7 90 ...GroupProviderInfo 0 instance 0000003f7215b2d8 _poolGroupProviderInfo
00007ffbc8629c78 40012a8 98 ...lClient.TdsParser 0 instance 0000003ff605d4b8 _parser
00007ffbc865e0e8 40012a9 a0 ...lient.SqlLoginAck 0 instance 0000003ff605daf0 _loginAck
00007ffbc84cb330 40012aa a8 ...ent.SqlCredential 0 instance 0000003cf2193680 _credential
00007ffbc6d9c620 40012ab 89 System.Boolean 1 instance 1 _sessionRecoveryRequested
00007ffbc6d9c620 40012ac 8a System.Boolean 1 instance 1 _sessionRecoveryAcknowledged
00007ffbc86558e0 40012ad b0 ...lient.SessionData 0 instance 0000003ff605ca70 _currentSessionData
00007ffbc86558e0 40012ae b8 ...lient.SessionData 0 instance 0000000000000000 _recoverySessionData
00007ffbc6d9c620 40012af 8b System.Boolean 1 instance 1 _fConnectionOpen
00007ffbc6d9c620 40012b0 110 System.Boolean 1 instance 1 _fResetConnection
00007ffbc6de46f0 40012b1 c0 System.String 0 instance 0000003ff605da20 _originalDatabase
00007ffbc6de46f0 40012b2 c8 System.String 0 instance 0000000000000000 _currentFailoverPartner
00007ffbc6de46f0 40012b3 d0 System.String 0 instance 0000003ff605dac0 _originalLanguage
00007ffbc6de46f0 40012b4 d8 System.String 0 instance 0000003ff605dac0 _currentLanguage
00007ffbc6d9f6f0 40012b5 8c System.Int32 1 instance 8000 _currentPacketSize
00007ffbc6d9f6f0 40012b6 108 System.Int32 1 instance 0 _asyncCommandCount
00007ffbc6de46f0 40012b7 e0 System.String 0 instance 0000003c71fa1450 _instanceName
00007ffbc860bbb0 40012b8 e8 ...ctionPoolIdentity 0 instance 0000000000000000 _identity
00007ffbc8656558 40012b9 f0 ...Tds+SyncAsyncLock 0 instance 0000003ff605c9e8 _parserLock
00007ffbc6d9f6f0 40012ba 10c System.Int32 1 instance -1 _threadIdOwningParserLock
00007ffbc8656970 40012bb f8 ...eoutErrorInternal 0 instance 0000003ff605d3c8 timeoutErrorInternal
00007ffbc6e3fb58 40012bc 118 System.Guid 1 instance 0000003ff605c9d0 _clientConnectionId
00007ffbc8656ea0 40012bd 100 ...lient.RoutingInfo 0 instance 0000000000000000 _routingInfo
Now Dump the _parser member
0:118> !do 0000003ff605d4b8
Name: System.Data.SqlClient.TdsParser
MethodTable: 00007ffbc8629c78
EEClass: 00007ffbc861d2f8
Size: 160(0xa0) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffbc6d9f6f0 400151a 70 System.Int32 1 instance 18394 _objectID
00007ffbc862b328 400151c 8 ...ParserStateObject 0 instance 0000003ff605d558 _physicalStateObj
00007ffbc862b328 400151d 10 ...ParserStateObject 0 instance 0000000000000000 _pMarsPhysicalConObj
00007ffbc86299d0 400151e 74 System.Int32 1 instance 2 _state
00007ffbc6de46f0 400151f 18 System.String 0 instance 0000003f7215ae18 _server
00007ffbc6d9c620 4001520 88 System.Boolean 1 instance 0 _fResetConnection
00007ffbc6d9c620 4001521 89 System.Boolean 1 instance 0 _fPreserveTransaction
00007ffbc865d050 4001522 20 ...ient.SqlCollation 0 instance 0000003ff605daa8 _defaultCollation
00007ffbc6d9f6f0 4001523 78 System.Int32 1 instance 1252 _defaultCodePage
00007ffbc6d9f6f0 4001524 7c System.Int32 1 instance 1033 _defaultLCID
00007ffbc7018568 4001525 28 System.Text.Encoding 0 instance 0000003c71fc23b0 _defaultEncoding
00007ffbc8629b20 4001527 80 System.Int32 1 instance 1 _encryptionOption
00007ffbc860ef98 4001528 30 ...ternalTransaction 0 instance 0000000000000000 _currentTransaction
00007ffbc860ef98 4001529 38 ...ternalTransaction 0 instance 0000000000000000 _pendingTransaction
00007ffbc6de0630 400152a 68 System.Int64 1 instance 0 _retainedTransactionId
00007ffbc6d9f6f0 400152b 84 System.Int32 1 instance 1 _nonTransactedOpenResultCount
00007ffbc86289d8 400152c 40 ...rnalConnectionTds 0 instance 0000003ff605c8b8 _connHandler
00007ffbc6d9c620 400152d 8a System.Boolean 1 instance 0 _fMARS
00007ffbc6d9c620 400152e 8b System.Boolean 1 instance 0 _loginWithFailover
00007ffbc73f0698 400152f 48 ...ng.AutoResetEvent 0 instance 0000000000000000 _resetConnectionEvent
00007ffbc862b4e8 4001530 50 ...ParserSessionPool 0 instance 0000000000000000 _sessionPool
00007ffbc6d9c620 4001531 8c System.Boolean 1 instance 1 _isShiloh
00007ffbc6d9c620 4001532 8d System.Boolean 1 instance 1 _isShilohSP1
00007ffbc6d9c620 4001533 8e System.Boolean 1 instance 1 _isYukon
00007ffbc6d9c620 4001534 8f System.Boolean 1 instance 1 _isKatmai
00007ffbc6d9c620 4001535 90 System.Boolean 1 instance 1 _isDenali
00007ffbc6e30eb0 4001536 58 System.Byte[] 0 instance 0000000000000000 _sniSpnBuffer
00007ffbc86281b0 4001537 60 ...ent.SqlStatistics 0 instance 0000000000000000 _statistics
00007ffbc6d9c620 4001538 91 System.Boolean 1 instance 0 _statisticsIsInTransaction
00007ffbc6d9c620 400153f 92 System.Boolean 1 instance 0 _asyncWrite
00007ffbc6d9f6f0 4001519 a60 System.Int32 1 static 18587 _objectTypeCount
00007ffbc6f0b5d0 400151b 1130 ...eading.Tasks.Task 0 static 0000000000000000 completedTask
00007ffbc8629b20 4001526 a64 System.Int32 1 static 0 _sniSupportedEncryptionOption
00007ffbc6e30eb0 4001539 1138 System.Byte[] 0 static 0000003c71fbec48 s_nicAddress
00007ffbc6d9c620 400153a a6c System.Boolean 1 static 0 s_fSSPILoaded
00007ffbc6d9fe78 400153b a68 System.UInt32 1 static 0 s_maxSSPILength
00007ffbc6e30eb0 400153c 1140 System.Byte[] 0 static 0000003cf1fc24d0 s_longDataHeader
00007ffbc6d96080 400153d 1148 System.Object 0 static 0000003cf1fc2508 s_tdsParserLock
00007ffbc6e30eb0 400153e 1150 System.Byte[] 0 static 0000003cf1fc2520 s_xmlMetadataSubstituteSequence
00007ffbc86575c8 4001540 1158 ...rd, System.Data]] 0 static 0000003cf1fc2588 __tvpEmptyValue
Then dump the _physicalStateObj
0:118> !do 0000003ff605d558
Name: System.Data.SqlClient.TdsParserStateObject
MethodTable: 00007ffbc862b328
EEClass: 00007ffbc861d750
Size: 440(0x1b8) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffbc6d9f6f0 40015ea 130 System.Int32 1 instance 18394 _objectID
00007ffbc8629c78 40015eb 8 ...lClient.TdsParser 0 instance 0000003ff605d4b8 _parser
00007ffbc862a388 40015ec 10 ...lClient.SNIHandle 0 instance 0000003ff605d8b8 _sessionHandle
00007ffbc6fca230 40015ed 18 System.WeakReference 0 instance 0000003ff605d710 _owner
00007ffbc8682960 40015ee 20 ...eader+SharedState 0 instance 0000003c77292e50 _readerState
00007ffbc6d9f6f0 40015ef 134 System.Int32 1 instance 0 _activateCount
00007ffbc6d9f6f0 40015f0 138 System.Int32 1 instance 8 _inputHeaderLen
00007ffbc6d9f6f0 40015f1 13c System.Int32 1 instance 8 _outputHeaderLen
00007ffbc6e30eb0 40015f2 28 System.Byte[] 0 instance 0000003ff605fab8 _outBuff
00007ffbc6d9f6f0 40015f3 140 System.Int32 1 instance 8 _outBytesUsed
00007ffbc6e30eb0 40015f4 30 System.Byte[] 0 instance 0000003ff605db60 _inBuff
00007ffbc6d9f6f0 40015f5 144 System.Int32 1 instance 7823 _inBytesUsed
00007ffbc6d9f6f0 40015f6 148 System.Int32 1 instance 7841 _inBytesRead
00007ffbc6d9f6f0 40015f7 14c System.Int32 1 instance 18 _inBytesPacket
00007ffbc6d9e008 40015f8 17c System.Byte 1 instance 3 _outputMessageType
00007ffbc6d9e008 40015f9 17d System.Byte 1 instance 1 _messageStatus
00007ffbc6d9e008 40015fa 17e System.Byte 1 instance 1 _outputPacketNumber
00007ffbc6d9c620 40015fb 17f System.Boolean 1 instance 1 _pendingData
00007ffbc6d9c620 40015fc 180 System.Boolean 1 instance 0 _fResetEventOwned
00007ffbc6d9c620 40015fd 181 System.Boolean 1 instance 0 _fResetConnectionSent
00007ffbc6d9c620 40015fe 182 System.Boolean 1 instance 0 _errorTokenReceived
00007ffbc6d9c620 40015ff 183 System.Boolean 1 instance 0 _bulkCopyOpperationInProgress
00007ffbc6d9c620 4001600 184 System.Boolean 1 instance 0 _bulkCopyWriteTimeout
00007ffbc86595d8 4001601 38 ...lClient.SNIPacket 0 instance 0000003ff6061b18 _sniPacket
00007ffbc86595d8 4001602 40 ...lClient.SNIPacket 0 instance 0000003f76cc7ae8 _sniAsyncAttnPacket
00007ffbc8659500 4001603 48 ....WritePacketCache 0 instance 0000003ff605d728 _writePacketCache
00007ffbc8659698 4001604 50 ...et, System.Data]] 0 instance 0000003ff605d770 _pendingWritePackets
00007ffbc6d96080 4001605 58 System.Object 0 instance 0000003ff605d7c0 _writePacketLockObject
00007ffbc6f9cf88 4001606 198 ...Services.GCHandle 1 instance 0000003ff605d6f0 _gcHandle
00007ffbc6d9f6f0 4001607 150 System.Int32 1 instance 1 _pendingCallbacks
00007ffbc6de0630 4001608 110 System.Int64 1 instance 3600000 _timeoutMilliseconds
00007ffbc6de0630 4001609 118 System.Int64 1 instance 0 _timeoutTime
00007ffbc6d9c620 400160a 185 System.Boolean 1 instance 0 _attentionSent
00007ffbc6d9c620 400160b 186 System.Boolean 1 instance 0 _attentionReceived
00007ffbc6d9c620 400160c 187 System.Boolean 1 instance 0 _attentionSending
00007ffbc6d9c620 400160d 188 System.Boolean 1 instance 0 _internalTimeout
00007ffbc8659f30 400160e 60 ...lient.LastIOTimer 0 instance 0000003ff605d860 _lastSuccessfulIOTimer
00007ffbc6d9ba28 400160f 68 System.Object[] 0 instance 0000003ff605d7d8 _securePasswords
00007ffbc6f08e38 4001610 70 System.Int32[] 0 instance 0000003ff605d808 _securePasswordOffsetsInBuffer
00007ffbc6d9c620 4001611 189 System.Boolean 1 instance 0 _cancelled
00007ffbc6d9f6f0 4001612 154 System.Int32 1 instance 12664123 _allowObjectID
00007ffbc6d9c620 4001613 18a System.Boolean 1 instance 1 _hasOpenResult
00007ffbc860ef98 4001614 78 ...ternalTransaction 0 instance 0000000000000000 _executedUnderTransaction
00007ffbc6de0db8 4001615 120 System.UInt64 1 instance 0 _longlen
00007ffbc6de0db8 4001616 128 System.UInt64 1 instance 0 _longlenleft
00007ffbc6f08e38 4001617 80 System.Int32[] 0 instance 0000000000000000 _decimalBits
00007ffbc6e30eb0 4001618 88 System.Byte[] 0 instance 0000003c7570fa38 _bTmp
00007ffbc6d9f6f0 4001619 158 System.Int32 1 instance 0 _bTmpRead
00007ffbc702fe68 400161a 90 System.Text.Decoder 0 instance 0000000000000000 _plpdecoder
00007ffbc6d9c620 400161b 18b System.Boolean 1 instance 0 _accumulateInfoEvents
00007ffbc865e328 400161c 98 ...or, System.Data]] 0 instance 0000000000000000 _pendingInfoEvents
00007ffbc6e30eb0 400161d a0 System.Byte[] 0 instance 0000003ff605d828 _partialHeaderBuffer
00007ffbc6d9f6f0 400161e 15c System.Int32 1 instance 0 _partialHeaderBytesRead
00007ffbc860f2b0 400161f a8 ...t._SqlMetaDataSet 0 instance 0000003c77298240 _cleanupMetaData
00007ffbc865ead8 4001620 b0 ...DataSetCollection 0 instance 0000000000000000 _cleanupAltMetaDataSetArray
00007ffbc6d9f6f0 4001621 160 System.Int32 1 instance 208 _tracePasswordOffset
00007ffbc6d9f6f0 4001622 164 System.Int32 1 instance 144 _tracePasswordLength
00007ffbc6d9f6f0 4001623 168 System.Int32 1 instance 648 _traceChangePasswordOffset
00007ffbc6d9f6f0 4001624 16c System.Int32 1 instance 0 _traceChangePasswordLength
00007ffbc6d9c620 4001625 18c System.Boolean 1 instance 0 _receivedColMetaData
00007ffbc862b0a0 4001626 170 System.Int32 1 instance 11 _sniContext
00007ffbc6d9c620 4001627 18d System.Boolean 1 instance 0 _bcpLock
00007ffbc862b288 4001628 1a0 ...Object+NullBitmap 1 instance 0000003ff605d6f8 _nullBitmapInfo
00007ffbc8574750 4001629 b8 ...bject, mscorlib]] 0 instance 0000000000000000 _networkPacketTaskSource
00007ffbc7010048 400162a c0 ...m.Threading.Timer 0 instance 0000000000000000 _networkPacketTimeout
00007ffbc6d9c620 400162b 18e System.Boolean 1 instance 1 _syncOverAsync
00007ffbc6d9c620 400162c 18f System.Boolean 1 instance 0 _snapshotReplay
00007ffbc865c0b0 400162d c8 ...ect+StateSnapshot 0 instance 0000000000000000 _snapshot
00007ffbc70106c0 400162e d0 ....ExecutionContext 0 instance 0000000000000000 _executionContext
00007ffbc6d9c620 400162f 190 System.Boolean 1 instance 0 _asyncReadWithoutSnapshot
00007ffbc860cb40 4001630 d8 ...qlErrorCollection 0 instance 0000000000000000 _errors
00007ffbc860cb40 4001631 e0 ...qlErrorCollection 0 instance 0000000000000000 _warnings
00007ffbc6d96080 4001632 e8 System.Object 0 instance 0000003ff605d848 _errorAndWarningsLock
00007ffbc6d9c620 4001633 191 System.Boolean 1 instance 0 _hasErrorOrWarning
00007ffbc860cb40 4001634 f0 ...qlErrorCollection 0 instance 0000000000000000 _preAttentionErrors
00007ffbc860cb40 4001635 f8 ...qlErrorCollection 0 instance 0000000000000000 _preAttentionWarnings
00007ffbc8574750 4001636 100 ...bject, mscorlib]] 0 instance 0000000000000000 _writeCompletionSource
00007ffbc6d9f6f0 4001637 174 System.Int32 1 instance 0 _asyncWriteCount
00007ffbc6de5290 4001638 108 System.Exception 0 instance 0000000000000000 _delayedWriteAsyncCallbackException
00007ffbc6d9f6f0 4001639 178 System.Int32 1 instance 0 _readingCount
00007ffbc6d9f6f0 40015e9 a74 System.Int32 1 static 18587 _objectTypeCount
And Finally the _outbuff, which has the actual sproc name
0:118> !do 0000003ff605fab8
Name: System.Byte[]
MethodTable: 00007ffbc6e30eb0
EEClass: 00007ffbc6e30e20
Size: 8024(0x1f58) bytes
Array: Rank 1, Number of elements 8000, Type Byte
Content: ...S............................p.r.c._.Q.u.e.r.y.B.u.i.l.d.s.B.y.U.r.i....@.p.a.r.t.i.t.i.o.n.I.d..&...>...@.b.u.i.l.d.U.r.i.T.
Fields:
Now that you know which sproc call is leaking you can start inspecting the code to see who is making these calls, also, if you are lucky enough you might be able to find the stack trace causing the leak!