getting error " 502 proxy error. The requested resource is in use. (170)." on the upstream proxy TMG servers
This one is about a case, where Admin had configured Two TMG 2010 upstream proxy servers and he had two TMG 2010 downstream proxy servers. Users started reporting that they intermittently get error
" 502 proxy error. The requested resource is in use. (170)." while accessing internet.
At the time of issue, even if you try to access it from the Upstream TMG server itself, you will see this error on the browser. To avoid the problem ,Admin was bypassing these upstream TMG proxy servers. So it appeared to be a Load issue.
Troubleshooting and Data analysis
During the course of troubleshooting, I collected TMG data packager while doing repro of the issue i.e. when issue occurred. In the TMG traces I found following.
************************************************
ERROR:Cannot create more than 3000 concurrent inspection sessions.
ERROR:spScanner->CreateSession failed, hr=0x800700aa(ERROR_BUSY)
ERROR:CreateInspectionSession() failed, hr=0x800700aa(ERROR_BUSY)
ERROR:Accumulate failed, hr=0x800700aa(ERROR_BUSY)
ERROR:pRequestContext->OnClientSendRawData failed, hr=0x800700aa(ERROR_BUSY)
Warning m_spProtocolHandler->GetExpectedSize failed, hr=0x80070015(ERROR_NOT_READY)
Info Parsing response headers hasn't completed yet, cannot check for filename.
ERROR:pRequestContext->OnClientSendRawData failed, hr=0x800700aa(ERROR_BUSY)
ERROR:SendResponseHeaders failed 170(ERROR_BUSY)
Info WPPISAPUBLIC:Returning error text "The requested resource is in use. " for error code 170(ERROR_BUSY)
*********************************************
We can see from above trace that we are getting error
"ERROR:Cannot create more than 3000 concurrent inspection sessions. " in the trace, which means we are hitting the maximum
threshold for concurrent inspection sessions.
Then as a follow through we get message that server was busy. or Requested resource is in use.
After doing some research on this behavior I suggested following :
1. To increase the default value of 3000 via this RAT flag in registry
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\RAT\Stingray\Debug\Web Filters
\EMP_MAX_CONCURRENT_INSPECTION_SESSIONS
2. To add another upstream server i.e. scale up the number of servers to share
the Load further to avoid reaching this limit of concurrent inspection sessions.
After we configured the reg key with a value of 5000 on the TMG servers,
we did not see issue occurring again.
We can change this value upto our requirements but better idea would be to scale up the
number of servers as we may have to keep doing this if user base would keep increasing.
Comments
- Anonymous
January 01, 2003
sorry guys i do not get any alerts for the comments on my blog, i just saw this and replying
*********************************************************
So key if you don’t see then create them as seen in the path given infront of key below, Then create Dword “EMP_MAX_CONCURRENT_INSPECTION_SESSIONS” and set a value as per your requirement , if you notice my blog post scaling up is a better idea.
Note Please take backup of your registry keys before any modifications
Key: HKEY_LOCAL_MACHINESOFTWAREMicrosoftRATStingrayDebugWeb Filters
Value: EMP_MAX_CONCURRENT_INSPECTION_SESSIONS
Type: REG_DWORD
Data: 4500 -5000(Decimal)
Hope that helps - Anonymous
January 29, 2014
Hi. How to view TMG traces? It's a ISALOG.BIN? - Anonymous
October 13, 2014
About the registry below I need the right syntax for the reg key value.
please advice.
[HKEY_LOCAL_MACHINESOFTWAREMicrosoftRATStingrayDebugWeb FiltersEMP_MAX_CONCURRENT_INSPECTION_SESSIONS] "5000"="" - Anonymous
March 31, 2015
Hi, our TMG is sometime reporting the same issue and we don't use upstream server. Also, the registry path showing here is not there in our TMG. Please provide the correct registry path - Anonymous
August 26, 2015
Hi. What application did you use to view the isalog.bin file? - Anonymous
March 30, 2016
FYI.
Isalog.bin is only readable by Microsoft.