Web Services: Concurrent Connections

The issue of concurrent outbound Web Service calls being throttled to two per endpoint comes up again and again with unmanaged and managed code – and it embarrassingly bit me last week in a BizTalk Performance and Scaling exercise despite me harking on about it all the time with standard .NET development L

The HTTP 1.1 Specification, specifies that a “single-user client SHOULD NOT maintain more than 2 connections with any server or proxy” . This is to ensure that a user cannot overload a remote Web Server, this is all understood and the various Microsoft HTTP stacks all respect this restriction as per the specification.

This is fine for user scenarios, but when you’re calling a Web Service from an ASP.NET Site, BizTalk Orchestration (via SOAP Adapter), etc. you will also hit this limit for the whole of your application rather than for each user of your application. This concurrent connection restriction is applied to an AppDomain which in the ASP.NET world means a Virtual Directory, which then means all users concurrently calling ASP.NET Pages will have any outbound Web Service calls throttled down to two concurrently, pending requests will then be queued.

This as you’d imagine causes a massive scalability and performance problem, and unfortunately this default behavior makes its way into live solutions more often that not, but fortunately it’s easy to fix.

In your code you can override this as per this article, in your initialization you need to set: ServicePointManager.DefaultConnectionLimit = 4;

This will then be hardcoded inside your application so my recommendation would be to control it via your .config file:

<system.net>

            <connectionManagement>

      <addaddress="*"maxconnection="40"/>

</connectionManagement>

</system.net>

More background on this here. As for the value you should use it’s really up to your scenario, as long as the remote Web Server can handle it (and is under your control) you probably want it as close to the number of possible threads available on the requesting machine. If it’s an ASP.NET page calling the Web Service you ideally want it set to 25*NumberofProcessors which matches the ASP.NET Worker Thread pool.

A quick and easy way to see if you’ve hit this issue is to monitor the number of Concurrent Connections to your Web Server, if you’re running IIS the “Concurrent Connections” counter under the Web Service performance object is the one to watch. If you never see this go about 2 * NumberOfRequestingMachines then you might be hitting the limitation. 

I was in the performance lab and we were only seeing 4 concurrent connections to the IIS Box hosting a Web Service, we had two BizTalk 2004 Servers (Quad Procs) and were putting a lot of load through the system and should have seen far higher connections. We put the above change into the BTSNTSVC.exe.config file and low and behold the number of Concurrent Connections got into the 30/40s J

For some reason I just assumed the BizTalk 2004 Install would “overcome” this limitation but it doesn’t, and the more I think about it – it shouldn’t either. The Installer doesn’t know how many connections your remote Web Service can handle – that’s something you need to tune.

Anyway, we live and learn – Hopefully this will help someone out there and save them some head scratching J 

If your calling Web Services from .NET, BizTalk Server 2004 and you have lots of concurrent users: Your almost certainly experiencing this problem!

Comments

• Anonymous
  March 07, 2005
  Hi Darren, good post - a real gotcha! The best article I've seen on ASP.Net threading is this one http://support.microsoft.com/default.aspx?scid=%2Fservicedesks%2Fwebcasts%2Fen%2Ftranscripts%2Fwct060503.asp
  
  Rob.
  
  

• Anonymous
  March 07, 2005
  Hi,
  
  Thanks for posting this information. We were having the exact same problem and did not know how to fix this. We put in the change that you suggested in the BTSNTSVC.exe.config and voila we are seeing more concurrent connections.
  
  Thanks once again.
  
  Radha.

• Anonymous
  March 09, 2005
  Darren this is great stuff. I'm assuming this also applies to the HTTP adapter? Do you know how (or if) this correlates with the HTTP adapter HttpOutMaxConnection setting in the performance white paper, that has a default value of 5?

• Anonymous
  March 09, 2005
  The HTTP Adapter team seem to have exposed the maxConnection setting via a Registry Key, that should override anything you put in a config file, but your best trying it to be sure.

• Anonymous
  March 10, 2005
  Thanks Darren, our experiences seem to show the HttpOutMaxConnection registry key does indeed override it.

• Anonymous
  March 18, 2005
  This is great information. I am developing a BizTalk solution that works with Axapta through a webservice and was frustrated to find only 2 simultaneous connections. After playing with a registry tweak for Internet SettingsMaxConnectionsPerServer, it's nice to have found an application-specific solution to this. I was even considering playing with the wininet.dll and doing some very tedious WinAPI coding. You saved me many frustrating hours.

• Anonymous
  September 04, 2005
  After reading a post from a coworker I found two links of interest regarding configuration changes required...

• Anonymous
  August 31, 2006
  Fix for security errors when installing BizTalk 2004 on Windows XP SP2 MSDN Biztalk Developer Center

• Anonymous
  January 22, 2007
  “single-user client SHOULD NOT maintain more than 2 connections with any server or proxy”.

• Anonymous
  May 29, 2009
  PingBack from http://paidsurveyshub.info/story.php?title=darren-jefford-web-services-concurrent-connections

• Anonymous
  June 11, 2009
  PingBack from http://castironbakeware.info/story.php?title=darren-jefford-web-services-concurrent-connections

• Anonymous
  April 12, 2011
  The comment has been removed