Share via

BizTalk Server: Tips for scaling for High Volume WCF calls

  • Article

Introduction

Most BizTalk administrators monitoring operations usually tend to realise that a majority of errors during load is caused when communicating with external systems. This article especially focuses on tuning parameters when interacting with external system(s) and attempts to explain a few ways to check if the lack of tuning , these parameters, is the source of your problem.

Applies to

  • BizTalk server 2013 R2 
  • IIS 6 - IIS 8.5 
  • .Net v4 - .Net v4.5 

Tip #1: Turn on authPersistNonNTLM on the service side for WCF HTTP calls using non NTLM authentication

Scenario

You have WCF client bindings that use http(s) transport (e.g. basichttpBinding, wshttpBinding) with transport credentials (e.g. windows Kerberos) for authentication.  These WCF clients talk to services hosted on IIS 6, IIS7 or IIS 7.5 . However you face performance timeouts or slow responses under load.

Symptoms

  1. BizTalk server experiences timeouts under heavy load when making external http(s) calls
  2. The IIS logs contain a lot of 401 errors when you load test.
  3. Inspecting the http header in a tool like fiddler, shows that Persist-Auth:false ,as per https://blogs.msdn.microsoft.com/benjaminperkins/2011/10/31/kerberos-authpersistnonntlm-authentication-request-based-vs-session-based-authentication/

Solution

  1. Turn authPersistNonNTLM  as per https://support.microsoft.com/en-us/kb/954873
<system.webServer>
  <modules runAllManagedModulesForAllRequests="true"/>
   <security>
     <authentication>
       <windowsAuthentication enabled="true" authPersistNonNTLM="false"> 
       </windowsAuthentication>
     </authentication>
   </security>
 </system.webServer>

Additional Info

  1. You may also face a .net bug where your application stops responding or hangs.

For details see KB 2497453 - FIX: Application may stop responding after several HTTP requests are aborted if you use the HttpWebRequest class to send requests in the .NET Framework

Tip#2: Increase max concurrent client http connections for your top services to avoid client side timeouts.

Scenario

Your BizTalk solution talks to a service, over http(s). When send multiple simultaneous requests to this service, the BizTalk server experiences timeouts.  On further investigation, you realise that the server response times are not an issue and client side http requests tracing through tools like fiddler seem to be responding with ok response times. Then your problem could be related to .net client side timeouts as the .net service point manager internally uses queues to process http request per endpoint+authentication , and the time-out settings usually include the queue time at the client side.

Symptoms

  1. BizTalk server experiences timeouts under heavy load when making external http(s) calls
  2. When you load test the service, tracing client side requests with fiddler show fast response times (order of milliseconds), but the client calls still time-out
  3. During the load test, the HttpWebRequest performance counters HttpWebRequests Queued/Sec & or HttpWebRequests Average Lifetime show high values. For details about these counters see msdn blog  New Performance Counters for HttpWebRequest
    1. Note: If during load tests, the client side performance counter HttpWebRequests Average Lifetime shows zero, when the HttpWebRequests Created per sec is not zero then you may not be closing/aborting the WCF connections correctly. If so, fix the code, before you increase max connection

Finally after investigating through all cases to isolate client side timeouts, vs clients not closing connections and server failing under load you definitely have sufficient evidence to suggest that these errors are due to client side http request timeout. Now you can now increase the client max client connections to an acceptable level.

Solution

  1. Increase max client connections (for Biztalk this is in config BTSNTSvc.exe.config or BTSNTSvc64.exe.config for 64-bit hosts) to the host to 12 * # number of CPUS as per https://msdn.microsoft.com/en-us/library/ee377083(v=bts.10).aspx


      <    system.net    >   
                  <      connectionManagement      >       
   
                    <!-- Ideally max connections should be set upto 12 * Number of CPUS       
                     Caveat:        
            Increasing the max connections may start to cause server side issues. So this setting needs to be a tradeoff between client side timeouts and server side load issues. If you think increasing the number of max client connections causes the server to fail, you can decrease the max connections and increase the client side timeout settings. -->       
   
                    <      add maxconnection="24" address="http://contoso.com/ ; :8080"/>   
                    <      add maxconnection="24" address="*"/>   
                  </      connectionManagement      >       
      </    system.net    >

Additional Info

  1. Understanding System.Net Connection Management and ServicepointManager https://blogs.msdn.microsoft.com/adarshk/2005/01/02/understanding-system-net-connection-management-and-servicepointmanager/
  2. Why Only Two Concurrent Requests for WCF Load Testing - http://blogs.msdn.com/b/wenlong/archive/2009/02/08/why-only-two-concurrent-requests-for-load-testing.aspx
  3. Http web requests performance counters
    1. https://msdn.microsoft.com/en-us/library/70xadeyt(v=vs.110).aspx
    2. https://blogs.msdn.microsoft.com/ncl/2009/08/07/new-performance-counters-for-httpwebrequest/
  4. WCF timeouts: https://msdn.microsoft.com/en-us/library/hh924831%28v=vs.110%29.aspx?f=255&MSPPError=-2147217396

Tip#3: Use NetTcpBinding instead of bindings that rely on http(s) for high performance in intranet scenario.

Scenario

Your BizTalk solution uses an intranet WCF service, over http(s) quite heavily. To improve performance, you can enable NetTCPBinding and configure it for burst load scenarios.

Solution

  1. Enable NetTcpBinding on the service side.
  2. Increase max concurrent service instances on the service side before service throttling kicks in
<system.serviceModel>
    <behaviors>
      <serviceBehaviors>
        <behavior>
          <serviceThrottling maxConcurrentCalls="100000" maxConcurrentInstances="100000" maxConcurrentSessions="100000" />
        </behavior>
      </serviceBehaviors>
</behaviors>
</system.serviceModel> 
  1. Increase listenbacklog and maxconnections on the service side nettcpbinding configuration.
<netTcpBinding>
    <binding  
         listenBacklog="1000" maxConnections="1000"   />
</netTcpBinding>
  1. If you have hosted the net.tcp enabled service on IIS then the listenbacklog, maxPendingConnections settings must be configured on SMSvcHost.exe.config (for 64 bit app pool process using .net 4.0 , this is located at C:\Windows\Microsoft.NET\Framework64\v4.0.30319  on 64 bit with .net 4.0) . For more info, see section additional info about modifying smsvchost config
<net.tcp listenBacklog="100" maxPendingConnections="1000" maxPendingAccepts="2" >
   <!-- ........-->
</net.tcp>

For the new settings to take into effect,  the SMSvcHost.exe must be  restarted. The only way to stop SMSvcHost.exe is to stop all dependent services such as Net TCP port sharing service.  So you must stop  nettcpportsharing and related servicesuntil the SMSvcHost.exe disappears from the windows task manager and start these services again.

Additional Info

Tip#4 Slow down sending requests to external services using host throttling 

Scenario

Your BizTalk solution process a huge volume of messages where scalability is far more important than high performance. Common use cases for this scenarios are messages from a one way port (ftp, file adapter, one way http receive, batch processing cases etc.) and eventually processing them successfully is more important completing them quickly. These messages rely on external systems to complete processing, but the external systems are unable to cope with a huge volume of concurrent requests causing failures.

Solution

  • Create a new BizTalk host  called say, BiztalkHostSuperSlowHttpSend, as per: https://msdn.microsoft.com/en-us/library/aa561079.aspx   
  • You now want to slow down the number of messages delivered to this host, as the rate of message delivery to this host is much faster than the adapter can process, decrease the value of the In-process messages per CPU , see sample value (10)  in the screen shot. PS: You may need to tweak (lower this value even further) this value depending on your requirements. 

  • Use this host (BiztalkHostSuperSlowHttpSend ) as the send handler for all your send ports where you want the messages to be slowed down 
  • PS: If you are using http to connect to the external system, there are other parameters such as maxConnections see section Tip#2 that can be tweaked. Decreasing max connections for that particular server:port and increasing the WCF send timeout will allow you to throttle the requests to that server and complete processing without timeouts. However the maxconnections settings applies to all outbound connections to that server:port and may not work well with cases where you want to prioritise certain messages to that server. An example of this would be BizTalk receiving a request response web operation and this relies on the same external server:port to complete its operation. In such cases you can configure 2 different send ports, one using use a slow lane (throttled BizTalk host eg BiztalkHostSuperSlowHttpSend) for batch processing and the other port using the default send handler for that transport for optimal message processing. 

Additional info

See Also

Another important place to find an extensive amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.