WCF netTcpBinding – What to do if: “..the socket did not complete within the allotted timeout of…”

Sometimes I happened to face a odd error occurring when a client tries to connect to a WCF Service which has a TCP endpoint (netTcpBinding). It looks like this error can raise when one or more clients attempt to establish a good deal of connections in the same time.

I’m talking about the following exception, that you can observe in the WCF service traces (refer to “Configuring tracing” MSDN article to know how to enable WCF tracing):

<ExceptionType>System.TimeoutException, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>

<Message>The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:00:05. The time allotted to this operation may have been a portion of a longer timeout.</Message>

Please note this exception can occur on the service side, while on the client side you should observe some failing connection attempts, since the server is not responding within the expected maximum time.
If you want to perform a WCF service tuning, you may want to consider many aspects: this is not the right place to explore the entire topic, I just want to stay focused on one of them which is often disregarded, but it can give us headaches during our load tests.

Let’s assume you have a burst of incoming connections towards our netTcpBinding WCF service and some of them fail during the establishment phase, if you enable WCF tracing on the service and suddenly face the exception reported above, chances are that you need to properly configure the ChannelInitializationTimeout.
The ChannelInitializationTimeout (default value: 5s) actually is the maximum time the channel can be in the initialization status before being disconnected.

How can you configure it? Let’s say you config file is like this:

 <services>
  <service name="Microsoft.ServiceModel.Samples.CalculatorService"
           behaviorConfiguration="CalculatorServiceBehavior">
    ...
    <endpoint address=""
              binding="netTcpBinding"
              bindingConfiguration="Binding1" 
              contract="Microsoft.ServiceModel.Samples.ICalculator" />
    ...
  </service>
</services>

<bindings>
  <netTcpBinding>
    <binding name="Binding1" 
             closeTimeout="00:01:00"
             openTimeout="00:01:00" 
             receiveTimeout="00:10:00" 
             sendTimeout="00:01:00"
             transactionFlow="false" 
             transferMode="Buffered" 
             transactionProtocol="OleTransactions"
             hostNameComparisonMode="StrongWildcard" 
             listenBacklog="10"
             maxBufferPoolSize="524288" 
             maxBufferSize="65536" 
             maxConnections="10"
             maxReceivedMessageSize="65536">
      <security mode="None">
        <transport clientCredentialType="None" />
      </security>
    </binding>
  </netTcpBinding>
</bindings>

Unfortunately the ChannelInitializationTimeout property is not directly accessible with the netTcpBinding, therefore we need to define an equivalent customBinding that additionally contains the ChannelInitializationTimeout property explicitly configured with a value greater than 5s (for example: 1 minute).

 <services>
  <service name="Microsoft.ServiceModel.Samples.CalculatorService"
           behaviorConfiguration="CalculatorServiceBehavior">
    ...
    <endpoint address=""
              binding="customBinding"
              bindingConfiguration="Binding1" 
              contract="Microsoft.ServiceModel.Samples.ICalculator" />
    ...
  </service>
</services>

<bindings>
        <customBinding>
           <binding name="Binding1"
        closeTimeout="00:01:00"
        openTimeout="00:01:00"
        receiveTimeout="00:10:00"
        sendTimeout="00:01:00">
             <binaryMessageEncoding />
             <tcpTransport maxBufferPoolSize="524288" maxReceivedMessageSize="65536" hostNameComparisonMode="StrongWildcard"
                           maxBufferSize="65536" maxPendingConnections="10" channelInitializationTimeout="00:01:00"
                           transferMode="Buffered" listenBacklog="10" portSharingEnabled="false" teredoEnabled="false"
                   >
             </tcpTransport>
           </binding>           
         </customBinding>
</bindings>

It’s important to say that the ChannelInitializationTimeout property has a rather “strict” default value because its purpose is to secure the WCF service in order to prevent any Dos attacks. One possible situation when the ChannelInitializationTimeout prevents a Dos attack could be the following: let’s assume we have authentication enabled between client and service: the former must send a few bytes so that the latter can begin the authentication phase: that number of bytes is much lower than the whole message size. At this point, instead of waiting for the ReceiveTimeout (default value: 1m) expiration, the service can close the connection if the expected bytes needed to begin the authentication are not received within the ChannelInitializationTimeout, preventing a malicious client can keep connections open during the initialization phase more than expected.

The morale of the story: if you need to increase the ChannelInitializationTimeout value because of expected workload peaks, feel free to do it but pay attention: the more a client is allowed to keep a connection in the initialization status, the less your WCF service is secure.

See you ,

Andrea