BizTalk Server: Calling Web Services In Large Batches

Article
1/17/2024

Introduction

This article will show you how to make a BizTalk solution that can handle large message batches and limit the number of concurrent web service calls (i.e. limit the rate of web service calls). The solution consists of configurations of the send port, BizTalk Server Configuration File, and DTC System Transactions.

Problem

Sending large message batches through BizTalk (via de-batching of XML or flat files) which calls web services, will sooner or later overload the web service, or cause the send host to time out messages it hasn't even tried to send yet. The first happens because BizTalk uses asynchronous calls to the web service, and the latter happens when the web service takes time to respond and the batch is large enough.

This article will show you how to make a BizTalk solution that can handle large message batches and limit the number of concurrent web service calls (i.e. limit the rate of web service calls).

Background on BizTalk infrastructure

Messages delivered to the message box are delivered in batches, and so are messages picked up by send hosts. The batch size can be configured, but in this scenario it seemed it didn't have any effect.

Whenever a batch is picked up by a send host, it is performed within a DTC Transaction, which has a time-out. Also, when the send host picks up a batch, the port's Send time-out starts counting. Most of the time we receive and send messages one at a time, and then the port's Send time-out makes sense. But when we send large batches of messages, we must take into account the transaction time for all messages in the batch for the Send time-out.

This means that there are two time-outs that must be taken care of; the DTC transaction time-out and the Send port's Send time-out.

Solution

The solution consists of configuration only, but some configurations are system-wide. In most cases, they won't affect your system's performance, unless you already have altered them for some reason.

DTC System Transaction Time-Out

The DTC System has configurable default and maximum values for the time-out. For each transaction, the programmer can set the time-out, up to the maximum value. These values have defaults of 1 and 10 minutes, respectively. You have to configure the maximum timeout value to a safe value, sufficient for your needs. It must be at least as long as the send port's Send time-out, calculated below.

The configuration is made in the DTC Server's machine.config files (both 32- and 64-bit, for the .Net framework version you work with. For example, the machine.config for 64-bit .Net Framework 4.0 is located in c:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config\

Add the following element near the end of each relevant machine.config file (before the </configuration> element):

<system.transactions>
  <machineSettings maxTimeout="01:00:00" />
</system.transactions>

The above example sets the maximum time-out to 1 hour.

Send Port Time-Out

The Send port's Send time-out must be calculated for the total transaction time of the whole batch of messages, as the average time for one web service call multiplied by the batch size, divided by the number of concurrent requests being made:

t_timeout = t_wscall * n / c

where

t_wscall is the average time for a single web service call
n is the batch size
c is the number of concurrent requests to the web service
t_timeout is the required send time-out time

Example:

  Batch size: n = 20,000
  Average Web Service call time: t_wscall = 0.5 seconds
  Number of concurrent requests: c = 4
  Required time-out time: t_timeout = 0.5 * 20,000 / 4 = 2,500 seconds = 41.67 minutes

Throttling Settings on the Send Host

In this case, you want BizTalk to run with your configured capacity and you don't want BizTalk so start throttling. Therefore, you have to configure your send host to accept the whole batch without starting to throttle.

The throttling setting to change is the "In-process messages" (sometimes referred to as "in-flight message count"). Set the value to something higher than your batch size n.

Also, check that the throttling setting "Message count in DB" is set to a higher value than "In-process messages", since BizTalk will probably have other things to do meanwhile processing your batch.

Limiting The Number Of Concurrent Network Connections To A Single Host

To limit the number of concurrent network connections to a single host (or general), there is a maxconnection setting in the BizTalk NT Service config files BTSNTSvc.exe.config and BTSNTSvc64.exe.config:

<configuration>
  <system.net>
    <connectionManagement>
      <add address="www.example.com" maxconnection="4" />
    </connectionManagement>
  </system.net>
</configuration>

The value you state here is the value of variable c in the formula above.

The default value of maxconnection is 2.

Do this in both the 32- and 64-bit configuration files so you don't run into problems later, when somebody alters the bindings in the future.

Note that

.Net framework does not use this setting for localhost, so if your web service is located on your BizTalk server, this solution won't work. In an ideal production environment that shouldn't be the case.
The setting is connections per host instance, so if the server you are connecting to has a hard limit on the number of connections then you either need to ensure that all connections are from a single host instance or the divide the maximum number by the number of host instances that could connect.

What about Send Port Ordered Delivery?

The standard recipe for batching web service calls is to check the box "Ordered delivery" on the send port, and be satisfied with that.

However, that will limit the concurrency to exactly 1 connection to the web service, that is, fix the value c in the formula above to 1, regardless of any maxConnection setting in the config file. It is simple, but depending on your batch size, the batch might take too long time. This check-box does quite some magic, since it changes the behaviour of BizTalk regarding transaction and send port time-out: Each message will get its own transaction and the send port send time-out will be calculated for one message only, not for the whole batch. If you are using Orchestrations that have multiple calls to the same port, then it will cause the Orchestration instance to dehydrate while waiting for its turn at the port and will hydrate again when it is its turn, this has a large impact on throughput.

Conclusion

These configurations give you the possibility to run large batches of data the same way you run small daily batches of data, such as when you have to start a new system that must be pre-filled or with migrated data. Often, the pre-filling/migration is done through SSIS directly into the database, but that has at least one disadvantage: The SSIS package might have bugs too. Another disadvantage may be that other third-party systems has to be fed the data too, and this is not always possible or can be expensive in time and money. Using the method described here, you run the big batches the same way you will run the system day-to-day, guaranteeing the same business rules.

Test and Verification

A Sample Time-Consuming Web Service

To test and verify the solution, you will most likely not want to spam your LOB server with tens of thousand messages, not even in your test environment. Therefore I made a "failing service", which has shown to be handy in many scenarios. It's a little web service that can sleep for a while, or throw an exception on demand, to test time-consuming scenarios or error-handling routines. It logs received calls to the Application event log.

The service exposes a single operation DoAction which returns a string. The operation can currently perform two actions: Delay and Throw. If it receives an unknown action it immediately returns a string. The operation also has an integer parameter, so that you can tell the service for example how long the delay shall be. The service must be configured for high throughput with a serviceBehavior.

BizTalk Test Application: Calling Web Services In Large Batches

The BizTalk application to test with is very simple. It consists of a receive location and a send port, a receive pipeline, and two schemas. The send port subscribes to anything received from the receive port. The send port also performs a mapping from the received flat file format to the failing service's XML format. The receive port has a pipeline with a flat file disassembler that performs a de-batching of the lines in the flat file.

References

Sample Code

The test and verification code is available for download:

Sample BizTalk application: BizTalk: Calling Web Services In Large Batches
Test web service: Sample Failing And Time-Consuming Web Service

Share via