Synchronous To Asynchronous Flows Without An Orchestration

Scenario

Some months ago I had the chance to work with a customer who asked me a very interesting and tricky question: does BizTalk Server 2006 R2 support synchronous-to-asynchronous messaging only flows, without the need of an orchestration to couple a Two Way Request Response Receive Location with a One-Way Send Port to send out the request and a One-Way Receive Location to get response back from the invoked system?

The use case the customer wanted to reproduce was the following:

1. A Two-Way SOAP Receive Location receives a request message and publishes it to the BizTalkMsgBoxDb.
2. The message is consumed by a MQ Series One-Way Send Port which writes the document into an outgoing MQ queue.
3. The response message is received by a given MQ Series One-Way Receive Location which reads the document from an incoming MQ Queue and publishes the message to the BizTalkMsgBoxDb.
4. The response message is finally received by the Two-Way SOAP Receive Location which received the original request and then the document is returned to the caller.

I started to create a POC of this scenario, using the FILE adapter in place of the MQ Series Adapter and folders in place of MQ queues. In the majority of BizTalk applications that use a Request-Response receive port, the request message is used to start a given Orchestration or it is directly relayed to a solicit-response Send Port. When the Orchestration or Send Port provides the response message, this message is routed back to the Two-Way Receive Location that submitted the original request message. These two design patterns are fully supported by the product group and the picture below depicts the case where a Two-Way Request Response Receive Location is directly coupled with a Two-Way Solicit Response Send Port which is responsible for invoking an external web service.

But what about our case in which the request and response are respectively sent and received to/from an external target system not using a Two-Way Solicit-Response Send Port, but using a One-Way Send Port while and a One-Way Receive Location? The standard solution is using an Orchestration to correlate the request submitted to the external system with the response received from this latter. But as you can see from the picture below, this pattern requires 8 roundtrips to the BizTalkMsgBoxDb to complete, 4 message writes and 4 message reads.

In other words, this approach requires multiple roundtrips to the MessageBox to publish and consume messages to/from the BizTalkMsgBoxDb and to hydrate and dehydrate the internal state of the orchestration and all these operations can dramatically increase the contention on SQL and reduce the overall throughput and speed of the application.

So the following question spontaneously emerged: is there any technique to implement the same behavior getting rid of the orchestration? This pattern would allow to eliminate 2 messages publications and 2 message reads and this would allow to speed up the execution and to decrease the contention on the BizTalkMsgBoxDb.

The solution to this problem rests in understanding how BizTalk matches a response message to an initial request message. After you understood the internal subscription matching mechanism implemented by BizTalk, the second step is to individuate the context properties that need to be propagated along and promoted in the One-Way Receive Location in order to allow the Two-Way Receive Location to receive the response when this latter is published to the BizTalkMsgBoxDb.

When a message is received through a Request-Response Receive Location and published to the BizTalkMsgBoxDb, an instance subscription is created to allow the Receive Location to receive the response back when this latter is published to the MessageBox. This subscription expression is always composed of the EpmRRCorrelationToken promoted property that is in the request message's context, and a RouteDirectToTP promoted property with a value of True.

In particular the EpmRRCorrelationToken is a string containing the name of the BizTalk node which received the request, the process id of the host instance which received the message and a GUID which represents a correlation id. For more info on these and other context properties you can review the following article on MSDN https://msdn.microsoft.com/en-us/library/ms966048.aspx.

I made some tests and I quickly realized that those properties were not sufficient to correlate a response back to the original request. In other words, even if the subscription expression is composed of EpmRRCorrelationToken and the RouteDirectToTP properties, promoting them with the expected value in the message context of the response document before publishing this latter is not sufficient to pass message back to the Receive Location. It was clear that it was necessary to propagate some additional context properties, even if they were not used by the subscription expression. After some attempts, I was able to identify the set of context properties that were necessary to correlate a response message back and I was able to create a sample were a synchronous WCF Request Response is couple to 2 asynchronous One-Way FILE Ports.

The Solution

I created a sample BizTalk application called SyncToAsync containing a Two-Way Receive Port called SyncToAsync.Request.ReceivePort. Subsequently, I created a WCF-NetTcp Request Response Receive Location called SyncToAsync.Request.WCF-NetTcp.ReceiveLocation and a WinForm driver application to submit requests to it. Using the WCF Service Publishing Wizard I created a MetadataExchange Endpoint for the web service exposed by the WCF Receive Location and then I used the Add Service Reference command provided by Visual Studio 2005 to create a proxy class in order to submit messages to it. I selected the XmlReceive and PassThruTransmit pipelines on the SyncToAsync.Request.WCF-NetTcp.ReceiveLocation to process respectively the incoming request and the outgoing response messages. In particular, The XmlReceive pipeline contains the Xml Disassembler component which is responsible to probe and individuate the message type and to promote this information along with other context properties. The second step was creating a One-Way FILE Send Port that subscribed all the messages published by the former Receive Location. In order to that I just used the BTS.ReceivePortName = SyncToAsync.Request.ReceivePort predicate as Filter expression. At this point I enabled the SyncToAsync.Request.WCF-NetTcp.ReceiveLocation WCF Receive Location and maintained disabled the SyncToAsync.Request.FILE.SendPort Send Port. Then, using the .NET driver application, I submitted some messages to the BizTalk application. Since no subscribers were active, those messages were suspended. Therefore, using the Administration Console I could review the promoted properties of the incoming request message. After some attempts, I realized that the properties highlighted below (CorrelationToken, EpmReqRespCorrelationToken, ReqRespTransmitPipelineID, IsRequestResponse) plus the RouteDirectToTP that is not promoted on the original request message, were the properties that had to be propagated and subsequently promoted on the One-Way FILE Receive Location used to publish the response to MessageBox.

For the demo I created a very simple XML Schema to represent the basic attributes of a Book entity. I called the schema below InboundBook and this represents the format of the incoming message sent by the .NET Driver application and received by the WCF Receive Location.

Then I created an extended version of the above XML schema in order to demote some of the aforementioned context properties inside the outgoing message. I called this schema OutboundBook. As you can see in the picture below, this schema is composed of a Header and a Body:

• The Header contains the CorrelationToken, EpmRRCorrelationToken and ReqRespTransmitPipelineID elements. In particular, this latter will contain the GUID of the transmit pipeline used by the WCF Receive Location to process the outbound response message. I promoted these 3 elements respectively to the properties with the same name exposed by the BTS.bts_system_properties schema contained in the Microsoft.BizTalk.GlobalPropertySchemas assembly. In this way, if on the Send Port I use any send pipeline containing the XML Assembler component (e.g. the XmlTransmit standard pipeline), at run-time, when the pipeline is executed, the value of those message context properties are demoted to those elements.
• The Body contains the payload of the message, that is the ISBN and Title elements.

Then, I created a map to transform an InboundBook message into a OutboundBook document on the FILE Send Port.

For the response message, I created an XML Schema to represent the author of a certain book. Even in this case, the schema is composed of a Header and Body elements. The Header contains the following elements:

• CorrelationToken.
• EpmRRCorrelationToken.
• ReqRespTransmitPipelineID.
• IsRequestResponse.
• RouteDirectToTP.

Each of those elements are promoted to the context properties having the same name and contained in the Microsoft.BizTalk.GlobalPropertySchemas assembly. In particular, the IsRequestResponse and RouteDirectToTP elements are defined as boolean and they have a Fixed value equal to 1.

Finally, I created a One-Way Receive called SyncToAsync.Response.ReceivePort and a corresponding FILE Receive Location called SyncToAsync.Response.FILE.ReceiveLocation. I set the standard XmlReceive pipeline on this Receive Location which contains the XmlDisassembler component that is responsible for promoting contained in the Header section of an Author message.

Inside the configuration file of the Driver application you can specify the location of the inbound and outbound folder used respectively by the FILE Receive and Send Ports. The client application used a FileSystemWatcher instance to trigger when a file is published by the Send Port on the outbound folder. When this event happens, a dialog opens where you can review the book information and enter the name and surname of the corresponding author. When you press the Submit button, an instance of the Author message is created for you by the Driver application, the value of the Header elements of the OutboundBook message are copied to the corresponding elements inside the Header of the Author message and the value of the IsRequestResponse and RouteDirectToTP elements is set to true. Then, the Author message is written in the inbound folder where it is received by the SyncToAsync.Response.FILE.ReceiveLocation which promotes the context properties and publishes the message to MessageBox. When the XML document is published, subscriptions are evaluated and the response message is passed to the WCF Receive Location which returns the message to the client application. The following picture depicts the overall architecture of the demo.

You can download the code here.

Conclusions

This technique is a good way to couple a synchronous Request Response Port with 2 Asynchronous One-Way Ports without using an Orchestration. This approach, compared with the standard approach which makes use of an Orchestration to implement the same behavior, allows saving 4 roundtrips to the MessageBox. Therefore, this pattern allows to decrease the contention on the SQL Server instance running the master BizTalkMsgBoxDb and speed up the overall message processing. By the way, I don’t think that the Product Group would support this solution, even if it doesn’t require any custom component. In fact, it uses context properties that were not probably meant to be used by developers, even if the BizTalk programmers’ community knows and uses them. Remember in fact that Microsoft could in any moment release a fix or a service pack that could change the behavior of the subscriptions evaluation and of the request-response correlation, invalidating this technique. On the other hand, this eventuality is not likely because the request-response correlation mechanism is quite consolidated and I don’t think they would have time and will to re-engineer this mechanism.

I have also some ideas on how to modify/customize the technique I described to obtain the same behavior in a slightly different way: for example, instead of including all the necessary context properties in the header of the response message and promote all of them, you could promote the EpmRRCorrelationToken, CorrelationToken and RouteDirectToTP, since the IsRequestReponse and ReqRespTransmitPipelineID are originally written properties. The solution I implemented is probably the fastest one, but it requires all the 5 properties to be propagated and promoted. Another approach could be to save the value of these properties in a custom DB or in a distributed cache like AppFabric Caching with a custom pipeline component in the transmit pipeline run by the One-Way Send Port and retrieve them in the receive pipeline on the One-Way Receive Location. This mechanism would allow you to propagate just one unique identifier or correlation id that should be repeated in the response message. If you don’t want to use the EpmRRCorrelationToken or the CorrelationToken, you could even use an existing ID already present in the request and response documents as the OrderID or CustomerID.  This approach is less performant (since you need to access a custom DB on the send and receive pipelines) but more flexible and less intrusive, and it’s relatively easy to implement.

Comments

• Anonymous
  July 23, 2008
  My friend and colleague Paolo Salvatori has just started a blog at http://blogs.msdn.com/paolos/ and

• Anonymous
  January 22, 2009
  While writing my previous blog posting, I remembered that Paolo Salvatori had posted about another very useful blog entry about Synchronous To Asynchronous Flows Without An Orchestration.

• Anonymous
  March 31, 2009
  Cool solution, Paolo.  Even though it involves a little bit of additional work in the intial stages, it would pay off big time in performance. Guru

• Anonymous
  January 22, 2010
  Hi Paolo, nice blogand interesting entry. I played with the code, it works fine. However, I have 1 question. You indicate "in order to allow the Two-Way Receive Location to receive the response when this latter is published to the BizTalkMsgBox". can you please confirm, according to your demo, how we can display or explout this final response. In the driver, once information is submitted back to "In" directory, if I amnot mistaken, once BTS processes it through the file receive location,a response is sent back to the WCF. So my question is about what this response contains and link it to driver. Thanks in advance

• Anonymous
  January 24, 2010
  The comment has been removed

• Anonymous
  February 10, 2010
  Hi Paolo, Thanks for this great post. I find easily how to transform Two way port by two one way when I use two orchestrations but It was more difficult when I tried with a Port directly. Your post give the solution with the differents properties to promote.

• Anonymous
  September 28, 2010
  It looks like the sample solution in the download is missing the pipeline project. Is that supposed to be there? This was a very helpful post, too. Thanks!

• Anonymous
  October 03, 2010
  Hi mate, I reviewed the SyncToAsync application installed on my machine and I can confirm that none of the Receive Locations or Send Ports use a custom pipeline:

• The SyncToAsync.Request.WCF-NetTcp.ReceiveLocation uses the XMLReceive and PassThruTransmit.
• SyncToAsync.Response.FILE.ReceiveLocation uses the XMLReceive pipeline.
• The SyncToAsync.Request.FILE.SendPort uses the XMLTransmit pipeline. Ciao, Paolo

• Anonymous
  September 16, 2015
  Unfortunatelly, all promoted properties are empty. Map is works but doesn't filled this fields... and i don't knowwhat do to..

• Anonymous
  September 16, 2015
  I have found reason of problem. In mapping this fields was empty - so mapper doesn't create it. I just add empty string to them and after this all properties are filled correct. But now i have another problem: ReqestResponsPort doesn't ake Response from MessageBox by this properties...