A Broker Pipeline Component for End-To-End Ordered Delivery
Artigo
Introduction
Recently I had chance to work with a health care company that makes an extensive use of the BizTalk Accelerator for HL7 for processing a large amount of messages containing clinical and administrative information. Health care companies need a reliable application platform that can ensure medical data is transmitted securely and reliably over their internal networks. They typically rely on the HL7 (Health Level 7) industry standard to transmit medical data (e.g. patient information) to one or many backend systems and need a messaging system that can guarantee ordered message delivery. In addition, their application platform must provide a reliable queuing mechanism which should be able to gracefully handle overload and backlog conditions. When implementing an HL7 solution on the BizTalk Server platform, ordered message delivery can be achieved through the sequential convoy singleton orchestration pattern and the ordered delivery feature of the native and the use of ordered-delivery compliant adapters, such as MLLP (Minimal Lower Layer Protocol), MQSeries or MSMQ adapter. In particular, a sequential convoy enables multiple distinct messages to join together to achieve a required result. A sequential convoy is a set of related messages that have a predefined order and a singleton orchestration can be used to guarantee that incoming documents are processed in rigorous sequential order.
For more information on ordered message delivery and convoys, see the following articles:
"BizTalk Server 2004 Convoy Deep Dive " topic on the MSDN site.
“Working with Convoy Scenarios” topic on the MSDN site.
“Ordered Delivery of Messages” topic on the MSDN site.
"Sequential Convoy" sample on the BizTalk Gurus site.
“Improving Ordered Message Delivery in BizTalk Server 2006 R2 During Concurrent Processing” whitepaper on the Microsoft site.
“Implement End-to-End Ordered Delivery Using Microsoft BizTalk Server and SQL Server Service Broker” whitepaper on the TechNet Wiki site.
Problem Statement
One of the customer requirements was to receive and process incoming messages end-to-end in sequential order. Besides, incoming messages could be routed to a pool of backend systems based on their type (context-based routing) and based on the information contained in the payload (content-based routing). Therefore, the development team of the health care company created a distinct sequential convoy for each backend system. In particular, each orchestration used a logical receive port configured to support ordered delivery and a Listen shape inside a loop to wait for one of several types of message and branch based upon the kind of the inbound document. Unfortunately, this pattern didn’t work out as using multiple Receive shapes within a convoy orchestration does not guarantee ordered message delivery. Hence, the customer adopted another strategy: they reverted back to a convoy orchestration with a single Receive shape that was subscribing and receiving all message types and filtering out those documents that were not intended to be consumed by the backend system. However, this solution was very costly in terms of performance, because each message was sent to a large amount of convoy orchestrations, but only a few of them were indeed processing the document in question.
Solution
To solve the problem and allow inbound documents to be dispatched only to the intended convoy orchestrations, I suggested an alternative architectural approach where the target systems are determined on the receive location based on the rules defined in a Business Rule Engine policy. The idea behind my solution is pretty straightforward. I created a Property Schema which defines a promoted property for each backend system. Then I created a convoy orchestration for each of these systems. In particular, the orchestration contains a single receive shape within a loop which is able to receive any kind of messages. To implement this pattern, I defined the inbound document of type XmlDocument. Each convoy orchestration uses a filter expression of the form <backend system promoted property> exists to receive only those messages that are effectively destined to the related backend system. The core component in my solution is represented by a pipeline component called BrokerPipelineComponent that runs within the receive location. This object invokes a Business Rule Engine policy to determine the subscribers of the current message based on their type. In particular, the business rules defined in the policy use the message content and context to promote the properties associated with the target systems. To validate the effectiveness of my ideas, I created an end-to-end solution depicted in the following picture.
Message Flow:
A Unit Test running inside Visual Studio 2010 submits a configurable amount of XML messages of 3 different types (Dvd, Cd, BluRay) to a transactional MSMQ queue. The 3 document types are represented by 3 different XML schemas in a BizTalk library called Schemas. The Schemas project contains also a PropertySchema where I defined a custom context property for each convoy orchestration.
A WCF-NetMsmq receive location reads the XML messages from the queue.
The XML Disassembler component within the BrokerReceivePipeline pipeline promotes the MessageType context property. Then the BrokerPipelineComponent applies the business rules contained in the BRE policy to determine the list of the context properties to promote, one for each intended sink for the message. The name of the policy to apply can be configured in the component properties. The component allows also to cache rules for each MessageType. In this case, the BRE policy only the first time that a certain message type is encountered, whereas the subsequent times the rule outcome will be retrieved from the in-process cache. This mechanism allows to speed up the subscription matching and message dispatching process. Finally, the TracePipelineComponent can be enabled to write the message content and context to the standard output. This allows to monitor the the behavior of the BrokerPipelineComponent using a tool such as DebugView.
The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb).
The inbound request starts a new instance of the SupplierA and/or SupplierB convoy orchestrations which, in my sample, handle the messages destined to a couple of distinct partners. As mentioned above, each convoy orchestration uses a filter expression of the form <backend system promoted property> exists to receive only those messages that are effectively destined to the related backend system. Both orchestrations share the same structure and in particular use a Call Orchestration shape to invoke the Tracer orchestration. The latter writes the document content to the standard output, applies a transformation map to it and sends the new message to a solicit-response send port.
The orchestration publishes the message to the MessageBox (BizTalkMsgBoxDb).
The message is retrieved by the a solicit-response WCF-SQL send port. There are 2 distinct send ports called respectively SupplierA and SupplierB that simulate the communication with 2 different backend systems.
Each send port invokes a distinct stored procedure that writes the information contained in the inbound document to the table associated with the related backend system. These tables are called SupplierA and SupplierB and are both located in the same SQL Server database called OrderedDeliveryBroker.
The send port publishes the response message to the MessageBox (BizTalkMsgBoxDb).
The response message is returned to the convoy orchestration that waits for the next message.
In the following section we’ll examine the artifacts that compose the end-to-end application.
Message Queue
The following picture shows the configuration of the transactional MSMQ queue used by the BizTalk application to receive the inbound documents sent by the unit test running in Visual Studio 2010.
OrderedDeliveryBroker Queue
Schemas
The OrderedDeliveryBroker application receives 3 different types of message that have the following format:
Cd Message
<Cd xmlns="https://schemas.appfabric.cat.microsoft.com/ordereddeliverybroker/2011/cd"> <Id>112</Id> <Name>The fat of the land</Name> <Price>4.99</Price></Cd>
The Schemas project also contains a PropertySchema that defines 2 custom context properties, one for each target backend system:
Receive Location
To retrieve messages from the MSMQ queue I defined a WCF-NetMsmq receive location. The picture below displays the URI of the queue.
WCF-NetMsmq Receive Location Transport Properties
The following picture shows that the Transactional and Ordered Processing features have been turned to increase the application reliability and guarantee ordered message delivery.
Binding Configuration
The following pictures shows the configuration of the BrokerReceivePipeline. In particular note the name of the Policy used by the BrokerPipelineComponent.
BrokerReceivePipeline Configuration
Orchestrations
The following picture shows the SupplierA convoy orchestration. The SupplierB orchestration shares the same structure. In particular, the Ordered Delivery property of the SupplierReceivePort is set to true to assure that incoming messages are processed in sequential order.
SupplierA Orchestration
The following picture shows the structure of the Tracer orchestration invoked by the SupplierA and SupplierB orchestrations. This orchestration writes the document content to the standard output, applies a transformation map to the inbound message passed as argument and sends the resulting message to a solicit-response send port.
Tracer Orchestration
Send Ports
The OrderedDeliveryBroker application uses 2 distinct WCF-Custom send ports to write messages to the SupplierA and SupplierB tables in a custom SQL Server database. Both send ports are configured to use the sqlBinding. The picture below shows the configuration of one of the 2 send ports. In particular, you can notice that the send port is configured to invoke a stored procedure called supplierA_InsertLogRetry .
The following picture shows the OrderedDeliveryBroker database and in particular structure of the SupplierA and SupplierB tables used to simulate two distinct backend systems.
Business Rule Engine Policy
The following picture shows the OrderedDeliveryBroker policy that defines the routing rules used to dispatch messages just to the intended convoy orchestrations.
OrderedDeliveryBroker Policy
BrokerPipelineComponent
The BrokerPipelineComponent represents the core component of the application. It invokes the BRE policy configured on the receive location and then promotes the list of context properties returned by the rule set to dispatch the inbound document only to the intended sinks. For your convenience, I included below the code of BrokerPipelineComponent. For brevity, I omitted the code of the ContextDictionary class used to define rules in the BRE engine and return results to the BrokerPipelineComponent.
To test the application, you can proceed as follows:
-
Build, deploy and configure the OrderedDeliveryBroker solution.
-
Open the Test View as illustrated in the picture below.
-
Run the SendMessage unit test. The code of the test method creates a blend of Cd, Dvd and BluRay messages and assign an increasing Id to the documents. This allows to verify that messages are received in the same chronological order with which they are submitted to the MSMQ queue.
-
Open the SQL Server Management Studio and execute a query to verify that the SupplierA and SupplierB tables contain the expected data in the right chronological order.
Conclusions
In this article we have seen how to create a broker pipeline component and Business Rule Engine policy to dispatch messages to a pool of convoy orchestrations. Here you can download the companion code for this article. As always, you feedbacks are more than welcome!
Comments
Anonymous
October 08, 2011
Nice article.
Anonymous
December 26, 2013
Hi Paolo,
I have the same situation (problem statement). Can you please explain in detail, the sentence 'using multiple Receive shapes within a convoy orchestration does not guarantee ordered message delivery' ?
Thanks in Advance!
Anonymous
January 06, 2014
Hi Syed
The sentence means that if you receive documents using multiple receive shapes within a convoy orchestration, each bound to a different receive location, this approach doesn't guarantee that incoming messages are processed in strict chronological order.
Ciao
Paolo