Pipelines in BizTalk 2006
The BizTalk adapters take up the responsibility of interacting with the end systems and getting the data. Now, it is the pipelines that come into the picture and transform the data into something that BizTalk can understand – Xml. Essentially, pipelines form a channel for the messages from the adapters to the message box where they are finally delivered. After it leaves the pipeline, the messaging engine is responsible for the further processing of the messages. The processing can include routing or executing a business process – as has been defined by the developer.
The messaging engine works on a certain format of data that it terms as a message. This is necessarily a XML message with a specific format containing the message headers and body parts. The Adapter is explicitly responsible for interacting directly with the end systems. To isolate the workload, the adapter does not take care of any kind of formatting of the message so that the message is understood by the Messaging Engine. This is exactly the reason where the pipelines play their role. The pipelines accept the data from the adapters and work on it to build up the Message that can be understood by the messaging engine.
The clear cut advantage that it provides here is of course the simplified processing for the BizTalk messaging engine. Even though the messaging engine deals only with plain simple XML messages, BizTalk as a product can still deal with messages from external systems that are in various formats – like a digitally signed message, or an encrypted message or a comma separated flat file, a MIME message and several other things that would’ve otherwise complicated the workings of the messaging engine.
Having said that, the pipelines form an integral part of the overall BizTalk solution development and hence their understanding is vital for any BizTalk developer.
Usually, for simple scenarios, you can simply pick one of the pre-built pipelines that are supplied by the BizTalk by default. However, as the scenarios get complex, there is a significant need to have a custom pipeline developed that can cater to the specific business needs. Developing a custom pipeline can involve simply creating a new pipeline using the standard pipeline components available in a pipeline editor or it can turn out to be a much more complex affair where you would need to create a custom pipeline component using a set of .Net based APIs that are provided for the pipeline development.
What to expect here |
As we move ahead, we will take a look at what the pipelines are and how they work. We will walk through the standard pipelines that are available out of the box with BizTalk and in what scenarios can they be used during the development. We would also take a look at various functional requirements that would need you to create custom pipelines before taking a closer look at how we can develop them. |
Getting the Concepts Right
The pipeline draws its concepts from those used in the Commerce Server. In the BizTalk context, the pipeline is responsible for making the message “BizTalk-ready”. As the message travels through the adapter to the pipeline, the pipeline executes a series of stages. At each of these stages, the pipeline executes a series of pre-built or custom built components that help in getting the message in the right format. This can include decryption of a message, verification of the signatures, verifying the sender who has sent the message, decoding the message format from something like MIME or plaintext to an Xml message and several other things that can be custom built with the help of the exposed APIs.
Each of the stages can contain one or more components in it. The pipeline can be configured to execute all these components or just one of them. The BizTalk messaging engine controls the execution of the pipeline as it passes the message from one stage to another.
As a logical statement, the pipeline exists wherever the BizTalk engine needs to communicate with the external systems. Hence, a pipeline can come into the picture either when the BizTalk is receiving the message from the adapter or when it is sending the message to the adapter, just to be sent to the external system. Obviously, the order of the stages in the pipeline and to some extent also the processing being done in the stages will definitely differ depending on whether the message is being received in BizTalk or being sent out of the BizTalk. To cater to this requirement, BizTalk Server offers two types of pipelines – the Receive pipeline and the Send pipeline. As the name suggests, the Receive pipeline plays its role whenever any message is being sent BizTalk from any of the LOB applications. It expects a raw inbound message from the adapter, decodes and decrypts it if necessary, and translates into an Xml to be understood by the BizTalk engine. The Send pipeline obviously works the other way when the message is being sent out of the BizTalk paradigm. It complements the receive behavior by assembling the Xml message received from the BizTalk engine in a format that can be understood by the LOB applications. Also, if needed, the Send pipeline and encrypt and digitally sign the message before it is being sent.
BizTalk provides a set of default pipelines that can work for some trivial scenarios. However, as the processing gets complex, there is definitely a need for development of custom pipelines or custom pipeline components. The Visual Studio integration provides a Pipeline Designer that can be used to create the custom pipelines. It gives an easy to use drag and drop way of creating custom pipeline components. The Pipeline Designer provides templates for creating the Send and Receive pipelines. These templates ensure that the stages are in proper order and the pipeline components are dropped only on the appropriate stages.
BizTalk developers are likely to encounter custom pipeline development when it comes to integrating some non-trivial systems. Of course development of a custom pipeline cannot be complete without the use of the pipeline designer and knowing how it varies for Receive and Send pipelines. The toolbox in the pipeline designer provides some out of the box pipeline components that can be dropped on the designer surface. The designer enforces the order of the stages and the constraints for each of the components. The components can be tweaked to suit the development needs by the array of properties exposed by them. In some cases however, the out of box components may not suffice. Typically, this would happen when you have a legacy application with its own format that cannot be interpreted by any of the components. In such cases, BizTalk exposes a set of interfaces that allow the developers to build their own pipeline components. This typically involves creating a custom .Net class library assembly that uses the exposed interfaces. This assembly can then be integrated with the designer toolbox and the component can be used as any other pipeline component.
Points to remember |
|
Understanding Receive Pipelines
Now that we have a fair idea of what the pipelines are, let’s delve deeper into the receive pipelines to understand them better. As we know by now, the receive pipelines help the message to be “BizTalk-ready”. It converts the legacy format message into an Xml and then publishes it to the message box for the messaging engine to process further.
The receive pipeline is executed after the receive adapter. It takes up the chunk of raw data that comes from the external system and works on it performing tasks like validation the signatures, decryption, transformation before disassembling it into zero or more BizTalk messages. The created messages are consumed individually by the BizTalk server.
The Receive pipeline consists of four stages – Decode, Disassemble, Validate & Resolve Party. These are executed in the same sequence as they appear above. When building a custom pipeline, it is not mandatory to build all the stages. So if the functionality of a particular stage is not required, it can be simply left empty. In the following section, we will now take a closer look at what each stage does to the message and how a complete, valid BizTalk message is formed at the end of it.
Decode Stage
Before starting to work on the message, it is a logical step to get it ready so that you can work on it. Typically, this stage would involve some pre-processing like verifying message signatures to ensure integrity, or decoding the S/MIME messages. Of course, this stage can have multiple components. For example, if a message is encrypted and then sent over digitally signed, then this stage would have two subsequent components for each task. The components in this stage are executed sequentially. The stage can have a maximum of 255 components. Needless to say, each additional component would come with a performance hit. Once all the components process the payload, the output of this stage is presented to the next stage – Disassemble stage. The output of this stage would be a stream that can be processed further in subsequent stages.
BizTalk Server 2006 comes with the MIME/SMIME decoder as a standard component for this stage. This component can take up the messages and attachments (if any) in the MIME or Secure MIME format. It can also verify signatures and decrypt the S/MIME messages. The component can convert the messages of both types in XML format.
Disassemble Stage
Once done with the decoding stage, the message is passed to the next stage that is the Disassemble stage. BizTalk thinks of a message as having a header and zero or more bodies. We might even be looking at a message which contains several messages within it. This is typically known as an envelope that contains several messages within it. In BizTalk lingo, it is termed as an ‘Message Interchange’. For an Interchange, BizTalk can even chop down the actual message in several small messages. These ‘child-messages’ can then be processed independently by the BizTalk messaging engine. An interchange can typically result in a scenario where the external system processes the requests in batch. Say the system processes 100 requests in a batch and then bundles them together in a single envelope. So, it forms a single message with a header and containing 100 message bodies. The traditional legacy systems work in a similar way due to the bandwidth constraints that were prevalent till near past. BizTalk works with a single message at a time and hence, the interchange processing feature helps to accommodate the workings of such systems as well.
It is here that the message gets converted entirely into the Xml format so that the messaging engine can understand it. The input is, of course a plain text or an Xml stream that is passed from the Decode stage. It is processed into one or more messages that are always in Xml format.
BizTalk 2006 comes with three standard components for this stage. The Xml Disassembler, Flat file Disassembler & the BTF Disassembler. The Xml Disassembler will be used in the simplest of scenarios where the incoming message is already an Xml. There is no formatting required to this message to make it ‘understandable’. The Flat file Disassembler is used in scenarios where the incoming message is a flat file. It can cater to positional as well as delimited files. The component exposes a property for the flat file schema. The incoming message is converted from flat file to Xml on basis of this flat file schema. The BTF Disassembler is one of the least popular. It accepts the messages that were sent using the reliable messaging mechanism. This mechanism is defined in the BizTalk framework and was implemented in previous versions of the BizTalk Server (BizTalk Server 2000). The BTF component also requires the orchestration to support the reliable messaging semantics.
New Features – Recoverable Interchange Processing |
In the BizTalk 2004, when processing the Interchange, if one of the messages in the interchange was bad, the entire Interchange was rejected. Carrying forward the same example, if the pipeline gets a batch of 100 messages, the disassemble stage processed it as per the defined envelope and body schemas. However, if the 97th message in the Interchange was bad, then entire Interchange of 100 messages was ignored. This functionality, although right for some solutions, caused a lot of problems in others. BizTalk 2006 fixes this issue by adding a ‘Recoverable Interchange’ flag to the component. In case of a recoverable Interchange, if one of the messages in the Interchange is bad, then it is ignored, however, the rest of the messages still go through. The best part though is that the Recoverable Interchange flag can also be set from the BizTalk Administration Console without having to recompile your pipelines assembly. |
Validate Stage
The disassembler stage can contain a lot of processing for the incoming message. Even though the output is expected to be an Xml stream, the Validate stage gives an opportunity to look into validity of the message before it lands in the message box. Typically the validate component would check to see if the message is in correct format or not. It can compare the message against a specific set of schemas, failing to which the message will not be allowed to process further. This component provides a good way of checking if the disassembler stage results in a desirable output or not. In case an incorrect format is generated, the message can be simply ignored instead of causing a routing failure after it reaches the messaging engine.
The BizTalk 2006 provides the Xml Validator component at this stage. This component validates the output of the Disassembler stage against one or more defined schemas.
Party Resolution
This stage, as the name suggests, resolves the party who has sent the message. If the incoming message was digitally signed, then this stage looks up the windows identity of the sender in the BizTalk Server’s Management database. If this stage is not able to determine the identity of the sender, then the default Anonymous identity is used. Once the identity is determined, this stage is responsible for setting the SourcePartyId context property for the message. This property can later be used in the orchestrations or Subscription filters to incorporate business logic or message routing.
The BizTalk 2006 comes with a standard component called Party Resolution that helps achieve this functionality. This stage can have as many as 255 components.
Points to remember |
|
Understanding Send Pipelines
With the receive pipelines, we have the messages coming in to the BizTalk paradigm. Once these messages are published in the message box, they can be consumed by the orchestrations for the workflows or can be routed directly to the Send Ports in case of a simple routing scenario. Irrespective of the flow, whenever the message wants to get out of the BizTalk paradigm to some other external system, it has to pass through a send pipeline that will make it ready. The Send pipeline usually subscribes to the messages based on their properties and then work on the messages that land with them. These are then forwarded to the actual adapters that take care of sending them to the actual end systems. It expects a complete well-formed xml message from the BizTalk engine and then works on it to assemble it into a message that the external system can understand. Complimenting the receive pipeline, this can again be a flat file, it can also be encrypted and digitally signed before sending it out on the wire.
The Send pipeline consists of three stages –Pre-assemble, Assemble & Encode. These are executed in the same sequence as they appear above. When building a custom pipeline, the same rules as the receive pipeline, apply to this. It is not mandatory to build all the stages. So if the functionality of a particular stage is not required, it can be simply left empty. In the following section, we will now take a closer look at what each stage does to the message and how it transforms the BizTalk message into something that the legacy systems can understand.
Preassemble Stage
As the message comes to the Send pipeline, this is the very first stage that gets executed. This is basically a place holder stage that can do any pre-processing of the message before it needs to be assembled. The stage can contain up to 255 components. For each message that enters the pipeline, this stage is executed exactly once.
The BizTalk Server does not provide any components for this stage. Hence, if this stage has any component, it is always a custom component.
Assemble
Once the preassemble stage does its job, the assemble stage takes over to actually convert the message in a format that can be understood by the external system. This can be a flat file format or an XML to which this method adds an envelope. The input to this stage is what is sent by the runtime engine and modified by the preassemble stage, if applicable. The components in this stage work with schemas that specify how the message has to be transformed. This stage however, can have only a single component within it.
Complementing the Disassemble stage in the receive pipeline, BizTalk provides three components for this stage out of the box. It comes with the Xml Assembler to build the Xml messages. The Flat file assembler helps build flat file messages with help of the flat file schemas provided as a part of its configuration. The BTF Assembler again helps compatibility for the reliable messaging that was a part of BizTalk 2000.
Encode
As the message is converted to the external format, it is time to do any additional processing required on the message so that the external system can decode it properly. The processing of this stage includes S/MIME encoding, custom encryption of the message or digitally signing it to maintain its integrity or any other processing that can be required to encode the message. This stage is typically related to the security measures that you would take when sending over the message on the wire. Just like the Decode in the receive pipeline, the stage can contain up to 255 components. All these components are executed for every message that comes in.
BizTalk Sever 2006 packs along with it the MIME/SMIME encoder. As the name suggests, the component can package the message in MIME or S/MIME format. If using S/MIME, the message can also be digitally signed or encrypted.
Points to remember |
|
Default Pipelines
As we move ahead, we now have an idea of what the pipeline is and what it does to a message. BizTalk of course comes with some out-of-the-box pipelines that would help you in achieving some of those basic scenarios. Dealing with non-trivial scenarios would anyways require a custom pipeline. With this section, let’s take a look at what BizTalk provides out of the box and let’s try to understand what all things they can help achieve.
BizTalk Server 2006 packs with it the Pass-through Send & Receive pipelines and the XML Send Receive pipeline. The Pass through pipelines do not perform any processing on the message. It can be used in scenarios where the message is simply routed and not processed by the messaging engine in any way. Because it does not do any processing on the message, the pass through pipelines assumes that the message is in clear-text and does not require any validation.
These pipelines are normally used in a secured environment wherein the messages need not be decrypted or validated once they are received. Because of its pass through nature, it can be useful in integrating the legacy applications that have already been integrated to some extent.
The Xml Send and Receive pipelines handle the messages that are in XML format. The receive part of this pipeline disassembles the XML into a BizTalk message and makes it ready for the BizTalk messaging engine to process. The party resolution stage in the pipeline also contains a component that resolves the identity of the message source before it is published to the message box. The Send counterpart simply assembles the message as an XML and passes it ahead to the adapter. This pipeline also does not do any processing for the message security. It assumes the message to be in clear-text format. Hence, just like the pass-through pipelines, this pipeline is also expected to be used within a secured environment. Unlike the pass through pipelines, the XML Send Receive pipelines allow basic message verification. The only criteria to using these pipelines is of course that the messages – incoming and outgoing have to be in XML format.
Summary
Pipelines in BizTalk act as tunnels through which the messages coming into the BizTalk Server and going out of the BizTalk Server travel through. The pipelines offer the flexibility of adding some processing that can make the message ready for BizTalk or the associated external system, The pipelines form an integral part of the BizTalk Architecture as they help simplifying internal message processing for BizTalk by allowing it to deal with Xml only.
BizTalk Server 2006 introduces the concept of Recoverable Interchange Processing (RIP) that enhances the processing for batch messages with envelope schemas. RIP can be configured using BizTalk Administration Console without recompiling the pipelines.
In a nutshell, BizTalk 2006 enhances the pipeline configuration mechanism to provide ease in development and administration.
Comments
Anonymous
June 12, 2006
Thats my first attempt to write some big stuff. This one looks at introducing everybody to the Biztalk...Anonymous
June 19, 2006
The comment has been removedAnonymous
February 28, 2007
Good Article Provides a clear understanding on what exactly a pipeline biztalk mean.............Anonymous
April 04, 2007
Great Article, Please keep posting more...Anonymous
May 23, 2007
Very interesting article. Was very helpful. Thanks!Anonymous
July 11, 2007
Article is nice. How can i create a customer pipeline?Anonymous
August 07, 2007
The comment has been removedAnonymous
October 24, 2007
Very interesting article. Was very helpful. Thanks!Anonymous
November 13, 2007
very confusing and too large to read.... Plz keep ur article concise and avoid repetitive copy pasting next timeAnonymous
January 14, 2008
Very nice article, explains pipeline from scratch...Anonymous
May 01, 2008
Hi, I am learning Biztalk Server 2006, I got a book. I want to buid an application say read xml file from some location and update table in a data base through BizTalk server and then generate a report using ssrs 2005 to a location on hard disk. I want to know the things I should do, in the book i purchased, they are eplaining to use existing templates. but i want to go for it without using any templates. please give me some reference. thanks and regards Ravi Kiran KaranamAnonymous
June 06, 2008
Explains pipelines in detail and is simple to understand. Thanks!Anonymous
June 08, 2008
Nice explanation and good detail about Biztalk and pipelines. Great info. You might want to also talk about error handling and how it can work in conjunction with your pipelines. Again excellent information. KevinAnonymous
April 22, 2009
Great article. Thanks!. One thing am not able to understand is - how pipelines are associated with ports. Say i have 2 receive ports, both will receive xml files, but only for 1 i want to assemble, for other i want it to pass just as it is. I did not understand how it is tied. Can you please explain that. Thanks for your help.Anonymous
May 13, 2009
Prakash, Each recieve port can be associated with a pipeline. In the case that you mentioned, you can use an XML recieve pipeline in the first case and a pass through pipeline in the second.Anonymous
March 23, 2010
Good Article. i am a beginner. Now i understood what is pipeline.Anonymous
April 14, 2010
Hi could any one tell me at which stage of pipeline the properties get promoted.as i can promote properties at for example... in case of receive pipeline.... a) a properties can be promoted at decode dissassemble ...but after that validate stage is there... so it get promoted at decode dissasemble or after teh validate.... bizashwani@gmail.comAnonymous
April 14, 2010
The comment has been removedAnonymous
May 04, 2010
hii..sanket a great article for the beginners... keep it upAnonymous
December 27, 2012
nice article