BizTalk Server: Custom pipeline optimization using the Virtual Stream class

Article
1/17/2024

Introduction

Performance of pipelines in BizTalk can be enhanced by applying best practices listed in the Optimizing Pipeline Performance article on MSDN. In case you are building a custom pipeline with custom pipeline component that this article is one you definitely need to read before you design, build and deploy your custom pipeline. When requirements drive you to build functionality for inbound messages in a custom pipeline you according to article leverage the VirtualStream, ReadOnlySeekableStream, and SeekAbleReadOnlyStream class found in the Microsoft.BizTalk.Streaming.dll assembly. The VirtualStream class for instance has the same functionality as a MemoryStream class in the System.IO namespace, however it uses disk space to store large streams instead of memory. Therefore, the benefit of using the VirtualStream class over XmlDocument is that you can avoid OutOfMemory issues, since you do not need to load the entire document into memory using an object.

Developers, who need to build a custom pipeline yet lack awareness of the Microsoft.BizTalk.Streaming.dll assembly, will use the XMLDocument since it has been widely documented. Most developers are quite familiar with this class. An instance of the XmlDocument class consumes a large amount of memory, which can be up to 10 times the size of the actual message size. With a substantial load of (large) messages the custom pipeline component can become a performance bottleneck.

Scenario

Let’s say you need to change the namespace of a certain type of messages coming in from a business partner. The business partner cannot based on certain constraints change the namespace of the messages it’s sends out to you. You receive messages through your middleware (BizTalk) and need to change the namespace for it can be published in the Message Box. The size of the messages the partner send out are a substantial size i.e. 1 to 5 Mb and the frequency quite high particular moments of the day. You like to change the namespace in a custom pipeline in the most optimal way. Therefore, you build a custom pipeline component that leverages VirtualStream to replace the namespace based on runtime properties of a custom pipeline. The custom pipeline is basically a XML_Receive type of pipeline with an additional pipeline component responsible for replacing the old namespace (from business partner) to a new namespace based on the requirements of your enterprise.

This example pipeline component could be improved by reading and transforming the message in an XMLReader and XMLWriter rather than writing it to a string. Using a string could use a lot of memory for large messages but it may be acceptable for small messages and a low volumes. See https://connectedcircuits.wordpress.com/2013/02/03/xmltranslatorstream-to-the-rescue/ for an implementation that uses XMLTranslator. XMLTranslator uses XMLreader and XMLWriter under the hood.

ReplaceNamespace custom pipeline component

The custom pipeline component is implemented according to the code displayed below. It will replace the old namespace (based on specification of the _oldNamespace) with a new namespace (_newNamespace). This will occur in case the property _newNamespace is set to true.

public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            try
            {
                //If enabled
                if (Enabled)
                {
                    //Create StreamReader, VirtualStream and StreamWriter instance
                    StreamReader sReader = new  StreamReader(pInMsg.BodyPart.Data);
                    VirtualStream vStream = new  VirtualStream(); 
                    VirtualStream vStreamOutput = new VirtualStream();
                    StreamWriter sWriter = new  StreamWriter(vStream);
 
                    //Write message body to a virutal memory stream
                    sWriter.Write(sReader.ReadToEnd());
                    sWriter.Flush();
                    sReader.Close();
 
                    vStream.Seek(0, SeekOrigin.Begin);
 
                    //create a string
                    string body = new StreamReader(vStream).ReadToEnd();
 
                    //Add property to Namespace.
                    //Manipulate the Namespace of the message
                    body = body.Replace(OldNamespace, NewNamespace);
                    vStream.Position = 0;
 
                    //Write the output
                    StreamWriter writer = new  StreamWriter(vStreamOutput);
                    writer.AutoFlush = true;
                    writer.Write(body);
                    vStreamOutput.Position = 0;
 
                    //Put the stream back
                    pInMsg.BodyPart.Data = vStreamOutput;
 
                }
 
                return pInMsg;
            }
            catch (Exception ex)
            {
                if (pInMsg != null)
                {
                    pInMsg.SetErrorInfo(ex);
                }
 
                throw ex;
            }
        }

Building the custom pipeline

Once the pipeline component is ready to be used in a custom pipeline you will need to deploy the pipeline component. In the pipeline component you perform create a post-build event like below to assist in a proper deployment.

"C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\gacutil" /nologo /i "$(TargetPath)" /f
xcopy "$(TargetPath)" "C:\Program Files (x86)\Microsoft BizTalk Server 2013\Pipeline Components" /d /y

This post built event will GAC the pipeline component and deploy it in pipeline component directory of the BizTalk installation. Once that is done you can add the custom pipeline to the Visual Studio toolbox.

http://i208.photobucket.com/albums/bb152/Steef-Jan/VirtualStream2_zpsd067efd6.png

Picture 1. Adding custom pipeline component to Toolbox.

Once the custom pipeline component is available in the toolbox you can drag onto the canvas into one of the stages of the custom receive pipeline.

http://i208.photobucket.com/albums/bb152/Steef-Jan/VirtualStream3_zpse842c4d5.png

Picture 2. Placing the custom pipeline component in one of the receive pipeline stages.

Source Code

Sample code to leverage the Virtual Stream Class see MSDN Code Gallery: BizTalk 2013 - Custom pipeline optimization using the Virtual Stream class.

Share via