Share via

BizTalk Server: Dynamic Schema Resolver Real Scenario

  • Article

Introduction

As a BizTalk consultant, you had an assignment that a service provider asked you to enhance the existing BizTalk solution.

You checked the current implementation and you found that the existing BizTalk solution as shown in Figure 1

**Figure 1 **- Current BizTalk Solution Architecture

They are receiving different flat files from different customers and each customer has a different flat file format. All customers send these flat files in one receive path with a specific unique file name format ####YYYYMMDDHHMMSSNNN.txt

The first 4 digits are alphanumeric of customer code.

Each customer publishes a message to BizTalk Engine and other send ports subscribe these messages.

In this article, I will focus on the dynamic resolver engine which is specifically in receive and send pipelines

Bad Solution

The implementation required the following steps to add a new customer to the system:

Adding New Source Steps

  1. Add a new flat file schema matching the structure of a new customer as shown in Figure 2

    **Figure 2 **- Sample Source Schema

    Figure 1- Current BizTalk Solution Architecture

  2. Add a new receive pipeline, add flat file disassembler and set Document Schema property to Flat file schema the created in step 1 as shown in Figure 3


    **Figure 3 **- Receive Pipeline

  3. Add a new receive location and set Receive Pipeline to the new receive pipeline that created in step 2 as shown in Figure 4

    **Figure 4 **- Set Receive Pipeline to receive location

  4. Click on configure button and set File mask as shown Figure 5


    **Figure 5 **- Set File Mask

In case there is a new Destination then you need to do the following steps 

Adding New Destination Steps

  1. Add a new flat file schema matching the structure of a new destination as shown in Figure 6

    **Figure 6 **- Sample Destination Schema

  2. Add a new send pipeline , add flat file assembler and set Document Schema property to Flat file schema the created in step 1 as shown in Figure 7

    Figure 7 - Send Pipeline

  3. Add a new map between source and destination like sample as shown in Figure 8

    Figure 8 - Sample Map between source and destination

  4. Add a new Send Port in BizTalk Administration as show in Figure 9

    **Figure 9 **- Send Pipeline in Send port

  5. Click configure and set file name format as shown in Figure 10

    **Figure 10 **- Setting File Name Format

  6. Select Outbound Maps and select the correct map as shown in Figure 11

    **Figure 11 **- Setting Outbound Map

  7. Select Filters and Set BTS.ReceivePortName == Rcv_Messages which is the name of receive port as show in Figure 12

**Figure 12 **- Setting Filter in send port

Problem

Now imagine that you want to add 100 customers as a source then you need to add 100 of Receive Pipelines artifacts and you need to repeat steps of 2-4 of New Source Steps section 100 times. Another issue, imagine that you want to add a new 100 destination then you need to create 100 of Send Pipelines artifacts and you need to repeat steps 2-7 of New Destination Steps section 100 times. 

Good Solution

We need to implement a dynamic schema resolver which is a mechanism of dynamically associate message types to inbound instances based on an identifier contained in each instance message.

We need to build a custom pipeline that takes a message type from configuration database, like Business Rule Engine or using your custom configuration database table and set Document Schema to dis-assembler using the unique key that distinguishes your source message, you can use context properties of the message or using any unique data of the content of the message.

In our case the unique key is the customer code which is the first 4 digits of the file name

For send custom pipeline, we have the xml message inside BizTalk engine which contains the schema type, we just need to set document schema into assembler at runtime

Figure 13 shown the Good solution architecture

**Figure 13 **- Good Solution Architecture using Dynamic Resolver Mechanism 

We need to do the following implementation steps

Creating BRE Policy

  1. Create new schema with 2 element one for Source Code and one for message type(Namespace#RootNode) as shown in **Figure 14 **

    **Figure 14 **- BRE Schema

  2. Create a new Policy SchemaResolverPolicy and add a rule for each source as shown in Figure 15

**Figure 15 **- Sample BRE Policy

Create Custom Pipeline Component

We need to create a custom pipeline for receiving and sending pipeline as the following steps

Create Custom Receive Pipeline

  1. Add a new class library project as shown in Figure 16

    **Figure 16 **- Add new class library to contain pipeline components

  2. Add a signature to the assembly, to allow assembly to be put into the GAC. To do this, right-click the new project and choose Properties. Then go to the Signing tab, as shown in Figure 17

    **Figure 17 **- Adding a signature to the class library

  3. To create classes that BizTalk recognizes as pipeline components, you need to implement some specific interfaces. These interfaces are specified in the Microsoft.BizTalk.Pipeline namespace. You therefore need to reference the assembly that contains this namespace and these interfaces. Right-click your project or on the References node in your project and click Add Reference. Go to the Browse pane and browse your way to <InstallationFolder>\Microsoft.BizTalk.Pipeline.dll and add this as a reference

  4. To delegate call to Probe method which is implemented to investigate the message to decide if message is recognizable or not. If the message is recognizable then it returns “True” otherwise “False”. We need to add <InstallationFolder>\Microsoft.BizTalk.Pipeline.Components.dll.

  5. To call BRE api we need to reference to <installation drive>:\Program Files\Common Files\Microsoft BizTalk\Microsoft.RuleEngine.dll

  6. Delete the Class1.cs file that was automatically created for you and instead add a new class to your project by right-clicking the project and choosing Add, New Item. Choose Class as the type and provide a name for your class as show in Figure 18

    **Figure 18 - ** Adding a new class to your class library.

  7. Mark your class as a public and decorate it with some attributes Guid telling the pipeline designer how the component can be used in our receiving custom pipeline we need to use:

    CategoryTypes.CATID_PipelineComponent This category does not affect which stages a component can be placed in, but merely serves as an indication for the pipeline designer that this is a pipeline component.

    CategoryTypes.CATID_DisassemblingParser Components in this category can be placed in the Disassemble stage of a receive pipeline and we are assigning document schema in disassemble stage. To use the attribute you need to use namespace Microsoft.BizTalk.Component.Interop as show in Figure 19 

  8. using Microsoft.BizTalk.Component.Interop;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
 
namespace TechNetWiki.SchemaResolver.PipelineComponent
{
    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]
    public class  SchemaResolverFlatFileDasmComp
    {
    }
}

    **Figure 19 **- Adding Pipeline attributes

  9. Decorate your class with GUID attribute by copy a new guid from TOOLS Create GUID As shown in Figure 20 and add namespace System.Runtime.InteropServices.

    **Figure 20 **- Create new GUID

  10. All custom pipeline components must implement the following two interfaces:

    • IComponentUI, which lets you specify an icon for the component and a method for validating any properties the user can set values in
    • IBaseComponent, which basically lets you specify three properties that the pipeline designer uses to get a name, version, and description from your component to show in the designer As shown in Figure 21
    using Microsoft.BizTalk.Component.Interop;
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;
namespace TechNetWiki.SchemaResolver.PipelineComponent
{
    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]
    [Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]
    public class  SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI
    {
        #region IBaseComponent Members
        public string  Description
        {
            get
            {
                return "Flat file disassembler which resolves schemas using message context";
            }
        }
        public string  Name
        {
            get
            {
                return "Receive Pipeline Schema Resolver";
            }
        }
        public string  Version
        {
            get
            {
                return "1.0";
            }
        }
        #endregion
        #region IComponentUI Members
        public IntPtr Icon
        {
            get
            {
 
                return System.IntPtr.Zero;
            }
 
        }
        public IEnumerator Validate(object obj)
        {
            return null;
        }
        #endregion
    }
}

    **Figure 21 **- Implement  IBaseComponent, IComponentUI interfaces

  11. Implement interface IDisassemblerComponent which exposes two methods needed for a disassembler as shown in Figure 22

    using Microsoft.BizTalk.Component;
using Microsoft.BizTalk.Component.Interop;
using Microsoft.BizTalk.Message.Interop;
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;
namespace TechNetWiki.SchemaResolver.PipelineComponent
{
    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]
    [Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]
    public class  SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI, IDisassemblerComponent
    {
        #region IBaseComponent Members
        public string  Description
        {
            get
            {
                return "Flat file disassembler which resolves schemas using message context";
            }
        }
        public string  Name
        {
            get
            {
                return "Receive Pipeline Schema Resolver";
            }
        }
        public string  Version
        {
            get
            {
                return "1.0";
            }
        }
        #endregion
        #region IComponentUI Members
        public IntPtr Icon
        {
            get
            {
 
                return System.IntPtr.Zero;
            }
 
        }
        public IEnumerator Validate(object obj)
        {
            return null;
        }
        #endregion
 
        private FFDasmComp  disassembler = new FFDasmComp();
        #region IDisassemblerComponent Members
        public IBaseMessage GetNext(IPipelineContext pContext)
        {
            // Delegate call to Flat File disassembler
            return disassembler.GetNext(pContext);
        }
 
        public void  Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            // Delegate call to Flat File disassembler
            disassembler.Disassemble(pContext, pInMsg);
        }
        #endregion      
    }
}

    **Figure 22 **- Implement IDisassemblerComponent interface

  12. Implement interface IProbeMessage which will implement Probe method which called, and the component is then executed only if the Probe method returns a value of True and this will do the functionality of schema resolver. We read ReceiveFileName context property from message context to read the first 4 digits of the file name to retrieve schema type from BRE  as show in Figure 23

using Microsoft.BizTalk.Component;
using Microsoft.BizTalk.Component.Interop;
using Microsoft.BizTalk.Message.Interop;
using Microsoft.RuleEngine;
using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;
using System.Xml;
namespace TechNetWiki.SchemaResolver.PipelineComponent
{
    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]
    [Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]
    public class  SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI, IDisassemblerComponent ,IProbeMessage 
    {
        private FFDasmComp  disassembler = new FFDasmComp();
        private static  Dictionary<string, string> cachedSources = new  Dictionary<string, string>();
 
        #region IBaseComponent Members
        public string  Description
        {
            get
            {
                return "Flat file disassembler which resolves schemas using message context";
            }
        }
        public string  Name
        {
            get
            {
                return "Receive Pipeline Schema Resolver";
            }
        }
        public string  Version
        {
            get
            {
                return "1.0";
            }
        }
        #endregion
 
        #region IComponentUI Members
        public IntPtr Icon
        {
            get
            {
 
                return System.IntPtr.Zero;
            }
 
        }
        public IEnumerator Validate(object obj)
        {
            return null;
        }
        #endregion
 
        #region IDisassemblerComponent Members
        public IBaseMessage GetNext(IPipelineContext pContext)
        {
            // Delegate call to Flat File disassembler
            return disassembler.GetNext(pContext);
        }
 
        public void  Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            // Delegate call to Flat File disassembler
            disassembler.Disassemble(pContext, pInMsg);
        }
        #endregion  
     
        #region IProbeMessage Members
        public bool Probe(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            // Check arguments
            if (null ==  pContext)
                throw new  ArgumentNullException("pContext");
 
            if (null ==  pInMsg)
                throw new  ArgumentNullException("pInMsg");
 
            // Check whether input message doesn't have a body part or it is set to null, fail probe in those cases
            if (null ==  pInMsg.BodyPart || null  == pInMsg.BodyPart.GetOriginalDataStream())
                return false;
 
            Stream sourceStream = pInMsg.BodyPart.GetOriginalDataStream();
            string messageType = string.Empty;
            //Read ReceivedFileName from the message context
            string srcFileName = pInMsg.Context.Read("ReceivedFileName", "http://schemas.microsoft.com/BizTalk/2003/file-properties").ToString();
            srcFileName = Path.GetFileName(srcFileName);
            //Substring the first four digits to take source code to use to call BRE API
            string customerCode = srcFileName.Substring(0, 4);
            //create an instance of the XML object
            XmlDocument xmlDoc = new  XmlDocument();
            xmlDoc.LoadXml(string.Format(@"<ns0:Root xmlns:ns0='http://TechNetWiki.SchemaResolver.Schemas.SchemaResolverBRE'>
                          <SrcCode>{0}</SrcCode>
                          <MessageType></MessageType>
                        </ns0:Root>", customerCode));
            //retrieve source code in case in our cache dictionary 
            if (cachedSources.ContainsKey(customerCode))
            {
                messageType = cachedSources[customerCode];
            }
            else
            {
                TypedXmlDocument typedXmlDocument = new  TypedXmlDocument("TechNetWiki.SchemaResolver.Schemas.SchemaResolverBRE", xmlDoc);
                Microsoft.RuleEngine.Policy policy = new  Microsoft.RuleEngine.Policy("SchemaResolverPolicy");
                policy.Execute(typedXmlDocument);
                XmlNode messageTypeNode = typedXmlDocument.Document.SelectSingleNode("//MessageType");
                messageType = messageTypeNode.InnerText;
                policy.Dispose();
                // Fail if message type is unknown
                if (string.IsNullOrEmpty(messageType))
                    return false;
                cachedSources.Add(customerCode, messageType);
            }
 
            // Get document spec from the message type
            IDocumentSpec documentSpec = pContext.GetDocumentSpecByType(messageType);
            // Write document spec type name to the message context so Flat File disassembler could access this property and
            // do message processing for a schema which has document spec type name we've discovered
            pInMsg.Context.Write("DocumentSpecName", "http://schemas.microsoft.com/BizTalk/2003/xmlnorm-properties", documentSpec.DocSpecStrongName);
            // Delegate call to Flat File disassembler
            return disassembler.Probe(pContext, pInMsg);
 
        }
        #endregion
    }
}

**Figure 23 **- Implement IProbeMessage  interface

Create Custom Send Pipeline

  1. Add a new Item class to your class library project and rename it
  2. Do same steps of creating custom receive pipeline class but for sending custom pipeline we need to use CategoryTypes.CATID_AssemblingSerializer attribute and implement IAssemblerComponent interface as show in Figure 24
using Microsoft.BizTalk.Component.Interop;
using System;
using System.Runtime.InteropServices;
using Microsoft.BizTalk.Message.Interop;
using Microsoft.BizTalk.Component;
using System.Collections;
 
namespace TechNetWiki.SchemaResolver.PipelineComponent
{
    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]    
    [ComponentCategory(CategoryTypes.CATID_AssemblingSerializer)]
    [Guid("ACF451CE-E60D-42E4-8AB0-ED4CADD8B23E")]
    public class  SchemaResolverFlatFileAsmComp : IBaseComponent, IComponentUI, IAssemblerComponent
    {
        System.Collections.Queue qOutputMsgs = new  System.Collections.Queue();
        private FFAsmComp assembler = new FFAsmComp();
 
        #region IBaseComponent members       
        public string  Description
        {
            get
            {
                return "Flat file assembler which resolves schemas using message context";
            }
        }
              
        public string  Name
        {
            get
            {
                return "Send Pipeline Schema Resolver";
            }
        }
            
        public string  Version
        {
            get
            {
                return "1.0";
            }
        }       
        #endregion
 
        #region IComponentUI Members
 
        public IntPtr Icon
        {
            get
            {
 
                return System.IntPtr.Zero;
            }
 
        }
        
        public IEnumerator Validate(object obj)
        {
            return null;
        }
        #endregion
 
        #region IAssemblerComponent Members
        public void  AddDocument(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)
        {
            qOutputMsgs.Enqueue(pInMsg);
        }
 
        public IBaseMessage Assemble(IPipelineContext pContext)
        {
            IBaseMessage msg = (IBaseMessage)qOutputMsgs.Dequeue();           
            assembler.AddDocument(pContext, msg);            
            return assembler.Assemble(pContext);
        }
        #endregion
    }
}

**Figure 24 **- Custom Send Pipeline component code

Create Receive and Send Pipeline artifacts

  1. Build you Class library project and copy the dll file and paste it to this location <InstallationFolder>\Pipeline Components

  2. Add new receive pipeline to your BizTalk project

  3. Right Click on Toolbox then select choose items then choose as show in Figure 25

    **Figure 25 **- Selecting Custom Pipeline components

  4. Drag Receive Pipeline Schema Resolver component from Toolbox and drop it to Disassemble part as shown in Figure 26

    **Figure 26 **- Drag Drop Receive Pipeline Schema Resolver component

  5. Add new send pipeline to your BizTalk project

  6. Drag Send Pipeline Schema Resolver component from Toolbox and drop it to assemble part as shown in Figure 27

    **Figure 27 **- Drag Drop Send Pipeline Schema Resolver component

  7. Deploy You Pipeline BizTalk project then open BizTalk Administration and create new Receive location in your BizTalk application, configure the location and select the receive pipeline of schema resolver as shown in Figure 28

    **Figure 28 **- Select receive schema resolver pipeline

  8. Create new send port, select file adapter,configure the location and select send pipeline as shown in Figure 29

    **Figure 29 - **Select send schema resolver pipeline

  9. Select all required Outbound maps to map files from source to destination as shown in Figure 30

    **Figure 30 **- Select Outbound maps 

  10. Filter your send port to subscribe from receive port as show in Figure 31

**Figure 31 **- Filter send port

We can realized now that if we want to add any new source or any new destination, you just need to add schemas, maps and configure BRE or your configuration table.
This means that you reduced the number of artifacts in your solution pipelines and receive and send ports.
In the bad solution if you have 100 sources and 100 destinations, then you will have 100 receive locations, 100 send ports and 100 pipelines.
However, in the good solution you will have 1 receive location, 1 send port and 2 pipelines and your configuration table.

One more thing, that we can do more enhancement to our solution by passing parameter generically to BRE instead of depending on the first 4 characters of file name,
we can use any property of the message context, but for demonstration purposes I just use the file name.

Moreover, we can use canonical schema and mapping (inbound and outbound mapping) dynamically which I will write another article about this subject later on.

Sample Source Code

You can find the sample source code in the following link sample source code

Conclusion

This article demonstrated

  1. The real scenario of receiving and send Flat File from different sources with different formats and sending files to different destinations with different formats
  2. How to implement this scenario using the static BizTalk solution
  3. How to implement the schema resolver mechanism using custom receiving and sending pipeline, which made our solution more dynamic and easy to maintain
  4. How to build custom receive and send pipeline in details
  5. Benefits of using dynamic schema resolver mechanism
  6. Call BRE API from C# code

See Also

Read suggested related topics:

Another important place to find an extensive amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.