Azure BizTalk Services: Pulling Messages from a Service Bus Queue

Article
1/17/2024

Introduction

Azure BizTalk Services (formerly called Windows Azure BizTalk Services or WABS) provides capabilities for EAI and B2B in the cloud. This relative new service was made available for customers in November 2013. Azure BizTalk Services is a cloud integration platform or integration platform as a service or IPaaS. Characteristic of IPaaS is that you build your integration on premise, deploy in the cloud, where it is hosted in a service (set of dedicated hardware). The service is offered through a subscription model for operating the service in the cloud (for BizTalk Services this means Azure hence the name Azure BizTalk Services).

Currently there are quite some players offering that service like Dell, Attunity, MuleSoft, TIBCO, and so on. Microsoft is currently pushing the service hard to catch up with some of these players and try to quicken/hasten the maturity of the product. Therefore, Microsoft has committed to have a release cadence for new features every three months. The recent release was in February 2014, which added new features for Azure BizTalk Services. These new features are:

EDIFACT Protocol Support and X12 Schema Updates
Pulling Messages from Service Bus Queues and Topics
Service Bus Shared Access Signatures (SAS) support with Service Bus Queues and Topics
BizTalk Adapter Services No Longer Needs SQL On Premises
Backup and Restore Support
Operation Log Support

This article will dive into the new feature of pulling messages from Service Bus Queue. When a developer designs and build a bridge he/she can now choose two new sources i.e. Service Bus Queues and Topics. A Bridge with Azure BizTalk Services is a workflow that processes a message pulled from a source and delivered to a destination (see picture below).

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics1_zps39c0697a.png

Figure 1 - BizTalk Services Bridge concepts.

Benefits

Benefits of pulling messages from Service Bus queues and topics are:

Durable messaging
Leveraging the support for multiple protocols

Durable messaging

The ability to pull messages from Service Bus queues and topics increases the durability of a bridge solution. You can leverage the pub-sub mechanism of the topics and subscriptions. It is even possible to have multiple bridges pulling messages from different subscriptions and have them send to multiple destinations. A single bridge will not enable you to do that.

Leveraging multiple protocols

Another benefit is that you as a developer will have more input channels for the bridge besides FTP, and HTTP (REST). Indirectly through the Service Bus Queues you could have support for AMQP.

Scenario

To explore the new capability of pulling message from the Service Bus we will explore the following scenario. A payroll company wants to offer a service in Azure BizTalk Services for organizations to send their payroll data for pay rolling. Organizations that like to outsource their pay rolling can connect to this service in Azure. Basically the organizations can send their payroll data to the service in any kind of format and structure they want. The payroll company will provide a bridge to provide connectivity to the source the organization pushes its data to. This can be FTP(S), HTTP or the Service Bus Queue or Topic (subscription). The data will be picked up from the source and processed to one unified format to be routed to a LOB system or database for a later pay roll run that the payroll company will do. Below you will find a diagram detailing the scenario.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics22_zps1aa58d83.png

Figure 2 - Diagram of the payroll scenario.

Building the solution

The solution will be a bridge for an organization that requires to push out its payroll data to the payroll company. In this case the data is pushed to a Service Bus Queue and the Bridge will pull the data from the queue, transform it to a format that enables a table operation in SQL Server on premise. To build this solution the following parts need to build, configured and/or created:

a LOB Target that needs to be configured for access to SQL Server through a LOB Relay endpoint,
generation of a schema for the table operation (insert operation),
a mapping,
a bridge design,
connection of the source to the bridge and to the destination,
configuration of the source, the bridge and the destination.

When solution is build it needs to be deployed to the BizTalk Service in the cloud.

Prerequisites

You will need a Microsoft Azure Subscription. You have different options to setup one by:

Signing up for a Trail for a month
Acquire through purchase plan
Acquire through MSDN Subscription

In case you have signed up or have a subscription you will need to provision a BizTalk Service, see TechNet Wiki Article Provisioning BizTalk Services Using the Microsoft Azure Management Portal. Finally you will need to install the SDK. In case you already provisioned a BizTalk Service and installed a previous SDK, you can upgrade it easily, see TechNet Wiki article Azure BizTalk Services: Upgrade to February 2014 SDK.

Create a BizTalk Service Project

To build an Azure BizTalk Services solution you need to create a BizTalk Service project:

Open Visual Studio and on the File menu, point to New, and then click Project. Use the details in the following table to create the project and then click Ok:

Installed Templates	Click BizTalk Services, then click BizTalk Service.
Name	Specify a name.
Location	Set this to a location on your computer.
Create directory for solution	Select this if you want this solution to have a separate folder in Windows Explorer.

Table 1 - BizTalk Service Project specifications.

LOB Target/Relay

A part of the Azure BizTalk Services is on-premise infrastructure. By downloading the Microsoft Azure BizTalk Services SDK you can set up that infrastructure. By infrastructure in this context is addition of Azure BizTalk Services templates in Visual Studio 2012 and installation/configuration of the BizTalk Adapter Service. This service is basically a IIS wrapper around the adapter pack that provides access to on-premise LOB systems. The BizTalk Adapter Services can be viewed as a gateway to premise LOB systems that can be securely accessed indirectly through BizTalk Services. The concept of BizTalk Adapter Service is to ability to create kind of endpoints to operation on your LOB systems. You create a LOB Target, which is your on-premise Line-of-Business (LOB) system and the operations (like SELECT or INSERT) exposed to your client applications (a Bridge). During the configuration of a LOB Target you create a LOB Relay, which is a URL that provides a connection to the cloud using Service Bus Relays.

For installation of the BizTalk Adapter Service see TechNet Wiki article BizTalk Adapter Service Installation.

Creating a LOB Target/Relay

A LOB Target is created through Visual Studio. You have to perform the following steps to create the LOB Target:

In the BizTalk Service project, open Server Explorer, right-click BizTalk Adapter Service and expand.
You will need to authenticate to further expand to LOB Types.
Provide credentials to access the BizTalk Adapter Service.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics2_zpsfa60da93.png Figure 3 - Credentials for accessing the LOB Types.
Click Ok. You can now expand LOB Types, right-click SQL, and select Add SQL Target.
You will see a Wizard pop-up and the first screen will be a welcome screen.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics3_zps4feaf729.png Figure 4 - LOB Target configuration Welcome screen/tab.
Click Next and configuration process of LOB target will start.
In the Connection parameters tab you specify the connection details of SQL Server and the credentials.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics4_zps41d2071f.png Figure 5 - Specifying the settings for connecting to SQL Server database.
Click Next.
In the Operations tab you can select object, operations and add them to the selected operations.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics5_zps1a664564.png Figure 6 - Operations tab for specification of operations on the LOB Target (SQL Server Object).
Click Next.
In the Runtime Security tab you specify the way you like the authentication to LOB target i.e. SQL Server.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics6_zps22b04f5a.png Figure 7 - Specify the run time security settings for access to LOB Target (SQL Server).
Click Next.
In the Deployment tab you specify LOB Relay. You can select an existing relay or specify a new one.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics7_zps269b36d1.png Figure 8 - LOB Relay settings.

To create a new LOB Relay, enter the following details:

Service Bus Namespace	Specify the Service Bus namespace on which the LOB relay endpoint is created.
Service Bus Issuer Name	Specify the issuer name for the Service Bus namespace
Service Bus Issuer Secret	Specify the issuer secret for the Service Bus namespace
LOB Relay Path	Enter a name for the relay.
LOB Target Sub-path	Enter a sub-path to make this target unique.
Target runtime URL	This read-only property displays the URL where the relay is deployed on Service Bus. This is the path where you could send a message to be inserted into the on-premises SQL Server. In the payroll scenario, this is where the bridge routes the message.

Table 2 - Specifications for LOB Relay target.

Click Next.
The summary tab will appear and you review your LOB Target/Relay setup.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics8_zpse44c875c.png Figure 9 - Summary tab LOB Target.
By click Create the LOB target/relay will be created. What basically will happen is that LOB target will be created in the BizTalk Adapter Service and is created as an application in IIS.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics9_zpse8680b37.png Figure 10 - LOB Target part as an application in IIS.

Generate schema for the insert operation

When a LOB Target is created you can generate the schema(s) for specified operation(s). The following steps describe how to generate the schema for the insert operation:

In the BizTalk Service project, in the Server Explorer, right click the LOB target you created, and then click Add Schemas To Payrollservice.
A dialog will appear, where you specify the credentials, folder name, ...

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics17_zps078b1058.png Figure 11 - Schema generation specifications and credentials.
Click Ok.
The schema(s) will be added to the BizTalk Service project.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics10_zpsbf2daac2.png Figure 12 - LOB Schemas in BizTalk Service Project.

Generate schema for request

In BizTalk Service project you can create the schema for the incoming request message containing the payroll data. The data is delivered per employee. The schema can be created as visualized by the screenshot below.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics11_zpscea2d736.png

Figure 13 - Payroll data schema.

Create Mapping from source schema to destination schema (XML Transform)

The following steps show how to transform the incoming request message (source) to a request message for LOB Target (destination):

In Visual Studio, right click the your project, point to Add, and then click New Item.
In the Add New Item dialog box, select Map, specify the map name as <name>.trfm, and then click OK.
In the Solution Explorer, double click the <name>.trfm file to open the Transform. On the Transform surface, select the source schema to Request.xsd and the destination schema to xxxxx_PAYROLLSERVICE_payrolldata_TableOperation.dbo.Employee.xsd.
Drag lines between the various elements in the source and destination schema like below:

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics12_zps701a5f95.png
Figure 14 - Mapping between payroll schema (incoming message) and SQL table operation schema (Insert).

Configure a one-way Bridge

The following steps describe how to configure the one-way bridge with a source and destination. You select the MessageFlowItinerary.bcs and:

Drag and drop the LOB Target Entity from the BizTalk Adapter Service onto the design surface (destination).
Drag and drop XML one-way bridge onto the design surface.
Finally drag and drop Service Bus Queue source to the design surface (source).
Double click the XML One-Way Bridge on the itinerary designer.
On the XML One-Way Bridge design surface, within the Message Types box, click the add icon [+] to open the Message Type Picker dialog box.
In the Message Type Picker dialog box, from the Available message types box, select the Request (PayRolldata) message type, click the right arrow icon to associate the request schema with the XML One-Way Bridge, and then click OK.
Configure the bridge to use the Transform created earlier. Within the Transform stage, select the Xml Transform activity, and then from the Properties window click the ellipsis button (…) against the Maps property to open the Map Selection dialog box.
Save the bridge configuration and go back to the Bridge Configuration designer surface

Configure the Service Bus Queue source

The source of the bridge needs to be configured (connectivity) to enable the bridge to receive or pull a message from. For the service bus queue you need to specify the connection string in the following format:

Endpoint=sb://<your service bus name>.servicebus.windows.net/;SharedSecretIssuer=<issuer name>;SharedSecretValue=<issuer secret>

Beside the connection string you specify the name (entity) of the source and name of the queue.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics13_zps32271af1.png

Figure 15 - Specify the settings for Service Bus Queue source.

Configure the LOB Target Destination

Once you have dragged a LOB Target on the design surface you will see a <your lob target>.config appear. You can double click this file in the solution explorer. This file resembles a kind of binding file where you need to specify the credentials of your LOB Target Relay. These are service bus credentials. After specifying those in the file you can save it.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics16_zpsf6f288a8.png

Figure 16 - Specify the issuer name and secret for LOB Relay in config file.

Connect the components on the bridge design surface

After configuration of the source, bridge and destination it is time to connect the source to the bridge and the bridge to the destination. The following steps describe how to connect the components and connection configuration from the XML One-way bridge to the LOB Target:

From the Toolbox, select Connection, and drag-drop the mouse pointer from the Payroll Queue component (source) to the bridge component to connect the two.
Select a Connection again from the Toolbox, and drag-drop the mouse pointer from the bridge component to the LOB Target/Relay component to connect the two.
Set the filter condition on the connection between the bridge and the LOB Relay entity.
Click the connection between XML One-Way Bridge and the LOB Relay entity.
In the Properties window, click the ellipsis (…) button for Filter Condition.
In the Route Filter Configuration dialog box, set the filter condition to Match All. This ensures that all the messages that are processed through the bridge are routed to the LOB entity.
Click Ok.
Set the Route action so that the outgoing message to the LOB application has the appropriate SOAP action header.
Open Server Explorer and navigate to the SQL LOB Relay created earlier. Right click the relay, click Properties, and for the Operations property, copy the value of the first operation (insert). This value denotes the value of the SOAP action header that must be set on the message that is routed to the LOB Relay.
On the Bridge Configuration surface, click the connection between XML One-Way Bridge and the LOB Relay entity.
In the Properties window, click the ellipsis (…) button for Route Action. In the Route Actions dialog box, click Add to open the Add Route Action dialog box. In the Add Route Action dialog box, do the following:
Under Property (Read From) section, select Expression, and then paste the value that you copied earlier.
Under Destination (Write To) section, set the Type to SOAP and the Identifier to Action. The dialog box resembles the following:

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics14_zps9d172c9c.png Figure 17 - Specifying the Route Action to the destination.
Click OK in the Add Route Action dialog box to add the route action. Click OK in the Route Actions dialog box and then click Save to save changes to an BizTalk Service project. The end-to-end message flow resembles the following:

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics15_zps73a49db6.png
Figure 18 - Message flow for payroll data.
Save change to the project.

Build, deploy and Test the solution

You have finished creating the application. In this steps, you will build and deploy the application under your BizTalk Services subscription. Subsequently you will use the Service Bus Explorer to test your solution.

To build and deploy the solution

To build and deploy your solution you can perform the following steps:

In Visual Studio, right click your bridge solution, and then click Build Solution.
Once the build succeeds, right click your solution, and then click Deploy Solution.
In the deployment window, the Deployment Endpoint is a read-only property and the value is derived from the BizTalk Service URL/Namespace set in the message flow surface. However, you must provide the ACS Namespace for BizTalk Services, Issuer Name, and Shared Secret.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics18_zpse94f0be1.png Figure 19 - Specify the settings for deployment to the BizTalk Service.
Click Deploy. The Visual Studio Output pane displays the deployment progress and result. The URL where the bridge is deployed is also displayed in the Output pane. For this scenario, the bridge is deployed at http://sltn2014.biztalk.windows.net.

Test the solution

To test the solution you can use the Azure Service Bus Explorer. This tool can be used to sent a message to a service bus queue. The following steps will describe the way to test the solution:

Open the Service Bus Explorer.
Connect to the appropriate namespace within the service bus where the queue resides that the bridge will pull the messages from.
Right click the queue and select send messages.
Paste/Load a message payload in the Message Text pane.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics19_zps8d8454ce.png

Figure 20 - Send message to the Service Bus Queue payroll using Microsoft Azure Service Bus Explorer.
Click Start.
Message will be sent to the queue.
You can see the log displaying the result.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics20_zps90ffeb87.png Figure 21 - Log of the sent message to the Service Bus Queue payroll.
In SQL Server you can view the Employee table to verify if the data is present in the table.

http://i208.photobucket.com/albums/bb152/Steef-Jan/WABS%20Pulling%20Messages%20from%20SB%20Queues%20and%20Topics/WABSSBQueuesandTopics21_zps5d02b85b.png Figure 22 - Payroll data in SQL Server table Employee.

Wrap up

The addition of queues and topics as sources to Azure BizTalk Services has extended the flexibility and ability to create durable cloud solutions. However, BizTalk Services will need more adapters (sources/destinations) to be able to compete with some of the other IPaaS vendors like MuleSoft, who offers a great deal of adapters. Hence more versatile integration capabilities.

This article demonstrated the use of pulling messages from a Service Bus Queue. This ability can add durability to your cloud integration solution using Azure BizTalk Services. The walk through of the payroll scenario demonstrated the ease of creating a bridge solution. Most of the configuration is done through Visual Studio a main component currently to build, deploy and manage a bridge solution.

The huge dependency currently of Visual Studio for building BizTalk Services is undesirable. Changes to the configuration of sources and destinations for a bridge has to be done in Visual Studio accordingly and then redeployed to the BizTalk Service hosting the bridge. Therefore, managing some parts of a bridge solution like the configuration of sources and destinations is preferable better off in a different tool like a management console/web application or within Microsoft Azure itself. Since this is still the first release after the go-live of BizTalk Services last year in November you can expect improvement of the service in many areas including the manageability.

Share via