BizTalk BAM (Business Activity Monitoring)

Article
1/17/2024

1. Introduction

BAM (Business Activity Monitoring) is extremely useful to many businesses. ECommerce is experiencing huge growth and there are many companies sell in this way.

Middle-ware systems play a vital role for such companies - enabling integration with different Vendors/Partners to complete the supply chain.

The business world expects all transactions to be monitored, irrespective of cost involved for each transaction. As Business Intelligence solutions depend on Scheduled data sources, real time monitoring requires a real time data source. BAM acts as a miniature real-time Business Intelligence model to cater real time monitoring in BizTalk or any WCF based tracking. As BAM ships along with the BizTalk Server package, it is used widely with BizTalk projects.

This article offers advice from the perspective of a BizTalk Developer. The idea is that this article works as a starter kit for BizTalk BAM professionals, easing the climb up the learning curve.

1.1. Prerequisites

This Article assumes you have basic knowledge on the BizTalk tool and configured BAM in BizTalk Configuration tool and MS Excel to proceed with the implementation

1.2. Business Intelligence

As we all know, Business Intelligence solutions are available in variety of technologies but the base of all Business Intelligence tools are three components

ETLà Extract, Transform and Load, Which constitutes of raw data and acts as a data source

Analysisà Data Analysis is the Process layer, in which the data is converted to Multidimensional views

Reporting à Reporting comprises of comprehensive Presentation layer with advanced data representation

1.3. MSBI and BAM

Let me start with a Microsoft way of thinking.

MSBI, Microsoft Business Intelligence uses SSIS, SSAS and SSRS as an ETL, Analyzer and reporting tool respectively to form the complete Business Solution.

SSIS --> SQL Server Integration Service àActs as an ETL tool

SSAS --> SQL Server Analysis Service à Acts as an Analysis Tool

SSRS --> SQL Server Reporting ServiceàActs as a Reporting Tool

BAM in-turn also uses all the three tools to form the comprehensive real-time tracking and reporting. You can see the BAM tracking in a very minute seconds. The BAM Infrastructure is designed for a low latency, high performance which makes you to visualize the data in a fraction of seconds in BAM portal.

1.4. Comparison Table

	Data Source	Data Analyzer	Data Reporter
MSBI	SSIS	SSAS	SSRS
BAM	Movement of data from Message box to Tracking data	Analysis Service(Measures/ Dimensions)	BAM Portal(Grid/chart view)

Business Intelligence

https://www.codeproject.com/KB/biztalk/868640/1._Business_Intelligence.png

Business Activity Monitoring

https://www.codeproject.com/KB/biztalk/868640/2.Business_Activity_Monitoring.png

1.5. SSIS and BAM Architecture

BizTalk relies on the powerful Publish subscriber Architecture, in which Message box database acts as a Heart of BizTalk Engine. From the Perspective of BAM, consider the Message box a real time source of data. But as we need to have control over the data, we need to copy the data to the BAM specific database called the BAM Primary Import.

The movement of data from BizTalk Message box to the BAM Primary Import database is the responsibility of Tracking Data Decode Service (TDDS), which internally uses SSIS packages to perform this action step at run time.

Also the movement of data between BAM specific databases/tables can be carried out by SSIS packages.

1.6. SSAS and BAM Architecture

We have the raw-data in BAM Primary Import Database, we need to make an analysis of available data and from the comprehensive data to users. Eg: Total number of transactions per day, Average of the transactions, etc. This analysis decisions are left to the business, in which way they want the report to be in.

This Analysis of data is done for the Measures and Dimensions in BAM.

Measures are the numeric aggregation of data.

Dimensions are the measures represented in terms of time.

Measures and Dimensions forms the comprehensive Business Report, for which BAM can provide the Grid or chart views.

1.7. SSRS and BAM Architecture

BAM Portal acts a User-Friendly presentation layer in BAM Architecture. Representation of data can be done in Grid or chart view. BAM portal adds a query service for quick search of the Milestones or data that is being tracked. BAM portal represents the data in the form of Activities and Views. Activities.

1.7.1. Activities

Activities represent atomic information units that need to be monitored. In General, Activities are the collection of fields that represent the Business milestones and Business data. Activities are represented in terms of Grid Views in the BAM portal.

1.7.2. Business Milestones

Milestones are the time specific data which is used to represent the state of business process. EG: ReceivedTime and SendTime

1.7.3. Views

View is the collection of activities, which is used to represent data based on the expectation of Different business roles.

As all the data in BAM Primary Import can be utilized as Relational data, BAM capability can be extended to make your own way of reporting through SSRS.

1.7.4. Real time Aggregation in Excel

BAM Data can also be viewed in Excel sheet through Real time Aggregation with the help of Pivot tables.

This Paper makes you to understand BizTalk Architecture at Run time with practical examples.

2. BizTalk Architecture

BizTalk works on the Principle of Publish-Subscribe Architecture, Where a message gets into BizTalk; it undergoes set of stages in different artifacts and published to message box with or without transformation (map). Then the Subscribers/Listeners will subscribe the messages, apply the transformation, passes through the pipeline and the message is sent out of BizTalk

Artifacts in BizTalk:

Adapter, Pipeline, Schemas, Maps etc

https://www.codeproject.com/KB/biztalk/868640/3._BizTalk_Architecture.png

3. Ecommerce Scenario

Let’s take a simple eCommerce solution; ABC Ecommerce Industry has a BizTalk as the Middleware and SAP as backend. The Customer order which passes through BizTalk to the Backend system for further supply chain processes.

3.1. Inbound Flow

https://www.codeproject.com/KB/biztalk/868640/4InboundFlowla.png

The Backend system asynchronously sends three responses: Acknowledgement/ Processed / Failed based on the status of the order

3.2. Outbound Flow

https://www.codeproject.com/KB/biztalk/868640/5OutboundFlowla.png

As this transaction status of Order is very important, this has to visible to the Business users, we will design a BAM monitoring for this scenario. Create a BizTalk Application and design a BAM solution.

3.3. Create a Simple BizTalk Application

Follow the below steps to create a Message only BizTalk application (Without Orchestration)

Create a new BizTalk Project in the Visual studio and name it as BAMTraining

Create an Inbound schema as below and name it as InboundOrder.xsd(all elements are string datatype)

https://www.codeproject.com/KB/biztalk/868640/6Create_schema.png

This will be the Request flow message for our scenario.

Create another schema as below and name it as outboundOrder.xsd (all elements are string datatype)

https://www.codeproject.com/KB/biztalk/868640/7CreateOutboundScehma.png

Deploy this project to BizTalk Administration Console

Create the Receive ports Receive Ports and Receive Location

https://www.codeproject.com/KB/biztalk/868640/8ReceivePorts.png

https://www.codeproject.com/KB/biztalk/868640/42TableRL.png

Create the Send ports and subscribe to the corresponding Receive ports

https://www.codeproject.com/KB/biztalk/868640/9SendPorts.png

https://www.codeproject.com/KB/biztalk/868640/43TableSP.png

Now try with the sample Inbound file and outbound file after restarting the host instances

Now let’s design the tracking solution for this message only project, which is been deployed in your machine.

3.4. Tracking Scenario

Before designing we need to make sure two questions are being answered

What to track àKey Decisive Elements/ Business Data / Business Milestones

Where to track à Receive side or send side

Let’s consider the Receive port as the Tracepoint1 and send port as the Tracepoint2.

https://www.codeproject.com/KB/biztalk/868640/10TrackingScenariola.png

We will be tracking below elements on the send and receive side

https://www.codeproject.com/KB/biztalk/868640/44TableTrackingProp.png

Note: Send port time indicates that the message has been successfully delivered to the end system. If it is not getting populated, it indicates that there is a transmission failure or any internal BizTalk Failures.

4. BAM Architecture

As we all know Messagebox is the heart of BizTalk, but BAMPrimaryImport database is the heart of Tracking. Message box database will transport the data to BAMPrimary Import database, isolating the Tracking from BizTalk Engine.

When a Message been published to Messagebox, the messages will get sustained in the Messagebox till the Life cycle of the Message (Until the message gets subscribed by all the subscribers).

After the Message life cycle the messages gets deleted from Message box. Inbetween these two tasks, if the BAM is been implemented to the BizTalk solution, the heartbeat signals will initiate the movement of data from the Messagebox to the BAMPrimaryImport Database.

This movement is done by the TDDS service (Tracking Data Decode Service). TDDS is also responsible for moving the data to the DTATracking DB. The Published message and subscribed message along with the milestones can be tracked with the help of BAM.

BAM Event Processors help in triggering the Business Events and mapping them to Activity models in BAM Primary Import Databases. Once you deploy the BAM solution, at run time the messages move from messagebox to other databases. Keep in mind that tracking will not happen in the message box, to reduce the overhead on it.

Note: message payload will be in compressed/encoded format, TDDS service (Tracking Data Decode Service) is responsible for decoding and moving the data to BAM Primary Import DB and DTA DB.

https://www.codeproject.com/KB/biztalk/868640/11BAMArchitechture.png

DTA Tracking can be implemented by enabling tracking at the ports through BizTalk Administration Console, which will help in viewing the message by Tracked Message Events in the BizTalk Admin console.

Once the specified time of 1 month (Online window) is reached, the message gets partitioned in the BAM Primary Import database. The Partition of BAM tables is done by the help of SSIS packages, which gets created during the deployment of BAM Definition File. Also, more than one month data will be archived in the BAM Archive DB. SSIS Packages, which gets created during the deployments, are responsible for these movements.

4.1. BAM Databases

List of Databases which gets created when BAM is installed and configured. Below Databases helps tracking to achieve its various Functionalities as below

https://www.codeproject.com/KB/biztalk/868640/12BAMDatabase.png

Databases	Purpose
BAM Primary Import	Data source/ Heart of BizTalk Tracking
BAM Archive	Archived data based on the Time window
BAM Star Schema	Data Analysis like Measures, Dimensions and Chart views are analyzed
BAM Analysis
BAM Alerts Application	BAM Alerts Management Database
BAM Alerts NS Main	Notification service for triggering mail through SMTP

Also BAM holds the concepts of Continuations and Relationships which used to enhance the relational activity model.

4.2. Continuations

Business world expects all possible relations in one single activity. The Split of details in different activities, will make it a cumbersome process to make a key Decisions. Hence BAM displays the continuity of operations with the help of Continuations. In General, Continuations are the unique concept in BAM which is desired to correlate the inbound and outbound messages.

4.3. Relationships

Activities are the atomic information unit of Monitoring. Different Activities are to be correlates to provide a comprehensive data. This Coalition between different activities is achieved through Relationships in BAM.

5. Implementing BAM Solution

Implementation of BAM tracking can be done by following steps

Create a BAM Definition File using MS Excel (For creating relevant tables, Views, SP’s etc.)

Create an TPP file using Tracking Profile Editor ( Responsible for binding the data to the ports)

5.1. Create a BAM Definition File using MS Excel

5.1.1. Create activities and Views in Excel sheet

In BAM everything is represented in the form of activities and view. The creation of activities and Views in Excel sheet is equivalent to designing the container that represents data to the end users. We need to know, what are the Key columns (KPI) that has to be tracked which will enhance the users understandability

Activities are the data tables which collect information from the source (Orchestration, Pipeline, WCF Interceptor)

View is a collection of activities, where in it can represents collective data information to the business

5.1.2. Create Activities

Goto MSExcel>> Add-ins>>BAM>>BAM Activity

https://www.codeproject.com/KB/biztalk/868640/13BAMCreateActivities.png

5.1.3. Creating Milestones/Business Data for Inbound Activity

Name the activity as InboundOrder and click on new item to create columns. All the columns should be assigned to a datatype as shown below

https://www.codeproject.com/KB/biztalk/868640/14BAMCreateMilestone.png

Note: Business Milestone is the datetimestamp which will be used to display the message arrived in time and Message dispatch time. All the other column datatype is text

5.1.3.1 Creating Milestones/Business Data for Outbound Activity

Follow the same procedure for creating a Outbound Order Activity with the below details

https://www.codeproject.com/KB/biztalk/868640/49BAMActivitytOutbound.png

After creating two activities, you will get a screen as shown bleow

https://www.codeproject.com/KB/biztalk/868640/50BothActivity.png

5.1.3.2 Create Views

As the activities are created, Views will helps in representing multiple activities in the BAM portal

Click on Ok, you will be navigated to the initial View Page

https://www.codeproject.com/KB/biztalk/868640/48BAMViewWindow.png

Click on the Radio button Create New View and click next

https://www.codeproject.com/KB/biztalk/868640/51CreatenewView.png

select the View name and select the activities in the next window, as shown below.

https://www.codeproject.com/KB/biztalk/868640/52SelectAllTraining.png

We can create n number of views for a single activity, to cater different users. Now select all the data items whichever you have created in both the activities

https://www.codeproject.com/KB/biztalk/868640/53SelectAllItems.png

and click couple of next, to get into the final page, as we are not covering measures and dimensions.

https://www.codeproject.com/KB/biztalk/868640/53FinalViewScreen.png

This Page will create the Excel sheet with full details.

The BAM defintion can be in Excel or in XML, as XML is cross server compatible, always prefer using the XML exported defintion file. Now export the BAM Definition file to the xml format and save it as BAMTraining.xml

https://www.codeproject.com/KB/biztalk/868640/15BAMExportxml.png

5.1.4. Deploy the BAM Definition file

The BAM Definition file (BAMTraining.xml) should be deployed using the BAM Command line utility.

Deploying of BAM definition file will create corresponding database tables, views, SSIS packages, etc which will organize the data in a relational view.

Follow the below steps for deploying BAM Definition file

Open Command Prompt as administrator

Navigate to the %BTSInstallPath%/Tracking folder

Use the Command

bm.exe deploy-all –DefinitionFile:”<<FileLocation>>\BAMTraining.xml”

https://www.codeproject.com/KB/biztalk/868640/16BAMDeploy.png

BAM Definition deployed successfully :)

5.1.5. At the Backend

When you deploy the BAM Definition file, details of your tracking solution will get into the BAM Metadata tables in BAM Primary Import and also following artifacts gets created in the BAM Databases

https://www.codeproject.com/KB/biztalk/868640/17ActivityTables.png

https://www.codeproject.com/KB/biztalk/868640/18ActivityViews.png

5.1.5.3. SSIS Packages for Partitioning/ Archiving/ Data Movements

SSIS Packages will get created in the Integration services, which will help in Partitioning and Archiving the BAM data for better performance.

SSIS Package Name: BAM_DM_<Activity Name>

5.2. Configure and deploy tracking profile editor file.

5.2.1. Profile Editor

Tracking Profile editor defines the way in which the data should be tracked. Tracking Profile editor tool will be Responsible for binding the data to the ports.

https://www.codeproject.com/KB/biztalk/868640/19TPEbond.png

5.2.2. Configuring Tracking Profile Editor for Inbound Order Request

As we had already deployed the BizTalk solution and BAM Definition file, Now its time to link the both using the Tracking Profile Editor

Goto Start>> Tracking Profile Editor

https://www.codeproject.com/KB/biztalk/868640/20TPEInterface.png

Left Pane à Activity Definition file

Right PaneàBizTalk Deployed Message/Context properties

Click on the Left pane “Click here to Import a BAM Activity Definition” and select the InboundOrder from the Popup

https://www.codeproject.com/KB/biztalk/868640/21TPEActivityAssigning.png

Activity Columns has got populated now

https://www.codeproject.com/KB/biztalk/868640/22TPESourceAssigning.png

Select the Event Source:

Event Source is the place from which data is populated in the BAM

https://www.codeproject.com/KB/biztalk/868640/23TPEPortAssiging.png

Event source	Description
Orchestration Schedule	Orchestration Properties
Messaging Payload	Deployed Schema Elements
Context Property	BizTalk Context Properties
Messaging Property	Message/Port Properties

Follow the below table and drag and drop the properties from right pane(event source) to the Left Pane(Activity Columns)

Activity Column Name	Event Source	Property/element Name	Port Name
OrderID	MessagingPayloadàBAMTrainingà InboundOrderàOrderID	OrderID	InboundReceivePort
ProductID	MessagingPayloadàBAMTrainingà InboundOrderàProductID	ProductID	InboundReceivePort
XProductName	MessagingPayloadàBAMTrainingà InboundOrderàProductName	ProductName	InboundReceivePort
ShipmentAddress	MessagingPayloadàBAMTrainingà InboundOrderàShipmentAddress	ShipmentAddress	InboundReceivePort
RcvPortName	ContextProperty à BTS.ReceivePortName	BTS.ReceivePortName	InboundReceivePort
RcvdTime	Messaging PropertyàPortStartTime	PortStartTime	InboundReceivePort
SendPortName	ContextProperty àBTS.SPName	BTS.SPName	InboundSendPort
SendPortTime	Messaging PropertyàPortEndTime	PortEndTime	InboundSendPort

To Insert Continuation>>Right click on Inbound Order>>New Continuation and Assign Interchange ID from Receive port

Also >> Right Click on Inbound Order>>New Continuation ID

Rename ContinuationID to Continuation (Both should be of same name)and assign Interchange ID from Send Port

At last your Tracking Profile should resemble like the above image, save it as TestInboundTPE.tpp file

Goto File>>click on Apply Tracking Profile

This makes you to successfully deploy the tracking profile file into the BAM solution

5.2.3. Configure Tacking Profile Editor file for Outbound Order Activity

Open a New Tracking Profile Editor and open the OutboundOrder Activity on Left Pane and assign the Values as shown below

https://www.codeproject.com/KB/biztalk/868640/24TPEPortAssigningforOutbound.png

Activity Column Name	Event Source	Property/element Name	Port Name
OrderID	MessagingPayloadàBAMTrainingà InboundOrderàOrderID	OrderID	InboundReceivePort
RcvPortName	ContextProperty à BTS.ReceivePortName	BTS.ReceivePortName	InboundReceivePort
RcvdTime	Messaging PropertyàPortStartTime	PortStartTime	InboundReceivePort
SendPortName	ContextProperty àBTS.SPName	BTS.SPName	InboundSendPort
SendPortTime	Messaging PropertyàPortEndTime	PortEndTime	InboundSendPort

In the Above image, InboundOrder indicates the relationship to the OutboundOrder. There should be one commonid which is Order id in our case.

Goto>>File>>Apply the configuration

The Binding between ports and BAM is applied, which will help BAM events get triggered when the message arrives in the port and populate the data in BAM tables.

6. How Continuation Works

Consider the Inbound Activity

We have configured the Tracking profile editor to track both the tracepoints into a single activity.

https://www.codeproject.com/KB/biztalk/868640/25TPEContinuation.png

Let’s take the above 1:1 scenario

Tracepoint1- Inbound message
Tracepoint2 – Outbound message

The Continuation between Tracepoint1 and Tracepoint2 is established out through one common id called InterchangeID (Which is same on the receive and send port).Whenever inbound message arrives, the Active table gets populated.

When outbound message delivered successfully, data will move from active table to completed table. All Instances (SQL view), which combines both Active and Completed table, will be responsible for the data to be displayed in BAM

6.1. Movement of data in the Activity Tables

https://www.codeproject.com/KB/biztalk/868640/26ConitnuationMovement.png

6.2. Testing the Continuation scenario

As we have configured the BizTalk Application, Stop the InboundSendPort and post the below message to the Receive location

https://www.codeproject.com/KB/biztalk/868640/27TPETestingContinuation.png

Message has been picked up and failed to send to send port location and suspended in BizTalk. Let’s see the Back end tables result

6.2.1. Active Table Values

The below snapshot shows the Business data and milestones tracked at receive port

https://www.codeproject.com/KB/biztalk/868640/28TPEActivityTablela.png

6.2.2. BAM View

Representation in BAM view

https://www.codeproject.com/KB/biztalk/868640/29BAMViewBackend.png

Also note that the Send side data is not populated because the message has not been moved out of the send port. Let’s start the send port now and see the result.

6.2.3. Completed Table Values

After starting the send port Business Data/ Milestones at tracepoint2 has been recorded in the BAM Activity Completed table, record gets moved from Active to completed table when the message has been successfully delivered

https://www.codeproject.com/KB/biztalk/868640/30CompletedTableShotla.png

6.2.4. BAM View

https://www.codeproject.com/KB/biztalk/868640/31CompletedTableView.png

You could also see the ExecutionTime when RcvdTime and SendportTime are also filled.

6.2.5. Full BAM Record View:

This is the full view of all the Data, Milestones and Activity ID

https://www.codeproject.com/KB/biztalk/868640/32FullBAMRecordView.png

7. How Relationships Works

As we get multiple responses back from SAP system for a single Order Request, We need to establish a relationship between the Response messages to request message.

https://www.codeproject.com/KB/biztalk/868640/33HowRelationshipWorksla.png

The Relationship scenario is applied on top of the Continuation scenario

https://www.codeproject.com/KB/biztalk/868640/34Relationshipexplained.png

This kind of 1: N scenario (1 request and Multiple Response), is addressed using Relationships in BAM.

https://www.codeproject.com/KB/biztalk/868640/35Relationship1toN.png

In the above figure,For One Order Request, there may be multiple Responses may arrive from the end system, the relationship between the one request and multiple responses is established using one CommonID which is common between the Inbound and Outbound Order. In our case it will be the OrderID.

Note: Relationship can be established only for the activities which belong to the same views.

7.1. Testing the Relationship Scenario

In our case Relationship will be between Inbound Order versus multiple Outbound Order status.

Lets post the below acknowledgement message on the Outbound Receive location.

https://www.codeproject.com/KB/biztalk/868640/36TestingScenario.png

The OrderID is being the same for both Inbound and Outbound Orders.

7.1.1. BAM View- Acknowledged

https://www.codeproject.com/KB/biztalk/868640/37AcknowledgementBAMView.png

7.1.2. Full BAM View of the Record

https://www.codeproject.com/KB/biztalk/868640/38BAMViewComplete.png

You could see the link of InboundOrder Activity on the Related Activities section

7.1.3. BAM View- Processed

Posting another message as Processed Status

https://www.codeproject.com/KB/biztalk/868640/39BAMViewProcessed.png

7.1.4. BAM View: For Multiple Response messages

https://www.codeproject.com/KB/biztalk/868640/40BAMViewforMultipleResponses.png

On the Inbound activity you could find two related activity of Oubound

Acknowledged

Processed

Relationship of- One Inbound Order with N Outbound Order status is achieved.

By Clicking on the Outbound Order Activity, we can navigate to the corresponding Inbound Order activities easily. This makes the business/Support team to easily monitor the successfully processed orders versus unsuccessful ones.

8. BAM Alerts

BAM Alerts are best in class alerting feature, which ships along with BizTalk. Alerts will be triggered based on the conditions which you can set during design time on par the business expectation. Usually the SLA violations are alerted using BAM alerts. If BizTalk system has an SLA to send the message within 20 minutes of received time, the milestones (Received Time and Send Time) will let us know the violation of SLA and the Alert service will run every 30 seconds to evaluate the condition and trigger an alert to the specified Subscribers/ groups.

9. Conclusion

BAM, Miniature Business Intelligence is a ready-made Business solution for BizTalk Developers for Tracking and reporting. As this IT era, mainly focuses on quick results with less number of resources, BAM will be a Best in fit with a proven track records and many future enhancements will be in BAM to cater the growing demand in near future. I hope this Paper has leveraged on providing you on the basics of BAM.

Happy Integrating :)

Share via