BizTalk Archiving, The BizTalk Way

Article
1/17/2024

The Pain

A frequent topic, and cause of much pain and frustration, on the Forums is Archiving. We have a very simple philosophy on most archiving scenarios, don’t.

Yes, that’s right, just don’t do it. We’ve found most archiving implementations suffer from one or more fundamental problems:

Archiving was not requested by the business or there is no business case.
The archiving process is poorly implemented.
The archived data is not very helpful.

If you recognize any of these situations, you’re probably doing archiving wrong.

The Relief

Just imagine if we have a facility that was easy to use, reliable, performed well, free and came with a UI for searching and retrieving. Well, this has existed since BizTalk Server 2004 and it’s call Tracking.

The sample companion Solution can be downloaded from the MSDN Code Gallery at:

While Tracking can archive the message in any format as it transits the system, by far, the most common scenario is storing the original incoming stream submitted by the Adapter. This could be a file, HTTP request, Message Queue body or any other stream of data.

Additionally, with just a few checkboxes, the messages are fully searchable and can be saved locally. Tracking requires no coding or deployment and ships with a dedicated UI easily used by non-technical personnel.

To demonstrate the most practical use of Tracking, we have a sample app that receives a flat file, parses it to XML and outputs the result to another folder. In an archiving scenario, we expect the user to request incoming message be located by Filename.

Setting Up Archiving/Tracking

The first step is to enable the FILE.ReceivedFileName Context Property for Tracking on the Receive Port. FILE.ReceivedFileName is a system property so in BizTalk Administration we navigate to .\BizTalk Group\Applications\BizTalk.System\Schemas\FILE.bts_file_properties. Right click and choose Properties then select Tracking.

FILE.bts_file_properties is the Property Schema that defines the context properties owned by the File Adapter.

Check FILE.ReceivedFileName and click OK.

We have just enabled Tracking on the ReceivedFileName Property. Note, FTP also defines a ReceivedFileName property which can be enabled in the FTP.bts_ftp_properties Property Schema.

Next, we have to enable Tracking on the Receive Port. Open the BizTalk Archiving Application and select Receive Ports. Right-click ReceiveTrackedMessage and choose Tracking.

In Tracking Options:

Check Track Message Bodies/Request message before port processing. This is the original data stream submitted by the Adapter.
Check Track Message Properties/Request message before port processing.
Check Track Message Properties/Request message after port processing. This is necessary for a reason described later.

We have just enabled Tracking on the Receive Port. It is the three-way combination of property definition, message property and message body tracking that enable search and retrieval of messages processed by the system.

Running the Sample

Make sure the Receive Location and Send Port are Enabled and Started. Copy the sample file TrackTest_01112015.txt to the IN folder. Wait for it to process and the Xml version to appear in the OUT folder.

Setup BizTalk Tracking Database Console

Open BizTalk Administration. Right-click BizTalk Server Administration, choose Connect to Existing Tracking Database.

Enter or select the SQL Server hosting the DTA Database in the SQL Server name field. Choose the DTA database for the Database name.

Note, the BizTalk Tracking Database console can be used to query an instance of a Tracking database, this includes a live Tracking database connected to a BizTalk Group or an instance maintained by the outputs from the Purge and Archive Agent Job.

The BizTalk Tracking Database provides the same query functionality of BizTalk Administration without any options to manage BizTalk Server itself. This is ideal for users who should not have access to the BizTalk Server administrative functions but who need to view application activities.

Retrieving the Archived Message

In the BizTalk Tracking Database console, select New Query and set the following parameters:

Search For | Equals | Tracked Message Events
Schema Name | Equals | http://MyAutoArchivedApp#IncomingMessage
Maximum Matches | Equals | 50
ReceivedFileName | Like | 01112015

Click Run Query.

Here we see the search results on the file name we were looking for. It’s very important to note the Service Name value in the results. AutoArchiveProcessing is the Orchestration that Receives the Message while IncomingCSV_Receive is the Pipeline that processed the message.

In other words, these results represent the message as it was published to the MessageBox and Received by the Orchestration, in it’s XML format. Close, but this is not yet what we’re looking for.

Right-click the Send result from IncomingCSV_Receive, which is the Pipeline that processed the message. Select Show Tracked Service Instances.

A new Query tab is opened showing the Pipeline instance that processed the message, the one received the original message from the Adapter and then Published to the MessageBox. Right-click the IncomingCSV_Receive result and select Show Tracked Message Events.

A new Query tab is opened showing the message processed by that Pipeline instance. A Pipeline can receive only one message and can output 0 or more messages. In this result list, the Receive message, Schema Name = Unparsed Interchange, is the unprocessed message submitted by the Adapter. In this case, it is the content of the file, byte for byte. This is what we're looking for.

Right click on the Receive message and select Save to File. Click OK to accept the warning about sensitive data, then choose a folder to save the message content.

We have just saved an un-altered archived message.

Interpreting the Output

Saving the tracked message from BizTalk Administrator results in two files. An .xml and an .out.

Most of the time, we just change .out to .xml and open file to examine. Occasionally, we’ve been asked why both aren’t .xml. That’s simply because BizTalk Server does not require messages in XML format. Fundamentally, every message is really a binary stream which can be any type of content. So, the agnostic .out is more appropriate. Note: Multi-Part Messages will result in one .out for each Part.

If we open the .out in a text editor, we see the unaltered CSV content.

The .xml is the content of the Message Context. The important piece of information here is the original filename.

<Property Name="ReceivedFileName" Namespace="http://schemas.microsoft.com/BizTalk/2003/file-properties" Value="C:\Data\BizTalkFolders\AutoArchiveApp\IN\TrackTest_01122015.txt"/>

Conclusion

BizTalk Server provides a robust and very flexible archiving solution out of the box. It is literally a check-box implementation. By using these tools that BizTalk Server already provides, many of the problems associated with archiving can be easily avoided.

We will also take this opportunity to address a likely counterpoint to this, that is 'performance'. Unfortunately, Tracking in general suffers from a negative perception and there are many persistent recommendations to disable it completely. This is a bad idea. On current hardware, even virtualized, Tracking is barely measurable and it's usefulness in diagnostics and system monitoring far outweigh any perf tax it requires.

When used as an Archiving solution, Tracking is best value performance wise since the only additional cost is movement of the message data from the MessageBox (where it is already) to the DTA database. A critical point here is that this happens essentially out of band to the BizTalk Host Instances since the process is executed at the SQL Server level.

Share via