BAM: Tracking Portal
As we all know BizTalk Server 2004 provides a great set of enabling technologies to address a number of business problems, specifically around integration and business process management, the very nature of this technology means that BizTalk is typically deployed into the very heart of the enterprise and often has very mission critical applications running on it. With this of course comes a big responsibility in terms of how we build real enterprise class applications on BizTalk. Over the coming weeks and months I intend to push out to a broader community some of the learning’s and findings that I beleive make a real different in terms of increasing the chances that a BizTalk Server deployment will be successful, not only in terms of getting it to go-live but also around ensuring it’s cost of ownership is kept to a minimum, I’ll try to keep these all under the theme of Enterprise Architecture. We’re also doing a much bigger piece of work around this but more about that another time :-).
The first of these areas is the Tracking Portal, Darren has been hounding me (and rightly so) to blog some stuff on this for quite a while now, so dude, here it is…finally :-). The Tracking Portal often takes a back seat to getting the solution out of the door, the fact of the matter is however that tracking is core functionality in my opinion that needs to be designed in from the outset. A Tracking Portal really needs to surface two views into the solution, first it needs to provide a view into the business for the users of the applications sometimes this means MI but other times simply the business view of long running transactions, most people get this and also that BAM is well placed to provide this, and I have to say what great technology BAM is and it usually provides the ‘wow’ to the business users. The second view is often neglected, but it’s equally as important, it needs to provide a view into the solution from a functional perspective for the solution support team to use in order to troubleshooting the live system. There are other tools like HAT that can be used to troubleshoot, but HAT does not have the understanding of what combination of message flows constitute a business transaction, and typically this is what both the business and support users care about rather than individual discrete messages.
The approach that we have been using over here in jolly old Blighty for a while now, is to build the Tracking Portal using BAM to capture the key state and business transaction information, and SQL Reporting Services, (another technology that I think is very cool indeed), to render the Business and Support views from the tracking database. Using this approach the Tracking Portal can be put together very cheaply with a rich reach UI, and in my experience providing these views, especially the Support view could get you out of some very difficult situations after you goes live. Ok, so let’s dig a little deeper…
Using BAM to Capture Key Business Transaction Information
There are essentially three way you can use BAM, TPE, BufferedEventStream or DirectEventStream, the first TPE, allows you to define what tracking data is captured at specific points in an Orchestration, while this is great to setup quickly it has the problem that every time you change your Orchestration you need to redefine your TPE. The second problem is that you can’t use it in the pipeline, for some scenarios this presents a break in your tracking data between the time the business transaction reaches the adapter to the time the message was published by the messaging engine and routed to the appropriate orchestration. For me, these two reasons typical cause me to discount TPE.
The other two are very similar to each other in usage, the main difference being in the way that the tracking data is transferred to the BAMPriamryImport DB, the BufferedEventStream is asynchronously processed which essentially means is less of a performance hit, while DirectEventStream causes the tracking data to be written directly to the BAMPriamryImport database.
In general the BufferedEventStream is preferable for performance reasons. As I mentioned earlier, to get a full view of how business transitions flow it generally means that the BufferedEventStream needs to be called from a custom pipeline component in the receive pipeline, the orchestrations, and again in a custom pipeline component on the outbound path, so providing a simple stateless wrapper API around the calls to the BufferedEventStream is a good approach, this API can then be called from a pipeline component or an orchestration.
In order to tie up the individual messages into a message flow which represents a ‘business transaction’, a unique business transaction ID is needed, often there is the notion of some unique identifier for message flows depending on the scenario, but in the scenarios where there isn’t, the system context property InterchangeID maybe used since it is guaranteed to be unique and will be flowed with the message even as the message is cloned through each stage in the pipeline. If the message is copied in the orchestration it is the user who responsible for ‘flowing’ the InterchangeID.
Each new business transaction will start a new Activity, this is done by calling the EventStream.BeginActivity() API, an activity is ended by calling EventStream.EndActivity(), of course for business transactions that span the receive pipeline, and orchestration etc, BAM has the concept of continuing an activity, for this the EventStream.EnableContinuation() API is called passing in a unique identifier to tie the activities together. The property BufferedEventStream.FlushThreshold is used to control the frequency at which the events are flushed to the BAM database, in general setting this to ‘one’ works well. Neither the DirectEventStream or the BufferedEventStream are serializable currently, the implications of using them from Orchestrations are that they need to be created each time they are used rather than defining a local variable in the Orchestration, this may sound like an issue, but in the past we’ve taken customer scenarios into the perf lab and profiled the BizTalk solution to understand the cost associated with using the EventStream in this manner, in the grand scheme of things the cost of creating it and using it with a flush threshold of one is not significant.
Pipeline components can get an EventStream from the pipeline context using IPipelineContext.GetEventStream(), this was added in QFE 1117, and is in SP1, this ensures transactional integrity between the tracking data and the message box interactions.
One final point is that it is possible to track message bodies also using BAM, though the restriction is that there is a limit to the size of the message bodies of 3885 bytes, which isn’t that big. Having said that, using this approach outlined here, we’ve measured the cost of this tracking on a complex customer scenario, and found it to be in the order of 20% slower than the same solution with no tracking, which is pretty impressive.
Using SQL Reporting Services to Build the Portal
Ok, so now we have a very flexible mechanism to collect tracking data for business transactions at run-time, we need a slick way to render the web based tracking portal, enter SQL Reporting Services. SQL RS can be used to build the tracking portal views (business focused and support focused) very easily, it allows you to use drag and drop techniques to build the UI and define the layout of the report. Since the portal is built on SQL RS, it’s really easy to provide rich querying capabilities as well as hierarchical views that for example allow you to look at a business transaction then drill down to the individual message flows. The approach here is to point SQL RS at the BAM views that generated when the activities are deployed, and allow SQL RS to query over these views.
Comments
- Anonymous
March 29, 2006
While working on a BizTalk POC last week, I needed to track Business Activity Monitoring (BAM) data...