다음을 통해 공유

Streaming Concept in BizTalk-Part1

  • 아티클

Streaming:

Stream procesing support in BizTalk

BizTalk Implement forwad only streaming design.

  • Pipeline: Out of box pipeline use streaming.
  • Orchestrations: Streaming can be used when calling .Net components
  • Mapping: By default xslt loads messages into memory. BizTalk uses virtual streaming if message exceeds a threshold, data is written  into temp disc file, which becomes the stream source. This introduces I/O overhead, but most cases beats memory issue.
  • BAM: event streams when saving data  into BAM database

Advantage of streaming in pipeline components

  • Reduce memory consumption by not loading entire data into memory.
  • Speeds up processing: Next component doesn't wait for the component to finish processing the entire message.
  • Current component gives the processed chunk to the next component, and starts working on a new chunk.

Difference b/w streaming & non streaming components

The no straming concept:

Consider a pipeline processing without streaming.

An adapter which is listining for data on the wire creates a new BizTalk message type IbaseMessage and if there are any message part also creates part of type IBaseParts. And then attaches the data stream to this message and promote context properties related to the receive end point. The adapter than submits the message to pipeline

Note that adapter has not actually red the data from the wire. It just created a message type that BizTalk understood and it only attach the data stream into that message but no data has been red yet.Now the first component of the first stage of the pipeline gets the message at these movement.The componenet read the entire data stream attach to the memory at once. This effectively load the message into memory. The component will do what ever processing required , than it create another stream composed of new data and passes it along hte other component in the pipeline chain.The other component read the entire data stream at once, causingthe entire data to be loaded into memory.Thus the required processing create teh new stream and passes it to next component. This process continues untill al component of the pipeline are executed and the final message submitted by the message to the message box by the message agent.

Streaming approach:

Adapter attaches the data stream to the IbaseMessage and passes it to pipeline. The first component receives the messagte. This component than wrap the message with custom stream without reaching any part of it. Now the message with new stream attached is passed to the next component. The next component does the same , it wrap yet new stream around the one passes to it and then passes on to the next stage.

Now the message has a chain of two streams  and still no data is read. The message goes to the same process while going through all pipleline component.

Finally its EPM calls the Read() method of hte stream. At this movement calls gets propagated through all those custom streams, that have been wraped around message by the pipeline component.All the way to the main data stream of the adapter. The adapter than perform the firstactual data read and passes it along the stream to the first component. Each pipelinee component than execute its own implementations of the read method on that particular stream chunk it receives. After passing through all components this stream chunk is than handed to EPM which hand its over to the message agent. This continue untill the stream is red entirely and message is stored in the messagebox. Once the message is stored entirely in the messagebox. The EPM is notify the adapter , so that it perform resource cleanup. For example delete adapter delete file and adpater disposes connection. On send sides message agent hands adapter a reference to the message.Adapter tahn pulls first stream chunk, and passes it to pipeline. Chunk goes into the pipeline component.Data are sent out in streaming fashion.

See Also

Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.