Share via

Azure LogicApps: Common EAI design patterns

  • Article

Introduction

"Enterprise integration is too complex to be solved with a simple 'cookbook' approach. Instead, patterns can provide guidance by documenting the kind of experience that usually lives only in architects' heads: they are accepted solutions to recurring problems within a given context. Patterns are abstract enough to apply to most integration technologies, but specific enough to provide hands-on guidance to designers and architects. Patterns also provide a vocabulary for developers to efficiently describe their solution.

Patterns are not 'invented'; they are harvested from repeated use in practice. If you have built integration solutions, it is likely that you have used some of these patterns, maybe in slight variations and maybe calling them by a different name. The purpose of this site is not to "invent" new approaches, but to present a coherent collection of relevant and proven patterns, which in total form an integration pattern language." – (Source: ‘EAI Messaging patterns’)

 All quotes in this chapter are copied from http://www.enterpriseintegrationpatterns.com/.

 1.1  Claim-check (Luggage handling)

Because the standard size-limit for messages in Azure service bus is 265 kb. A common pattern will be to store large messages in a separate (BLOB) storage during message processing. This pattern is referred to as the "Check luggage" pattern, because this works like checking-in and retrieving your luggage during a flight. The “Reading” app will store the payload and will pass its location to a message queue. A second app will then read the message queue, will retrieve the payload, and continue processing.

Above figure shows a simplified example of an existing LogicApps implementation. ​

  1. This first LogicApps will store the (large) message as a new file in Azure Blob storage.
  2. Then create a message in a message queue, containing the container (folder) and filename.

Above figure shows a simplified example of an existing LogicApp implementation. ​

  1. A second LogicApp will be triggered by the new message created in the Service Bus queue.
  2. This LogicApp will retrieve the (LARGE) message from the Blob storage, by specifying the container and filename from the message contents.
  3. Subsequently the process will execute all required actions for this message.
  4. Afterwards the message will be completed in the queue,
  5. and the blob might be deleted.

1.2  Request and reply

When two applications communicate via Messaging, the communication is by default one-way. To be more transparent about the completeness of the interface, allowing better maintainability, recommendation will be to implement two-way conversations where possible. To do so, the logic app receiving messages, will also update the sender after the message was successfully transferred.

The next image will show the simplest example of a Logic App responding to a success. In case of a file-based interface, you might decide to only delete / archive the file when successfully processed. When data was retrieved from a table, only update the status to SUCCES after completion. All these examples are basically implementing the same design pattern.

Above figure shows a simplified example of an existing LogicApp implementation. ​

  1. This LogicApp will be triggered by an HTTP-Post call from a sending application.
  2. The connection will remain open until a response has been returned to the Sender. This to ensure the nested actions have been performed (Persisted).
  3. After all nested activities have been executed as expected.

1.3  Content based routing

"The Content-Based Router examines the message content and routes the message onto a different channel based on data contained in the message…”

*
*

Above figure shows a simplified example of an existing LogicApp implementation. ​

  1. This LogicApp is used as a single API-Endpoint for receiving all kinds of messages from a holding organization.
  2. [optional] After receiving the message, the contents are being AS2 decrypted.
  3. After receiving a (HTTP Post) message (and possibly decrypting the contents), a compose activity is used to extract the "Message type".
  4. Bases on the "Message type", different actions might apply. As a result, the messages will usually be send to a different Queue.
  5. The DEFAULT case might send all remaining messages to a "Dump Queue» or might send an error response back to the HTTP-Sender.
  6. After executing the Switch-Case logic, a success (Code 200) response is being returned to the HTTP-Sender.

1.4  Splitter

"A Splitter is useful to break out a single message into a sequence of sub-messages that can be processed individually."

In this example the Logic App might receive a single message containing an XML containing multiple messages which can all be processed without dependencies. Another example will a situation where the app will retrieve multiple rows of data from a database, then send each row to be processed separately. This will reduce the amount of connections to the database and will allow leveraging the scalable capacity of Azure by processing subsequent processing steps in parallel.

Above example shows a LogicApp retrieving multiple records from a database, then sending a separate message for each row, also updating the state of each record one-by-one.

Above a second example, where the LogicApp will receive a text file, then apply “Flat File Decoding” action to transform this message into an XML format. Then use a “Compose” action to transform this into a JSON format. Then loop all [Line]s in the [Body] of the [Root] of the XML and send a message for each. Afterwards the entire file can be deleted / archived / marked as completed.

1.5  Pipes and filters

The next design pattern is based on the “single responsibility principle”. This principle states that every ‘class’ should have a responsibility over a single part of the functionality. In LogicApps this would lead to a solution in which a pipeline of connected LogicApps each perform a single action to a message.

A first app might for example receive inbound messages and put these in a message queue. A second might (AS2) decrypt the message. A third might validate or transform the message, and a last LogicApp will send the results to a target application.

Each of the LogicApps might transfer messages via a Message queue, storage queue, database or file storage. Advantage of this pattern, is the fact that:

  • Each logic app will perform a simple operation.
  • The location where a message gets stuck, indicates what action had failed.
  • Each step in the process can be individually enabled / scaled / improved.

Main drawback of this design, will be the fact that messages can get stuck in many different message queues. As a result, monitoring and debugging will become more complex. To overcome this, we recommend passing a ‘Correlation Identifier’ to help following messages through the pipeline.

1.6  Load leveling (queue based)

“Pioneered by the Lean guru Taiichi Ohno, referred to load leveling as “heijunka” (HIGH JUNK AH), meaning a technique for reducing the mode of (waste). Load leveling is a method for reducing large fluctuations in customer demand.” - [Source

](http://www.sixsigmadaily.com/load-leveling-production-leveling-production-smoothing/)

Load leveling is a principle originating from the ‘lean / six sigma’ management philosophy. The principle explains the fact that spreading your work-load over time, will prevent high peaks and low valleys in the required production capacity. Even though this will limit the maximum throughput of an interface, the principle will also reduce the impact on a target application.

When building such a pattern in LogicApps, a first app will receive (high volume) of (potential parallel) messages and put these on a queue. A second app will then periodically retrieve a fixed number of messages from the queue and process these one-by-one.

1.7  Idempotent receiver

The word “Idem” is Latin for “The same”, “Matching” or “Equal”. As a result, the word ‘Idem-Potent’ is an object capable of handling duplicate inputs. The word ‘IdemPotent’ is also used in Math, to describe a function for which its output does not change, when its own output is again used as input for the same function.

f(f(x)) = f(x)

In the example of LogicApps, this results in a LogicApp in which we first check whether the action was already performed, before executing the action itself. For example, check whether a sales order is already existing, when not existing create the sales order.

By applying this pattern, you will allow all historical messages again, without changing the end-results. This will also allow your app to retry the processing, until no more errors exist. For example, when an order cannot be inserted because an item is blocked, then retry until the item is no longer blocked, then complete the order entry process.

Implementing this patter is highly recommended in cases where you also wat to implement the ‘Competing Consumers’ pattern. This because two instances of LogicApps running in parallel, might try to insert the same order twice. By adding the idempotent receiver pattern, this risk is limited. (not eliminated)

1.8  Circuit breaker

The ‘Circuit breaker’ pattern is a solution for preventing errors later in the pipe-line, by running a small health check up front. For example, first check whether D365FO is online (by retrieving the ‘SystemParameters’ entity) before asking the message queue for messages to be send towards D365FO. When the initial circuit breaker failed, do not bother picking messages from the queue, preventing the maximum retry of messages to be hit, for example when D365FO is temporarily unavailable during maintenance hours.

 

1.9  Other useful design patterns

  • Messaging gateway / ESB
    **

    **Instead of creating a direct interface between each pair of applications, create an interface from each application to a central gateway with the role of transporting the message to other systems. And extended function of this gateway would be to also have the gateway coordinate and transform messages, which is called an “Enterprise Service Bus” (ESB).
    http://www.enterpriseintegrationpatterns.com/patterns/messaging/MessagingGateway.html

  • Process manager
    **

    **In situations where you’re the process your message needs to go through exists of many steps, potentially depending on other applications or external parties, a ‘Process manager’ Logic App might be implemented to guide a single message through all required steps. This will centralize coordination, but will also create a single point of complexity (read: failure).
    http://www.enterpriseintegrationpatterns.com/patterns/messaging/ProcessManager.html
    **
    **

  • Event messages (Event hub)
    **

    **Event messaging is the principle one system transmitting messages, and many systems potentially subscribing to channels / event types. The transmitting system will be able to send messages without bothering whether (or how many) systems will be receiving the messages. The transmitting application is also not (directly) informed of each successful delivery. This pattern can be closely compared to Email newsletters. A company will create the newsletter, and everyone can subscribe. When a new letter is created, all subscribers will receive a copy. When a new subscriber is added this subscriber will not receive all previous letters.

    When volumes are very high, recommendation will be to use ‘Azure Event Hubs’ instead of Logic Apps for this type of messaging. Drawback of event hub, is the fact that messaging is not guaranteed to be presented. When a receiving LogicApp is temporarily unavailable, messages received during this period will get lost.
    **
    **

  • Competing consumers
    **
    **In the pattern of ‘competing consumers’, multiple copies of the same LogicApp will be reading messages from the same message queue. Each instance will pick the first message, which will temporarily be blocked for processing by other instances. Main advantage of this model, will be the fact that all instances will be processing on full capacity, until all work has been completed. This model will scale to a very high throughput.

Priority queue

**

**The ‘Priority queue’ is a pattern where Incoming messages are sorted into multiple queues based on their priority. Messages are then being read by consumer LogicApps. More apps or more capacity is assigned to the ques with higher priority messages. This can for example be applied in a process sending customer information from CRM to D365FO. Newly created customers might have more priority than customer updates.