In the previous post of the series on the Enterprise Integration Patterns on Azure, I explored the Messaging Endpoint patterns, which abstract integration interfaces from the application internals when building messaging-based integration solutions. In this post, I will cover the Message Routing patterns on Azure, which provide guidelines to decouple the source from the intended receivers by implementing message routing in the integration layer.
This post is a part of a series describing how to implement the Enterprise Integration Patterns using the Azure Integration Services:
- Introduction
- Message Construction
- Messaging Channels
- Messaging Endpoints
- Message Routing (this)
- Message Transformation
- Platform Management
The remaining posts will be published in the following weeks/months.
The patterns covered in this article are listed below.
- Pipes and Filters
- Content-Based Router
- Recipient List
- Message Filter
- Message Validator (*)
- Dynamic Router
- Load Balancer (*)
- Splitter
- Aggregator (Batching)
- First-in, First-out (*)
- Re-sequencer
- Composed Message Processor
- Scatter-Gather
- Routing Slip
- Process Manager
- Message Broker
Pipes and Filters
The Pipes and Filters is an architectural style that allows us to divide a large process into a sequence of smaller and independent steps. Each step is called a filter and they are connected using pipes.
Implementation
|
Service Bus or Event Grid can act as pipes. |
|
Logic Apps and Azure Functions can act as filters. |
Content-Based Router
The Content-Based Router pattern suggests that a router is in charge of routing messages into the corresponding channels based on their content. It can be used as part of the Publish-Subscribe Channel pattern.
Implementation
|
Azure Service Bus topics and subscriptions can be used as a Push-Pull Channel to route messages based on their Message Header (*). Messages put into a topic can be routed to zero, one or more topic subscriptions. Different types of filters can be used for routing. |
|
Event Grid subscriptions provide content-based routing on a Push-Push Channel. Event Grid provides different options for filtering. |
Recipient List
The Recipient List pattern is similar to the Content-Base Router pattern. However, instead of sending the message to a single recipient, the message is routed to multiple receivers. This recipient list is defined dynamically based on rules and the content of the message.
Implementation
|
Azure Service Bus Topics and Subscriptions can be used as a Push-Pull Channel to route messages based on their Message Header (*). The list of subscriptions that get the message is defined dynamically at run time based on filters.
|
|
Event Grid Subscriptions provide content-based routing on a Push-Push Channel. The list of subscriptions that get the message is defined dynamically at run time based on filters. |
Message Filter
A Message Filter can be implemented so that messages that do not match a receiver’s criteria are discarded for that receiver. A filter can be stateless or stateful.
Implementation
|
When using the Azure Service Bus topics subscriptions, messages that do not match any topic subscription filters are discarded. Filters in topic subscriptions are stateless. Additionally, Service Bus supports message deduplication. This feature is a stateful filter. |
|
When utilising Event Grid subscriptions, messages that do not match any subscription filters are discarded. These filters are stateless. |
Message Validator (*)
A type of Message Filter is the Message Validator (*). In many cases, we need to validate if messages have a valid structure/data, and “dead letter” those which don’t. Even though this pattern is not described in the Enterprise Integration Patterns book, it is commonly implemented in messaging-based enterprise integration solutions.
Implementation
|
Messages can be validated in Logic Apps workflows.
|
|
With Azure Functions, we can use a JSON schema to validate a payload or even use FluentValidation. |
Dynamic Router
The D pattern suggests that potential receivers can create or update routing rules at run-time based on changing conditions.
Implementation
|
Service Bus allows you to update the topic subscription rules at any time. Subscribers could potentially modify these rules based on dynamic conditions. |
Load Balancer (*)
The Load Balancer Pattern allows you to deliver messages to one of a set of endpoints using different load balancing policies. The difference between the Load Balancer (*) and Competing Consumers is that the load is distributed by a dispatcher and not by the consumers. This pattern is useful to distribute load across multiple regions or to minimise the dispatch of messages to unavailable endpoints. The most common load balancing policies are round robin, random, based on performance, and failover.
Implementation
|
Service Bus provides customer managed failover for regional (geo) disasters. |
|
Event Grid provides server-side regional failover for disaster recovery. |
Splitter
The Splitter pattern proposes that messages that contain multiple elements or records can be broken out into a series of individual messages so that each one is processed independently.
Implementation
|
Logic Apps allow us to split a message array into independent messages using the SplitOn property. With this approach, each array item starts a new workflow instance. An alternative is to use the ForEach loop structure within a workflow. |
|
With Azure Functions, a batched message can easily be broken out into individual messages using custom code. |
Aggregator (Batching)
The Aggregator pattern suggests that related messages can be grouped so they can be processed as a whole. The Aggregator requires persistent storage to collect and store individual messages and a mechanism to define how messages are combined and when they are ready to be published.
Implementation
|
Logic Apps provides a way to batch related messages and specify criteria for releasing and processing those messages. Messages are aggregated or batched based on a unique key or partition and can be released based on message count, batch message size, or a schedule. |
First-in, First-out (*)
Service Bus sessions can be used to create a Message Sequence construct. The next question is, how can we make sure that the messages are processed in the intended order once they reach the receiver? The First-in, First-out (*) (FIFO) pattern intends to describe this.
The Enterprise Integration Patterns book does not describe this pattern as such. It describes how to construct a Message Sequence and how to re-sequence them when required (Re-sequencer). But when messages are in the right order in the Messaging Channel, how can we make sure that they are processed in the correct order?
Given that the FIFO pattern has an impact on the solution throughput and complexity, alternative approaches must be considered as suggested here.
Implementation
|
Service Bus with Sessions supports the First-in, First-out pattern thanks to its Push-Pull nature. However, you need to be aware that:
|
|
Logic Apps together with Service Bus can be used to implement the FIFO pattern as described here. |
|
Azure Functions together with Service Bus can be used to implement the FIFO pattern as described here. |
Re-sequencer
The Re-sequencer pattern might be required when First-In, First-Out processing is required, but the channel cannot provide sequential delivery. This pattern requires an internal buffer to store the out-of-sequence messages until a complete sequence is obtained. Different approaches are required for bounded and unbounded sequences. The solution must also be resilient to potential scenarios such as head-of-line blocking and missing or failed messages.
Implementation
|
Logic Apps or Azure Durable Functions together with a persistence layer like Azure Functions Durable Entities, Azure Storage, or Cosmos DB could be used. Custom logic would be required to put the messages in the right sequence before they are processed or delivered to the intended receiver. |
Composed Message Processor
The Composed Message Processor is a composed pattern that is comprised of a Splitter, a Content-Based Router, and finally an Aggregator.
Implementation
The implementation of each of the parts of this composed pattern is described previously in this article.
Scatter-Gather
The Scatter-Gather pattern is useful when a message must be broadcast to multiple recipients and their responses must be reaggregated into a single message.
Implementation
|
Logic Apps allows you to send requests to multiple recipients and wait for their responses using the Webhook action, as described here. You can define a timeout to wait for the responses. The number of recipients must be known at design time. |
|
Azure Durable Functions allows us to broadcast requests, wait for their responses, and aggregate them as described in this post. A dynamic set of receivers can be defined at run time. |
Routing Slip
The Routing Slip pattern is useful when we need to dynamically route a message through a series of processing steps which are unknown at design time. A routing slip is defined based on the message content and business rules, and can be attached to each message at run time.
One way to implement this pattern is to rely on the Pipes and Filters architectural style and add the routing slip as a message header in an initial filter. Subsequent filters would process the message and pass it to the next filter based on the header state. Each filter must update the header.
Implementation
|
Logic Apps and Azure Functions can act as filters. |
|
Service Bus can act as a pipe. The routing slip can be a Service Bus message header. |
|
Event Grid can act as a pipe. The routing slip can be an event message header. |
Process Manager
The Process Manager pattern allows us to route a message through multiple steps which are not necessarily sequential or known at design time. The Process Manager is commonly defined via a workflow or business rules and must keep a state of the processing sequence. This pattern can be subdivided into the Orchestration and Choreography patterns. The Orchestration pattern is a Process Manager that is tightly coupled to the individual steps using Command Messages. In the Choreography pattern, the process manager typically uses Event Messages to communicate with the sub-tasks so that communication is not tightly coupled.
Implementation
|
Logic Apps provides rich workflow capabilities to implement a stateful Process Manager. |
|
Azure Durable Functions provide workflow-as-code capabilities to implement a stateful Process Manager. |
|
When temporal decoupling is required, Service Bus can serve as a channel for command messages in an orchestration implementation. |
|
Event Grid can serve as a channel for event messages in a choreography implementation. |
Message Broker
The Message Broker pattern allows us to fully decouple the message senders and receivers while maintaining central control over the flow of all message. The Message Broker can receive messages from multiple sources for multiple destinations and determine the correct destination based on business rules.
Implementation
|
A Logic App workflow can route messages to the corresponding channel based on the message content and business rules. |
Wrapping Up
In this post, we have covered the Messaging Routing patterns and how to leverage the Azure Integration Services to implement them. Understanding these patterns allows us to consider different approaches when architecting integration solutions. Stay tuned for the next instalment covering the Message Transformation patterns.
Happy integration!
Cross-posted on Paco’s Blog
Follow Paco on @pacodelacruz