Context 

A common requirement often encountered when implementing a system of record (SOR) is to allow near real-time dissemination of a record change to disparate systems. One example of a SOR is customer relationship management (CRM), which is responsible for maintaining records about an organisation’s customers.  

The need to distribute information from this central point stems from how CRMs are usually the source of truth for an organisation. Therefore, any changes to these records stored by the CRM will need to be propagated to other disparate systems to keep information updated and consistent.  

Connecting a central system to multiple separate systems can be fraught with complications. A few things that will need to be considered and discussed in this article are the issue of the overhead in the CRM and the compatibility of data between the different systems being integrated.  

The blog post aims to provide a practical example of common integration challenges involving the CRM Microsoft Dynamics 365 Customer Engagement (D365 CE) platform and backend systems using the Azure Service Bus (ASB) message broker and MuleSoft Anypoint Platform.  

Complexities and solutions 

Problem #1: 

One of the problems faced when synchronising information between a CRM and backend systems is the increasing amount of overhead required from the CRM as the number of applications go up. With a bigger number of applications that a central system needs to send information to, the more events the CRM will need to produce and send out for each one of them. This would consume more resources as the system performs more actions to produce more events.  

Solution: 

One option to rectify this problem is to decouple the CRM and the downstream system. A design pattern naturally suited to allow this is an event-driven architecture (EDA). An EDA allows multiple separate systems to receive near real-time data via events and can typically follow a publish/subscribe (also called pub/sub) model or an event stream model. 

Through an EDA, the onus of propagating an event to downstream systems is moved from the CRM to an event broker and integration adapters that will receive the data from the CRM source. Using a pub/sub model, the CRM will need to publish the event only once for the event broker to consume, which in turn can communicate the event to multiple subscriptions. Downstream systems can then subscribe to this broker to receive events.  

Problem #2: 

Another aspect of integration needed to be considered is the compatibility of the data produced by the CRM with the data expected by downstream systems. Additionally, different downstream systems will have different methods and protocols for receiving events e.g., RESTful APIs vs Web Service endpoints. 

If the event producer directly communicates with many downstream systems, multiple different events with contrasting data models are required to cater to each disparate systems’ demands.  

Solution: 

By introducing an integration platform, the event can be transformed into the format required by disparate systems. Moreover, an integration platform allows you to accommodate for the protocol used by target endpoints. 

Essentially, having an integration platform bridging the communication between the CRM system and the downstream systems insulates the event producer from the technical details of these system’s different interfaces.  

Problem #3: 

Lastly, as with any points of integration, there is a potential for failure in the transmission of messages from the CRM to the API implementation via an event handler. There needs to be mechanisms in place to ensure events are not lost if an error occurs, so that they will be replayed and reach their target systems. 

Solution: 

To ensure the delivery of events even after the occurrence of an error, we can apply error handling strategies for our event broker that will resend events if the publisher does not successfully get an acknowledgement from the target listener. Additionally, we can also implement an automatic retry mechanism in the message listener utilised by the integration platform itself to retry the processing of messages in the API implementation.  

To further refine the exception handling strategy, retries with exponential backoff can also be enforced, which will allow the event broker to retry after an exponentially increasing wait time. This can mitigate problems with message retries within seconds of each other that may all fail due to, for example, issues with connectivity during that period.  

It would also be beneficial to impose a maximum delivery count to ensure that we do not unnecessarily burden the system with repeatedly failing requests. Retries that exceed the max number set will then be sent to the event handler’s dead letter queue, from which events can be manually reviewed.  

Handling messages in your event broker queues, including the dead letter queue, is a strategy that will differ between organisations and implementations but should not be overlooked. Messages sitting idly in queues can quickly grow in numbers and reach the limits of resources for event brokers. Strategies can include enforcing the time-to-live of messages to specify the expiration of messages.  

A practical integration example 

High level view 

In a recently completed Deloitte engagement, the following technology stack were implemented to achieve an event streaming model that allows the flow of events from a central CRM to multiple different systems: 

• Microsoft Dynamics 365 Customer Engagement (D365 CE) platform  
• Azure Service Bus (ASB) 
• MuleSoft Anypoint Platform 

The ASB was the chosen event broker:   

• Azure is an established Cloud platform within the organisation 
• D365 CE has a customizable in-built functionality to publish events to ASB Topics 
• ASB can broker the publishing and subscribing of messages to a topic within a namespace but also supports First In, First Out (FIFO) message delivery with the use of queues. 

Queues facilitate the consumption of messages by one receiver. But by using message delivery via topics and subscriptions this allows scaling for the consumption of multiple recipients. This functionality allows for D365 CE to publish an update to one topic, which can be propagated to several subscriptions. Each subscription can be subscribed to by a variety of MuleSoft adapters customised to handle request payloads for specific downstream systems 

MuleSoft Anypoint Platform provides a supported out-of-the-box message listener for ASB subscriptions to receive messages published by D365 CE. Additionally, REST/Web Service connectors provided by MuleSoft can be used for communications with external systems. In addition, Anypoint Platform is also equipped with DataWeave scripting, which allows highly customisable data transformations.  

D365 CE Notifications  

The enablement of the publishing of messages from D365 CE to the ASB can be achieved through an Azure-aware Plugin. Microsoft provides an out-of-the-box interface,  IServiceEndpointNotificationService, within the Microsoft.Xrm.Sdk Namespace, which facilitates the publishing of an execution context according to a specific business event (such as an update to a  

record).  

Azure Service Bus (ASB) 

To handle the funnelling of messages to specific subscriptions where needed, system properties or custom application properties can be applied. For example, adding a Label to a subscription’s configuration filters messages by requiring them to meet a conditional expression for the message to be received. In this way, the ASB allows for control over which subscriptions will receive the event published by D365 CE.  

MuleSoft Anypoint Platform 

A ready-to-use MuleSoft ASB Connector Message Listener  can be configured to subscribe to an ASB topic. The key fields to configure are as follows: 

For the ASB Connector Configuration, the Service Namespace, shared access key name, and shared access key are required: 

For the connection to downstream systems, the out-of-the-box MuleSoft connectors can also be used. The HTTP Connector is used for REST APIs while the Web Service Consumer is used for sending request payloads to web service endpoints. 

Conclusion 

CRM platforms allow organisations to manage information about their customers but integrating them with external systems may not always be straightforward. Leveraging the pattern of an EDA mitigate some of the issues that comes with CRM integration, as demonstrated by integrating D365 CE with downstream systems using the ASB and MuleSoft Anypoint Platform. 

I hope this post will be helpful in your integration endeavours!