In the modern enterprise, it is rare to see a single monolith application that rules the entire landscape. While there still may be a large mainframe or another system which holds the principal data and source of truth (SoT) for an organisation, most landscapes today feature a medium to large set of applications that fulfill various business capabilities. These can range from a handful of apps to literally hundreds depending on the size and complexity of the enterprise.
Whilst it is clear that the cost of integration increases along with the number of applications, one can also argue that the cost of change decreases as you move away from the “one app does it all” model. The reasoning behind this argument is that every single change, no matter how small, typically results in a full redeployment of the monolith application and necessarily incurs the cost of regression testing for the entire system.
But aside from cost and the numbers of applications, there is an additional dimension to be considered – time. Not all apps are created equal. The problem with any architecture diagram is that is represents a single point in history – it is essentially a snapshot. The reality is that applications change over time; some are upgraded, other are modified or expanded, and others may be dropped or replaced. The rate of change varies across these applications, as some systems change more slowly than others. This can be represented in a layered view:
The concept of layers in an application architecture is not new; Gartner created the Pace Layered Application Strategy almost ten years ago to address the common disconnect between business leaders (who want their systems to be flexible and adapt to changes in the business landscape) and IT owners (who typically want to keep systems consistent as long as they are running smoothly). Identifying these varying rates of change and grouping the applications accordingly helps to apply an appropriate level of governance, change control, testing and DevOps – enabling the business to innovate where it needs to whilst protecting its critical data and core processes.
Gartner’s Pace-Layered model consists of three layers:
Image based on presentation by Michael Guay (Gartner)
Let’s look at an example enterprise, for example a banking institution, and see how some of their applications might map to the pace-layered model.
LAYER | SYSTEMS / APPLICATIONS | CHARACTERISTICS |
Systems of Record |
ABC Bank has several key systems including a core banking system, a loan management system, and a document repository. |
|
Systems of Differentiation |
The automated loan processing capability is managed by a custom-built integration solution which integrates with several externally SaaS services for real property valuations, title searches, credit scoring, and an online web forms provider. |
|
Systems of Innovation |
ABC Bank wishes to expose a chat-based interface for lodging loan applications powered by an intelligent bot. |
|
So now that we understand the pace-layered model, how do we implement integration within it? Let’s look at how a logical model of APIs/Services to see how they combine into applications across the layers:
Starting from the bottom layer, we see that each System of Record is typically a package which may contain several services/APIs. However, these APIs may not be directly consumable by the business due to misalignment with the logical data model, outdated protocols, or other reasons. In these cases, it is a good idea to introduce a “sub-layer” of APIs that bridge the SoRs with other APIs within the organisation. These abstraction APIs (called Product Adapters) are tightly coupled with the underlying SORs yet expose functionality in a much more palatable format. They may also introduce stricter access control, validation and security than the SORs themselves. The APIs generally represent the core data entities (Customer, Product, Order, etc.), therefore they are granular and designed for reusability. Because they are so tightly linked with the core systems, they move at the same pace and therefore are considered part of the Systems of Record layer. Governance and change control will normally be very strict at this level because of the criticality of the data and the high dependency of services and processes that consume these APIs.
In the Systems of Differentiation layer, we see applications that are composed of the granular services/APIs sourced from the System of Record layer as well as possibly external APIs. Here is where the business logic of the organisation lives, for example Loan Processing or User Provisioning. Functions that an application may perform in this layer include data aggregation, routing, filtering and often choreography/orchestration. Since they are specific to a process, they are less reusable than the underlying SOR APIs that they may consume. It is in this layer that the bulk of the integration within an organisation occurs. And because business processes can (and will) change over time, these applications will need to adapt as well, most certainly at a faster pace than the SOR applications. Governance should also be applied at this level, although perhaps not quite a rigidly as within the SOR layer; the organisation will want their business processes to be flexible enough to accommodate improvements in efficiency and expanding functionality.
The Systems of Innovation layer also features applications that consume both the SOR APIs and external APIs, as well as possibly business processes in the Systems of Differentiation layer. As the fastest moving layer, it will have even lighter governance in order to facilitate the experimentation of new apps and technologies. The functionality enabled in this layer is often peripheral to the core capabilities of the business, and therefore presents less risk to the organisation in case of failure. Also apps that are created quickly to prove a concept will rarely feature automated testing or a mature CI/CD pipeline, as they would be manually deployed and tested.
Finally, we use a Message Bus in order to facilitate both inter-layer and intra-layer communication. Asynchronous messaging patterns such as publish-subscribe keep systems loosely coupled and promote both extensibility and flexibility. The publisher doesn’t need to know anything about the subscriber, and you can always add or subtract subscribers without breaking existing integrations. A message bus is a critical element for lubricating the friction between applications that change at different paces.
This diagram provides a cross-section view of several attributes and how they vary across the layers:
Microsoft offers a range of services and products, both on-premises and in the cloud, to help build powerful integration solutions that can cope with the different paces of enterprise application layers. Here we will discuss only some of these offerings and how they might fit into a pace-layered architecture (note there are many possible solutions; these suggestions are only one possible way to look at this):
The following products lend themselves very well to build the API layer on top of the SOR applications:
TECHNOLOGY | USE WHEN | CONSIDERATIONS |
Product APIs |
|
+ Tight integration with System of Record – Difficult or expensive to change or customize – May not suit business data model |
Web Service / REST APIs |
|
+ Inexpensive to host + Easy to consume + Can be hosted on-prem or in Azure (IaaS) – Development effort required |
API Management |
|
+ Customizable façade + Developer portal facilitates new app creation – Requires VNet Integration – No on-prem option – Expensive option if not using additional features |
Service Fabric |
|
+ Can host anywhere + Supports containers – Significant development effort required (no OOTB software components supplied) |
BizTalk Server |
|
+ BAM tracking available + Single platform integration – Expensive option – Specialised dev skills required – Future support model in question |
These products are capable of implementing business logic, providing connectivity to internal and external applications and services, and supporting hybrid connectivity for applications that span both the cloud and on-prem:
TECHNOLOGY | USE WHEN | CONSIDERATIONS |
Logic Apps |
|
+ Rapid development + More than 200 built-in connectors! – No VNet support (until ISE becomes available) – No on-prem option (yet) |
Azure Functions |
|
+ Good CI/CD support + VNet support + Can run on-prem – Not as many connectors as Logic Apps |
Web/Mobile Apps |
|
+ Good CI/CD support + Numerous deployment options + Azure Relay / VNet Integration supported – Not ideal in themselves for long running processes – Consider security layer for hybrid apps |
Service Fabric |
|
+ Can host anywhere + Supports containers – Significant development effort required – Infrastructure investments (on-prem only) |
BizTalk Server |
|
+ Single platform for integration + Can be hosted on-prem or in Azure (IaaS) – Expensive option – Specialised dev skills required – Future support model in question |
Here we need the ability to expand the range of technology into the areas of artificial intelligence, predictive analytics, and business insight whilst enabling rapid (even ad hoc) development. There are numerous possibilities here, as it is less about which technology you use as opposed to how you use it. But these products all lend themselves well to innovative solutions:
TECHNOLOGY | USE WHEN | CONSIDERATIONS |
Microsoft Flow |
|
+ Rapid development + Can be easily migrated to Logic Apps * Requires Office365 |
Power Apps |
|
+ Easy integration with Flow / SharePoint / + Multi-platform * Requires Office365 |
Power BI |
|
* Relies on access to data sources |
Cognitive Services |
|
+ Multiple services/APIs available (Vision, Knowledge, * Programming skills required |
Machine Learning |
|
* Data science skills required |
Bots |
|
* Programming skills required * Bots need to be trained well to function as expected |
If you are integrating mainly on-premises systems, BizTalk Server is at its heart a powerful messaging engine which can not only support the full gamut of messaging patterns, but also provide several out-of-the-box adapters for connectivity as well as a robust business process automation capability. This is why it features within so many layers. When integrating within the cloud however, Azure Service Bus provides a number of offerings for enterprise messaging, big-data streaming, event handling, and hybrid connectivity:
TECHNOLOGY | USE WHEN | CONSIDERATIONS |
Event Grid |
|
+ Resilient (retries up to 24 hours) + Push-push model + Easy to integrate * Small message size
|
Event Hubs |
|
* Needs at least one downstream processor – No on-prem option |
Relays |
|
+ Can use Hybrid Connection or WCF Relay |
On-Prem Data Gateway |
|
+ Good alternative to a VNet if using Logic Apps – Only supported by a handful of Logic App connectors |
Service Bus Queues |
|
+ Extremely resilient and full-featured – No on-prem option |
Service Bus Topics |
|
+ Extremely resilient and full-featured – No on-prem option |
BizTalk Server |
|
+ Single platform for integration – Expensive option – Specialised dev skills required – Future support model in question |
Here are some tips on how to maintain adaptive integration within a pace-layered enterprise architecture.