Integration at Pace Within the Adaptive Enterprise

In the modern enterprise, it is rare to see a single monolith application that rules the entire landscape. While there still may be a large mainframe or another system which holds the principal data and source of truth (SoT) for an organisation, most landscapes today feature a medium to large set of applications that fulfill various business capabilities. These can range from a handful of apps to literally hundreds depending on the size and complexity of the enterprise. 

Whilst it is clear that the cost of integration increases along with the number of applications, one can also argue that the cost of change decreases as you move away from the “one app does it all” model. The reasoning behind this argument is that every single change, no matter how small, typically results in a full redeployment of the monolith application and necessarily incurs the cost of regression testing for the entire system.

But aside from cost and the numbers of applications, there is an additional dimension to be considered – time.  Not all apps are created equal. The problem with any architecture diagram is that is represents a single point in history – it is essentially a snapshot. The reality is that applications change over time; some are upgraded, other are modified or expanded, and others may be dropped or replaced. The rate of change varies across these applications, as some systems change more slowly than others. This can be represented in a layered view:

The concept of layers in an application architecture is not new; Gartner created the Pace Layered Application Strategy almost ten years ago to address the common disconnect between business leaders (who want their systems to be flexible and adapt to changes in the business landscape) and IT owners (who typically want to keep systems consistent as long as they are running smoothly). Identifying these varying rates of change and grouping the applications accordingly helps to apply an appropriate level of governance, change control, testing and DevOps – enabling the business to innovate where it needs to whilst protecting its critical data and core processes.

Understanding the Pace-Layered Architecture

Gartner’s Pace-Layered model consists of three layers:

• Systems of Record (SOR) – These systems enable the core capabilities of an organisation, without which the business simply could not run. Because these are typically standard across the industry, these capabilities are not unique to a given brand or business (for example, every bank needs to manage accounts, transactions, customers, etc.). Consequently, these systems often are vendor supplied Commercial Off-the-Shelf (COTS) products. Because an organisation’s core capabilities don’t change very often, neither to these systems – change is incremental and at a slow pace.
• Systems of Differentiation – While core capabilities don’t vary much from one organisation to another in the same industry, business processes do. For example, your bank and my bank may both offer loans, but the way those two banks process the loans is likely to differ. Applications in this layer represent the processes that make an organisation unique, and typically are not accommodated out of the box by vendor provided Systems of Record platforms. Business processes may not change every day, but they do tend to change faster than core capabilities, for example streamlining processes and/or incorporating new technologies.
• Systems of Innovation – This layer moves the fastest of all. This is the “sandpit” where new ideas and technologies are tested. Experimentation here will likely include ad hoc proof-of-concept (PoC) applications which are rapidly developed and then manually deployed and tested.

 Image based on presentation by Michael Guay (Gartner)

Let’s look at an example enterprise, for example a banking institution, and see how some of their applications might map to the pace-layered model.

Integration Within a Pace-Layered Architecture

So now that we understand the pace-layered model, how do we implement integration within it?  Let’s look at how a logical model of APIs/Services to see how they combine into applications across the layers:

Starting from the bottom layer, we see that each System of Record is typically a package which may contain several services/APIs. However, these APIs may not be directly consumable by the business due to misalignment with the logical data model, outdated protocols, or other reasons.  In these cases, it is a good idea to introduce a “sub-layer” of APIs that bridge the SoRs with other APIs within the organisation. These abstraction APIs (called Product Adapters) are tightly coupled with the underlying SORs yet expose functionality in a much more palatable format. They may also introduce stricter access control, validation and security than the SORs themselves. The APIs generally represent the core data entities (Customer, Product, Order, etc.), therefore they are granular and designed for reusability. Because they are so tightly linked with the core systems, they move at the same pace and therefore are considered part of the Systems of Record layer. Governance and change control will normally be very strict at this level because of the criticality of the data and the high dependency of services and processes that consume these APIs.

In the Systems of Differentiation layer, we see applications that are composed of the granular services/APIs sourced from the System of Record layer as well as possibly external APIs.  Here is where the business logic of the organisation lives, for example Loan Processing or User Provisioning. Functions that an application may perform in this layer include data aggregation, routing, filtering and often choreography/orchestration. Since they are specific to a process, they are less reusable than the underlying SOR APIs that they may consume. It is in this layer that the bulk of the integration within an organisation occurs. And because business processes can (and will) change over time, these applications will need to adapt as well, most certainly at a faster pace than the SOR applications. Governance should also be applied at this level, although perhaps not quite a rigidly as within the SOR layer; the organisation will want their business processes to be flexible enough to accommodate improvements in efficiency and expanding functionality.

The Systems of Innovation layer also features applications that consume both the SOR APIs and external APIs, as well as possibly business processes in the Systems of Differentiation layer. As the fastest moving layer, it will have even lighter governance in order to facilitate the experimentation of new apps and technologies. The functionality enabled in this layer is often peripheral to the core capabilities of the business, and therefore presents less risk to the organisation in case of failure. Also apps that are created quickly to prove a concept will rarely feature automated testing or a mature CI/CD pipeline, as they would be manually deployed and tested.

Finally, we use a Message Bus in order to facilitate both inter-layer and intra-layer communication. Asynchronous messaging patterns such as publish-subscribe keep systems loosely coupled and promote both extensibility and flexibility. The publisher doesn’t need to know anything about the subscriber, and you can always add or subtract subscribers without breaking existing integrations. A message bus is a critical element for lubricating the friction between applications that change at different paces.

This diagram provides a cross-section view of several attributes and how they vary across the layers:

How Does Microsoft Help with Pace-Layered Integration?

Microsoft offers a range of services and products, both on-premises and in the cloud, to help build powerful integration solutions that can cope with the different paces of enterprise application layers. Here we will discuss only some of these offerings and how they might fit into a pace-layered architecture (note there are many possible solutions; these suggestions are only one possible way to look at this):

System of Record Layer

The following products lend themselves very well to build the API layer on top of the SOR applications:

System of Differentiation Layer

These products are capable of implementing business logic, providing connectivity to internal and external applications and services, and supporting hybrid connectivity for applications that span both the cloud and on-prem:

System of Innovation Layer

Here we need the ability to expand the range of technology into the areas of artificial intelligence, predictive analytics, and business insight whilst enabling rapid (even ad hoc) development. There are numerous possibilities here, as it is less about which technology you use as opposed to how you use it. But these products all lend themselves well to innovative solutions:

Message Bus

If you are integrating mainly on-premises systems, BizTalk Server is at its heart a powerful messaging engine which can not only support the full gamut of messaging patterns, but also provide several out-of-the-box adapters for connectivity as well as a robust business process automation capability. This is why it features within so many layers. When integrating within the cloud however, Azure Service Bus provides a number of offerings for enterprise messaging, big-data streaming, event handling, and hybrid connectivity:

Tips and Best Practices

Here are some tips on how to maintain adaptive integration within a pace-layered enterprise architecture.

1. Consider how the applications you are integrating will be used.
   • Is the integration mission critical? Then Logic Apps would be a better bet than Microsoft Flow.
   • What are the security risks to consider? An API Management layer can provide policy-based governance, threat protections, and access control.
   • How quickly might the solution change? This will influence how much investment is made in automated testing, CI/CD pipelines, etc.
2. Make sure your System of Record layer is solid.
   • Are your APIs granular enough and well-defined? Remember that these will form the composable units for applications in the other layers.
   • Is security and data validation strongly enforced? Don’t rely on consumers for this; protect your critical data close to the source!
3. Limit customisation within each System of Record.
   • If you customise the SOR, what will happen with the next vendor upgrade?
   • Keep customisation with the System of Differentiation layer as much as possible, or at least in the API layer over each SOR.
4. Consider using canonical data models to avoid tight-coupling with vendor systems.
   • This typically requires a sound information architecture to define the business data entities.
   • An information architect can build a logical model independent of software implementation; invest in this.
5. Loosely couple inter-layer communications.
   • Vertical dependencies are difficult to maintain – unless you are the owner of the whole stack.
   • Choose a publish-subscribe messaging model wherever possible to maximise loose coupling and extensibility.
6. Allow room for innovation!
   • Employ the right level of governance at each layer. Avoid crippling change control policies where experimentation is both necessary and safe.
   • Empower business leaders to create their own solutions (e.g. using Microsoft Flow to automate mundane processes).
   • Encourage experimentation! Use Microsoft iPaaS features to achieve “integration at the speed of business” (Jim Harrer, Microsoft).