Our industry is determined to deliver value ever more rapidly, safely, and securely across software development lifecycle environments. One way to achieve this is through GitOps. In broad terms, GitOps is about applying the Git feature workflow to operations. I took this idea and narrowed it down to exactly one use case, using Git to drive deployments on OpenShift. This is a walk-through of the concepts and components we used to enable GitOps driven deployments all the way from development to production for our enterprise client Education Payroll Limited (EPL) by deploying a payroll solution known as EdPay.
Another way to define GitOps is by applying a developer experience (DX) to operations. Developer experiences for Application Programming Interfaces
GitOps follows this flow across four major actors to drive deployments on OpenShift:
Underlying the pipelines and environments is OpenShift. Conceivably, you could also run Git on OpenShift but that was not the case at this client.
The users of this system are the people either proposing or merging a change to an environment (env). For example, a developer has completed a feature and produced version 1.2.3 of an API service in a development environment. Once that feature is ready for testing, the developer proposes version 1.2.3 of that API service to a test env via a pull request. The test env owner reviews the pull request, approves it, and merges it. The merge fires a webhook that kicks off the pipeline run that deploys version 1.2.3 of that API service to the test env.
Naturally, Git provides source control but it also provides excellent access control and audit capabilities. In Git we have two types of repositories (repos).
Environment repos contain the configuration and versions for the services in a particular env (e.g. prod). The env repos also contain all of the settings for the repo itself to enable GitOps. This is achieved by setting repository and branch permissions, pull request rules and reviewers, and webhooks.
Service repos container the source code of the services that compose the application(s) running in the env. Depending on how your OpenShift Container Platform creates container images, the source code may be laid out in a certain structure (e.g. Source-to-Image). It can also contain deployment artefacts particular to the service such as a DeploymentConfig, ConfigMap(s), and Jenkinsfile. For more info see OpenShift 3 Demystified.
Pipelines are a sequence of steps executed in Jenkins, running on OpenShift. As input, pipelines consume environment configuration and source code from Git. As output, pipelines produce container images and deployments of the EdPay application for an environment. For day to day development, the pipelines can be considered as your interface to the OpenShift Container Platform.
There are two types of pipelines and one type of build:
An environment (an OpenShift project) is a combination of running services and configuration, primarily, specific versions of the running services and the DB to which they are configured to connect. Each environment contains its own instances of all of the services that make up the EdPay application.
There are two types of environments:
The primary differentiator is that dev envs do the builds that create the container images for all of the services in the stack. Dev master and dev stable are where all application builds take place. All envs do deployments of these container images.
It’s worth noting that the environment itself is versioned. Ultimately, the environment is just another application dependency.
The deck below is a walk-through of the concepts and components used to enable the GitOps driven deployments described above.
Because there are a fair number of concepts and components at play, I also did a screencast to explain the system.
Note: Always set limits on compute resources. Always! We learned this one the hard way when we deployed Jaeger Tracing on OpenShift and it began to cause node instability. My solution was to add max traces and compute resource parameters to the Jaeger Tracing OpenShift all-in-one template.
The ideas and concepts for a system like this have been rattling around in my head for a long time. I had assistance from a few people around me at the client EPL that worked hard bring this to life. Without their help and support this wouldn’t have been possible and my thanks go out to them.