So, you want to cut-over hey…
Releasing your hard-earned code, platforms and new customer experiences can be daunting and challenging but also a very rewarding experience. After all, you have spent your blood, sweat and tears to finally get to a point where your software can be used. Whether it be a daily deployment, a monthly release or a mammoth year-long go-live, each cut-over needs to be intricately planned, prepped and executable for the greatest chance of success.
This blog post is intended to provide release managers, release engineers and service delivery experts comfort that they have considered a range of criteria and are as much prepared for a cut-over as possible. So, without further ado… Here is your top 11 list!
Note: The purpose of this article is not to advise on the complexities of software deployments, rather, everything else around a go-to-market release that should be considered. For consideration of specific software deployment techniques, my esteemed colleagues have written many articles such as:
As a starting point, a visualisation or graphical depiction of the cut-over period is an excellent way to determine the sequence of events required to make a successful cut-over. It is also perhaps one of the most challenging steps. It allows various stakeholders to understand and agree upon the sequence of events (including deployments) and encourages collaboration across teams, which is especially important for a multi-vendor environment. The level of detail required should be relative to the complexity of a release, the example below is a multi-vendor, multi-platform release over a whole weekend.
Figure 1: Example Cut-over plan visualisation
The visualisation of a cut-over is an incredibly useful tool for communicating with team members and senior stakeholders and will provide them with confidence in the ability of your team to release successfully.
The most frequently used and well-understood tool for a successful cut-over is a comprehensive runsheet, described simply as a document that lists the action steps to take during a specific process, i.e. Cut-over.
Tips for creating an effective runsheet:
Figure 2: Example runsheet
The runsheet is a necessary best-practice asset. By curating and sharing a comprehensive runsheet, it enables you and multiple teams to understand and track the discreet steps required during a cut-over.
Each task must have an identified executor, backup team member and escalation point. Most importantly this must be understood and agreed upon by all parties prior to cut-over as it will typically be done after hours. Chances are someone will get sick or incur a snap lockdown. This step allows you to avoid single points of failure and is a key risk-mitigation tool.
If you haven’t noticed, we folks here in Core Business Engineering are incredibly passionate about automation. Any steps you can take to automate deployments, testing, in-built security functions, communications and anything else in the runsheet is time well spent. Benefits include:
Don’t leave this to the last minute, a solid rollback strategy must be in place, documented, tested, and should be included in the runsheet. In addition, the ‘point of no return’ must be identified, understood, and communicated. This is the point in time at which rollback is no longer possible.
A comprehensive rollback plan is an essential risk mitigation tool that allows for you to return the production system or systems back to their pre-cut-over state should cut-over be unsuccessful.
In your visualisation and runsheet, various points in time must be identified as a stocktake and/or decision point to ensure you have completed all essential prior tasks. For each of these checkpoints, there should be an agreed ‘definition of done’. This helps track if the execution of cut-over is being performed successfully.
These checkpoints also serve as a valuable point in time to provide communications on progress to senior stakeholders or any other interested parties. It is important to select and know your who, when, how and why of each of these communications items. This includes your agreed means of communication, be it SMS, email, or virtual meetings.
The most important step of any cut-over is, of course, testing that your newly released code works as expected! Three must-have considerations of cut-over testing are:
Testing is an essential step of any cut-over to make sure your production system is working as expected.
You must plan and execute a Dress Rehearsal prior to any major release. What this looks like will vary greatly depending on the type of cut-over, what is possible, the environments available and the data available. The information you gain from this process will be invaluable, whether it be to determine the actual time of loads and processes or whether your different platforms and respective teams can successfully communicate with each other.
Dress rehearsals can be categorised into both technical and operational rehearsals:
Note - Don’t forget to plan for and remove your data if you are doing Dress Rehearsal testing in production!
Prior to any cut-over, you need to ensure your entry and exit criteria are defined and agreed on. This is the entry point to a release and determines the success criteria to allow you to enter a cut-over. It should be actively tracked well before any cut-over, so you understand what the shortcomings are prior to any release and whether you should be entering into a cut-over. This is a considerably large topic area and one I will be covering in future blog posts.
The Command Centre/War room is where all the action takes place. This may be coordinated virtually, but it is advisable to get as many of the key players in a single area as possible to actively resolve issues. Co-location allows for an easier means of communication, faster resolutions, and less waiting time to make sure someone is online and available! Some key considerations are:
Cut-overs can be demanding and stressful, but also a fun and exciting time. Make sure you add the little things to make the whole process fun. Some classic ideas to pump up the adrenalin and make those midnight deployments less stressful are:
By creating a fun atmosphere, you allow for a greater level of team collaboration, energy, and a more memorable experience!
I promised you a Top 11, but I could go on forever. So, here are some honourable mentions:
Ensuring a successful cut-over is not an easy job. It requires cross-team collaboration, technical proficiency, and a willing and able team ready to take on and change their organisation. I hope this list has: