Picture this - it's 3:55 AM on a brisky Friday morning. You've just arrived at the office to do a release. It's a fairly routine process and you've done many of these on this project - so you probably didn't prepare as much as you should have and felt comfortable enough to "wing it". You try to login to your PC and it's just black - no errors, no sounds - just a white cursor and a black screen. Oh $#!^

After 30 minutes of messing around (and turning it off and on again!), you’re no closer to seeing anything on your desktop. Nobody is awake because you didn’t organise a backup person for a release that was meant to be completed 20 minutes ago.

Out of pure desperation you eventually resort to downloading a VPN client on your phone, log into Jenkins and trigger a release - hoping that nothing goes wrong with your scripts…

I wish this was a story that ChatGPT wrote about a release gone wrong, but sadly this is exactly what happened to me a few years back.

I am sure that if I asked around, many would have their own release challenges that they have had to conquer! Although I probably couldn't have avoided my issue - I definitely could have been more prepared for it.

Enter release & rollback plans! Simply put, a release plan is a document that outlines the processes and timelines for a product release. Not to be confused with a Product Roadmap, a release plan looks specifically at how we ship specific set(s) of code to Production, as opposed to looking at how we iteratively design, implement and deploy a set of features over a prolonged period of time. Generally, good releases start with good communication! As a result of this, it's important that each team appoints a Release Manager for a release. This person’s role is to organise and communicate the relevant information before, during and after a release.
 
 

But what does a Release Manager actually do?

Before the Release

"Failing to plan is planning to fail" - Benjamin Franklin (most likely).

In his book "The Power of Habit" Charles Duhigg talks about how Pre-Flight checklists came to exist for the aviation industry. TL;DR - even though people had good intentions and a wealth of experience, they were still prone to making mistakes and forgetting things. This was often exacerbated by the fact that when humans are in “high stakes” scenarios, the chances of them making these mistakes are even higher.

The release plan aims to solve exactly this problem. By having a step-by-step guide of the tasks and executors of these tasks, we can reduce the chance for human error. The process of completing this guide is most of the battle - by going through and planning what needs to happen and talking through who is going to execute these tasks, we ensure that everyone is on the same page for the release. The plan or "checklist" is just a way of holding ourselves accountable to what we had discussed leading up to the release.

"But Jesse, even if you had a release plan, if you can't see anything on your desktop then a release plan is just as useful as not having one" I hear you say.

To justify my memes, we need to look at the roles and responsibilities during the release.
 

During the Release

A good release plan would not rely solely on the “Release Manager” to do all the work. Rather, the Release Manager should coordinate the release, rather than wear the hat of “releaser” too. Imagine you are the pilot of a plane that is experiencing engine issues. Do you think you also have the time to walk through and calm all the passengers, and ask them if they’d prefer chicken or beef? Nope, you need to be wholly focused on ensuring that the plane lands safely.

Similarly, the Release Manager should not be the person actually executing the release, but rather a coordinator ensuring that everyone is executing on their defined tasks. You should think of them as the Chief Flight Attendant, and the various releasers as the pilot(s). During the release there is a lot going on - various teams need to align on ensuring their code is deployed, someone needs to get QA to start validating the release, stakeholders are generally testing and asking a few questions, and teams that have dependencies on your release will also be asking when they can begin their release & testing.

Throwing these responsibilities onto the same person who is also trying to execute the release, troubleshoot deployment issues etc. is a lot to ask of one person - especially if we are expecting them to not drop any of the balls they are juggling!

This thinking is exactly what would have solved my problems that fateful morning. Instead of me trying to do everything, we could have planned to have a backup person available for the release. Once I couldn't login to my PC, we could have gotten the secondary releaser to assist us. Worst case scenario, the Release Manager and I could have potentially swapped roles - and I could have assumed that role while someone else ensured there was continuity in our release.
 

After the Release

If things didn't go smoothly, then it is time to get out your Rollback Plans.

These should be the plans that are always created but hardly used. We should aim to exhaust our options before we execute on these. You wouldn't want the pilot going all the way back to the departure airport if it was safe to land at your destination!

These are arguably more important than release plans. Generally, it's fairly simple to revert our deployed code to a previous version. The more complex task is reverting the state of your system. This means having database scripts that are created to undo the changes you have applied, as well as any data that could be have been inserted before the rollback was executed. You may also need to contact the other teams and alert them to the fact that you are rolling back - like letting the other airplanes know that you are turning around and they need to get out of the way!

Remember to test these scripts as well - far too often people create rollback plans and scripts and find out the hard way that these do not work as expected.

If things went smoothly, then it's time to close out the release and get a much-deserved break. Usually, this includes sending some communication through to your stakeholders & team to let them know everything is completed. If you are using a system to track the releases, then you will also need to close out the respective tickets with the right status. All that's left is to send a message to your peers to brag about the new features that are now live.

To all those with upcoming releases - may the odds be ever in your favour.
 
 
Yours faithfully, Jesse “The Red Sage” Leresche