profile

Exceptional Frontend

How to prevent critical errors and mistakes (12 hours left)


I was put in a very sticky situation earlier this month

Let's just say I had to use dozens of cherry-picks to merge a huge pull request to production. Now, if you're even a half-decent developer, you know that this is a massive, HUMONGOUSLY BIG nono.

If you're not familiar, a git cherry-pick is a command you can do in your project's terminal that lets you pick up a commit from one branch and apply it to another. It's an alternative way to pass changes to another branch without directly merging all changes from one to the other.

The problem with cherrypicks though is that it messes strongly with git history. (It does this by making git treat the cherrypick as completely new, different code. For example, if you cherrypick a change from branch A to B, then try to merge A into B, git will not recognize that the code is the same between the branches). Basically it's a huge mess and a headache to do lots of cherry-picking.

And I had to do this for PRODUCTION

Depending on the nature of the changes, this could be pretty risky. This was going to take a surgeon's precision to do well. So, before I tell you how I tackled this... here's my question:

What steps and/or prevention measures would you take to make sure this didn't fail?

Let's say you successfully created the branch to merge into production with all the cherry-picks and you (think) that everything you had to cherry-pick was there. What would you test? What would you ignore? What would you prioritize?

(Which by the way, this is the basis of why I'm creating The Exceptional Questions Toolkit - A guide I'm preselling on how to be a better, more strategic and valuable developer. It's all on the art of asking important, critical questions. The presale ends in only 12 hours. So if you were planning on getting it at the lowest price possible, you only have today to get it)

Back to our horrible developing standards - here's the first thing I did to make sure this PR went as smoothly as possible:

  1. I became a pessimist for only a moment and asked myself this important question: "What's everything that I can imagine that could go wrong?"
    I made a small list in my head that looked a little like this:
    1. The site could go down because something went wrong with the build
    2. An error could prevent users from making purchases
    3. An error could prevent users from logging in
    4. A key feature that was supposed to be in the PR is missing because of incorrect merge-conflicts
    5. Unexpected errors in random places happen because of updates to certain key libraries installed in the project
    6. There's a key difference between the production environment and UAT that hasn't been noted because of prior cherry picks (I know this is bad) that now need to be included
    7. More unexpected and unimaginable errors pop up due to the complexity of the project
  2. I look at the list, (cry a little inside), then go into optimist mode to think up a list of prevention methods and solutions that could help mitigate the risk of any one of these happening. My list was:
    1. Make sure the final branch can locally build on my machine
    2. Make sure the final branch can build using docker
    3. Run the branch locally with npm start and test critical errors like the ability to login or make a purchase
    4. Run tests locally to see if everything passes as expected
    5. Do a basic smoke test around the site and see if there's anything missing (or elements that shouldn't be available). Also seeing if any errors occur
    6. Compare the list of features that the branch is supposed to introduce and check locally if they're there
  3. Execute on the list from step 2 and fix any errors or problems that might come up.
  4. Think that you're done but then... a voice creeps into your head. "What if something fails anyway? What will you do then?". Aghh, the pessimist evil voice in my head is right. It's not enough. So, I ask for help from others. Mainly...
  5. Ask the frontend team to take a look at the PR, let them know of the situation and let them test themselves in that branch in case they run into errors you don't. "It works on my machine" isn't a good excuse in cases like these where you know things could go wrong for any number of reasons.
  6. Ask the devops team to have your back, so that even if all the counter measures fail and something critical happens, it's easy to execute a roll-back. It's better for the site to be down for just 2 minutes than to have trouble getting a hold of somebody and it being down for 20 minutes.

Then, and only then, did I feel semi-comfortable merging to production

And thankfully, only 2 errors escaped through the cracks once the code finally hit production. But man, this isn't a situation I would wish on anybody 😅

But hey, if it ever does, I'm always thankful to have the tools necessary to deal with it. Tools you can get yourself by checking into the Exceptional Questions Toolkit! You only have 12 hours to get it at a massive discount before the price doubles.

113 Cherry St #92768, Seattle, WA 98104-2205

New to Exceptional Frontend?
🔺​Subscribe by taking the red pill here​ 🔺

Sick of these emails?
🔹 Unsubscribe by taking the blue pill here 🔹

Exceptional Frontend

A story-first newsletter for frontend devs who want to stand out, earn more, and engineer their ideal career

Share this page