Adding new technology to your stack.

January 6th, 2020

Chances are, you already have your eye on some Shiny New Tech. Something cool you saw mentioned on Twitter, or a conference talk maybe. The initial feeling of "that solves all the problems!".

What problem does it solve?

But before you dig in and invest time in actually using this amazing new technology, create a short list of the problems you assume it solves. This way you can confirm if it does - and also check alternatives. With all the available solutions out there, what are the odds that you need that alpha-version from Unstable Inc.?

If your shiny new tech does seem to fit the bill, it probably requires some integration code. Choose a library. Usually there are multiple options, so when comparing keep an eye out for:

• supported versions of your chosen solution
• how actively maintained it is (recent activity, open issues, number of downloads)
• how much of it you actually want or need to use
• how many dependencies the package has and the stability of those
• if the library is intended for your specific case? (for example: not all code is suitable for use in a long-running process)

Write code, delete code.

Next, create a POC. Just get it to work, no matter what. It's probably best if you don't think about good design or abstractions yet. You might discover some that your chosen library doesn't really have a nice API, or that the documentation is a Lie! That's all fine. You're just tinkering, and swapping at this point is still cheap.

Once you have it running, consider tossing your proof of concept out. Just delete it. Because now that you know how to make tech X work, you can implement it without all the original n00b questions, plus you'll have gained some experience with the library at hand.

The Mikado Method is a nice way of exploring the changes required for you to start using your new thing without committing to your first attempt.

End the honeymoon.

After creating a better implementation - one that you are happy putting in production - end the honeymoon. Try to break your new thing. Starve the installation of memory, overload the process, do whatever you can. Change settings, read articles about scaling, make sure google "Shiny New Tech memory problems" and "bad performance with Unstable Inc".

If you can break it (and you should be able to), investigate how it breaks and how it informs you when it does. Check what you can improve in the logging, metrics and alerting.

Maybe involve colleagues and do a pre-mortem or a game of Challenge cards in order to find weaknesses in the Shiny New Tech solution.

Get comfortable with any admin panels. Know what is where, and what things mean. There is nothing worse than looking at a graph under the pressure of downtime and losing 30 minutes googling if "2.7 floogles per plork" is normal or way too high. Or not realising you can set the graph to show the past 4 hours instead of the default 1 minute.

Document the usage, management urls etc. In the documentation, give pointers on where to look when things go sour.

Have a show-and-tell, since you did all the research your colleagues might not have all the information you have. Try to get them as comfortable as possible with the idea of maintaining this new thing in production. Remember, it's not just your baby anymore!

The road to production.

When finally sending Shiny New Tech to to the front lines, always take into account that provisioning may fail, deployments might not run as you expected. Think about a way back if things go badly - or rather think about a way forward that is not a huge destructive change but an incremental step to a new situation.

Don't think about the deployment as an instant thing, but as a process with a timeline. Figure out if there are dependencies (steps that need to happen in a certain order). See which parts you can prepare and which parts can only really be done once you're doing the dance.

If there are large actions, like data migrations, try to make an educated guess (based on experiments) for the time it will take.

Write a playbook for the change. It should really be as simple as possible, which means any one-time commands that can be prepared should be copy-pastable. Any commands that can verify the success/failure of individual steps should already be included in the document. And even better, once you have these commands, you can combine them into scripts to automate multiple steps into a single action.

Let team members read your playbook and comment on parts they don't understand or don't agree with. Make sure everyone is on board with the way this release will be done.

Now, finally: Go to production. With confidence, because you rock

Ramon de la Fuente

Pointy haired boss