2 minutes to midnight - automation for the on-call DBA

You know the drill. You’re an on-call DBA and the phone howls in the night. It’s not 7PM or 7AM. It’s always when you’re in the darkest depths of your nightmares.

This session is inspired by a Rob Sewell sticker I have on my laptop: “Automate all the things”. I live by that oath. Not only because programming is more savage than pointy-clicky-administration. But also because automating things almost always means better testing.

Back to the inferno. You’re the on-call DBA for a monstrous SQL Server farm. You get a phone call 2 minutes to midnight. The entire SAN in your primary data center is fried. We need to failover to our DR site. And we need to prove that we didn’t lose more than five minutes of data.

What do you do? Do you grab the folder with your rotten runbooks from the shelf, peel the mold off of it and start working? Or do you have it all automated, so you can just run a pipeline, wait for it to finish and then read the automated report that the pipeline produces? What you and your team did every day when disasters did NOT happen is going to determine how much you’re going to scream, now that disaster DID happen.

In this session we will look at some tools and processes to make you more “2 minutes to midnight” ready. “Plan for the worst, hope for the best” is the motto. That way we can sleep well at night. And in the rare case that hell breaks loose, we should have planned for it to happen, automated the recovery process and regularly tested that the automated process works.

Our weapons for today’s carnage are (in order of importance):
– Pen and paper. Don’t trust the technology that just stabbed you in the back when you are about to fix things.
– Music. No matter how well prepared you are, you’re gonna wait for things to finish. Music makes waiting less torturous.
– Coffee – See “Music”.
– dbatools – PowerShell and SMO are probably enough, but why struggle?
– VSCode – Any editor really. But I like VSCode.