The runbook that runs my systems without me
The problem with knowledge that only lives in your head
For a long time my systems ran on memory. Not server memory, my memory. I knew which cron job restarted the render queue, which server held the video store, and which config file I'd hand edited at 2am and forgotten to commit. None of that was written down anywhere an agent, or a future version of me, could read it.
That works fine until it doesn't. The day I got paged for a modem that "went offline" and burned an hour before remembering the fix was a routing table entry at a specific priority, not a hardware failure, I understood the actual cost. The knowledge existed. It just wasn't retrievable by anything except my own recall in that exact moment.
A runbook fixes that by turning tribal knowledge into text that a process, human or AI, can act on without you narrating every step.
What a runbook actually is here
I don't mean a glossy ops manual. My runbook is a set of markdown files that describe what each system does, how to check if it's healthy, what "healthy" actually means for that specific system, and what to do when it isn't. Every entry answers three questions: what is this, how do I know it's broken, and what's the fix.
The files live next to the code they describe, in the same repo, so they get version controlled and reviewed the same way the code does. That matters more than it sounds. A runbook that lives in a wiki drifts from reality within a month because nobody updates a wiki when they fix a bug at midnight. A runbook that lives in the repo gets touched in the same commit as the fix, or it doesn't get touched at all and you notice because the next incident references a step that no longer applies.
Where AI agents actually fit
This is the part that gets overhyped elsewhere, so let me be specific about what I actually do. I run agents with tool access, mainly through MCP, against my own infrastructure. They can read logs, query a database, restart a service, or open a pull request. What makes that safe enough to leave running unattended is not the model being smart. It's the runbook constraining what "handling an incident" means before the agent ever sees the alert.
An agent that gets paged with "server X modem offline" and has to figure out from scratch whether that's a real outage or a known false positive from an IP detector will guess. Sometimes it'll guess right. I don't want sometimes. So the runbook entry for that alert says explicitly: check the customer-facing path, not the internal status column, because the internal detector has a known false-offline pattern. The agent isn't reasoning its way to that insight. It's reading it and following it, the same way I would if I hadn't slept in two days and needed the shortcut spelled out.
That's the actual value of putting a runbook in front of an agent. It's not that the agent gets smarter. It's that the agent stops needing to be smart about things I've already solved.
The parts I don't hand off
Some entries in my runbook are explicitly marked do not automate. Anything that involves rotating a credential, force-pushing to a shared branch, wiping a device, or touching customer billing gets a line that says an agent stops and asks a human before acting. Not because the tooling can't do it. Because the blast radius of getting it wrong is large enough that the few seconds of asking me first is cheap insurance.
This isn't AI caution theater. It's the same judgment I'd apply to a junior engineer on their first week: full trust on read operations and routine restarts, a pause before anything destructive or customer visible. The runbook is where I write that judgment down so it applies consistently instead of depending on whether I happened to think about it that day.
What actually broke before I had this
The recurring failure mode, before any of this existed, was rediscovery. I'd solve a problem, move on, and six weeks later hit the identical symptom and spend another hour rediscovering the same fix because nothing captured it. Multiply that across a dozen systems and the tax adds up fast, and it's a tax that compounds because the more systems you run, the less of your own history you can hold in working memory at once.
Writing the runbook entry the first time a problem happens is the whole trick. It costs maybe ten extra minutes over just fixing it and moving on. The payoff isn't the second time you hit the same bug, it's the tenth time, and it's every time an agent hits it instead of you.
What this doesn't do
I want to be direct about the limits, because it's easy to oversell this kind of setup. A runbook doesn't make a system self healing in any deep sense. It makes known failure modes cheap to handle, by a human or an agent, because the diagnosis and fix are already written down. Novel failures still need a person to think, investigate, and eventually write the new entry. Nothing here replaces the judgment of someone who actually understands why a system was built the way it was.
It also doesn't remove the need for monitoring or for someone to own the outcome. An agent following a runbook can restart a service or flag an anomaly, but a human still decides what "acceptable risk" means for a given system, and still reviews what changed. I check what the agents did. Not because I distrust the specific action taken, but because the runbook is only as good as the last time someone updated it, and the fastest way to find a stale entry is to watch what happens when an agent follows it.
Making it stick
The practical habit that keeps this alive is small: every incident ends with a one line question, does the runbook need updating. Most of the time the answer is no, the existing entry covered it. When the answer is yes, the update happens in the same sitting as the fix, while the details are still fresh, not as a follow up task that quietly never gets done.
The other habit is treating the runbook as something agents read, not just humans. If an entry is written vaguely enough that only I can fill in the gaps from context I'm carrying in my head, it's not actually done. The test I use is whether an agent with tool access but no memory of past incidents could follow it correctly. If not, it gets rewritten until it can.
None of this is exotic. It's closer to writing good documentation than it is to anything that needs the word "agentic" in front of it. What changed for me isn't the concept, it's that I finally have something, agents with real tool access, that reads documentation as a first class consumer instead of a human skimming it once and forgetting. That's what made the ten minutes worth spending.
If you're curious how the rest of this setup fits together, from the agents to the infrastructure they operate on, you can find more on the [home page](/).
Get new guides and videos first — join the Telegram channel.