I put an AI on log watching duty
# I put an AI on log watching duty
A few months ago one of my servers fell over in the middle of the night. It was the quiet kind of failure, the kind that leaves a clear warning in the logs hours before anything actually breaks. The next morning I went digging and found that warning sitting there in plain text, written long before the crash, buried under tens of thousands of perfectly normal lines that had scrolled past it in both directions.
That stung more than the outage did.
The problem with logs was never a shortage of information. A handful of small servers can produce tens of thousands of lines an hour, and almost all of it is routine. A request came in. A request finished. A backup ran, a health check passed, and the same ten messages repeated until the file rotated. Somewhere inside that murmur, occasionally, one line says something you badly need to hear, and the odds of a human spotting it in real time are close to zero.
So I built a small AI layer whose only job is to watch that firehose and speak up when something deserves a person's eyes. This post covers how it is put together, what it caught, and the places where I still refuse to trust it.
Why alert rules were never enough
The traditional answer to log volume is alerting. You write a rule that says, if this exact phrase ever appears, wake me up. I have those rules and I keep them. They fail in a specific way, though: a rule can only catch trouble somebody already imagined. The worst nights I have had all came from problems I had never seen before, which by definition had no rule waiting for them.
Rules have a second blind spot that took me longer to name. They judge one line at a time. A single slow request means nothing. A hundred slow requests creeping upward over an hour is a server about to fall over, and no individual line in that hour is scary enough to trip a threshold. The trouble lives in the pattern, and pattern is invisible to a rule that reads each line in isolation.
That gap is what I wanted to fill.
The design I refused to build
The first idea everyone reaches for is the obvious one: pipe every log line into a big language model and let it read everything. I priced that out and dropped it fast. At tens of thousands of lines an hour it is slow, it is expensive, and most of the spend goes toward having a clever model read the words backup completed successfully several thousand times a day.
The model's attention is the costly resource in this system. You do not spend a costly resource on work a free rule finishes in a microsecond. So the whole build rests on one principle: cheap, dumb, reliable machinery handles the easy ninety nine percent, and the model only sees the small slice that is genuinely hard to judge. That single decision is the difference between a layer that costs me a few dollars a month and one that would cost more than paying a person to do the job.
A bouncer at the door
The first layer contains no AI at all. Before anything reaches the model, plain filtering rules throw away everything that is obviously fine. Routine success lines are dropped. Health check chatter is dropped. At the same time, a short list of hard rules catches the things I already know are always serious. A database connection dropping does not need a model's opinion; it fires an alert directly, the dumb way, instantly.
Think of this pass as a bouncer. The vast majority of lines never get inside the building.
What survives is the ambiguous middle: lines that are neither obviously fine nor obviously a known disaster. Unfamiliar errors, odd frequencies, messages that might be nothing. That residue is small, and it is exactly the material a model is worth spending on.
What the model does with the leftovers
The suspicious lines go to the model in batches, grouped by time, along with one plain question: is anything here worth a human's attention, and if so, what and why.
This is the part a rule cannot do. A rule matches patterns someone wrote in advance. The model can look at a line it has never seen and form a reasonable opinion about whether it smells like trouble. It notices when an error that normally appears once a week shows up a dozen times in an hour. It connects a few scattered lines into one story. Because it reads a grouped batch rather than a single line, it can see the shape of the hour, and it will point out that three unrelated services started complaining at the same minute, which usually means something shared underneath them is sick.
A good human on call reads logs exactly this way, in clusters and drifts and sudden changes. Doing it at three in the morning is the part humans are bad at, and that is the part I handed over.
The briefing matters more than the model
A log line on its own is a cryptic fragment. So along with each batch I send a short briefing: what this server does, what normal looks like for it, and which failures have actually hurt me in the past. The difference in judgment before and after that briefing was dramatic. Without it, the model flagged things that are weird everywhere but harmless here. With it, it got much quicker to worry about the patterns that have bitten me before.
Writing that briefing took more iterations than any other part of the build. Same model, same filter, completely different usefulness depending on how well the context was written. It is the gap between a new hire reading your logs cold and someone who has run the system for a year.
Quiet by design
The fastest way to ruin an alerting system is to make it chatty. An alert you have learned to ignore is worse than no alert, because it leaves you feeling covered while nothing is covering you.
So the model does not message me about everything it finds mildly interesting. It reaches out only when it judges that a person is needed, and the message is short: what it noticed, why it thinks that matters, and the actual lines underneath. Something I can read on my phone in five seconds. Most days it sends nothing at all, and the silence is the point. A monitor that stays quiet ninety nine days out of a hundred, and is right on the hundredth, is the one you will still be listening to.
The save that paid for it
One evening it pinged me about a service logging slow response warnings slightly more often than usual, a gentle climb over a couple of hours. No hard rule had fired, because no single line was alarming. The model saw the trend across the batch and suggested a look. A disk was slowly filling and quietly dragging the whole service down with it. I cleared it in ten minutes, on my own schedule, before dinner.
The boring alternate ending writes itself: the disk fills at three in the morning, the service falls over, and I clean up a much worse mess while half asleep. The model fixed nothing. It noticed early, and the noticing was the entire value.
Where I still refuse to trust it
The filter in front of the model can wrongly discard a line that mattered, and then the model never sees it. The model itself can be confidently wrong in both directions, reading an innocent burst of activity as a crisis or waving past something real. That is why every alert includes the raw lines it was worried about: the final call stays with me, made with the actual evidence on the screen.
Underneath all of it, my hard deterministic alerts keep doing their dumb reliable job. Database down, disk completely full, process dead. The AI sits on top of that floor and catches the subtler things the rules miss. It never replaces the floor. Anyone who lets a clever model become their only line of defense is going to have a very bad night eventually.
Running cost stays small because the model only ever sees a filtered sliver, easily a few dollars a month at my scale. It is still a paid service, and I would rather say that plainly than round it down to free.
Should you build one
If you run one small server that barely logs, no. Read the logs yourself and spend the afternoon on something else.
If you are drowning in volume, if the line you need keeps vanishing into noise, and if the obvious hard alerts already exist and you still feel blind to the weird stuff, then a thin triage layer like this is worth the afternoon it takes. Build the boring filter first, because that is where most of the work happens. Then put a model on the ambiguous middle and brief it the way you would brief a person taking over your on call rotation, because that is the job you are handing it.
If you want more builds like this with the failure modes left in, [there is more on xavierfok.com](/).
Get new guides and videos first — join the Telegram channel.