Which alerts I muted and what it cost

The first alert I muted

It was a disk space warning on a machine that renders video overnight. The threshold fired at 85% full, and on a server that's constantly writing and cleaning up temp files, 85% is just Tuesday. I muted it inside a week. Not because the alert was wrong, but because it was right too often to be useful.

That's the honest starting point for talking about alert fatigue: almost every mute decision looks correct in isolation. Nobody wakes up and decides to ignore something important. You mute the thing that's noisy, and noisy usually means it fires a lot, and it fires a lot because the threshold is bad or the check is too broad. The problem is that "too broad" and "actually catching something" aren't always different conditions. Sometimes they're the same alert on different days.

Why muting felt safe at the time

Running AI agents and automated pipelines in production means you accumulate monitoring fast. Every script that runs unattended eventually gets a heartbeat check, because the failure mode of unattended automation isn't a crash you see, it's a silent stop nobody notices for three days. So you wire up watchdogs: did the job run, did it finish, did it write output, is the queue moving. Each one seems cheap to add and cheap to justify.

What's not cheap is living with all of them at once. A pipeline with five stages easily produces fifteen to twenty possible alert conditions once you count timeouts, empty outputs, and stale-state checks. Multiply that across several pipelines and a handful of agent-driven tasks that call out to APIs, local models, and file systems, and you get a channel that pings constantly. At that volume, the response isn't vigilance, it's habituation. You stop reading the message and start reading the icon. Eventually you stop reading the icon too.

Muting felt safe because each individual alert I turned off had a clear, defensible reason. The disk warning fired on normal cleanup cycles. A retry-count alert fired every time an external API had a slow afternoon, which was often, and self-resolved within a retry or two. A "queue empty" alert fired every night between batches, which is a completely normal state, not a failure. Muting any one of these looked like good hygiene. It was only later that the pattern mattered more than any single decision.

What actually broke

The real cost showed up when a genuine failure looked exactly like the noise I'd trained myself to dismiss. A render queue sat empty for a full day, and I didn't catch it because "queue empty" was muted for the ordinary reason: it fires nightly between batches and always had. This time it wasn't between batches. A step upstream had stalled and stopped feeding the queue, and the empty-queue state that used to mean "waiting for the next batch" now meant "nothing is coming." Same symptom, opposite cause, and I'd already decided the symptom wasn't worth watching.

The fix wasn't to unmute everything. That just recreates the original noise problem. The fix was recognizing that the alert itself was badly specified. "Queue is empty" is not a useful condition on its own. "Queue is empty and has been empty for longer than one full pipeline cycle" is. The first version fires constantly and tells you nothing. The second version fires rarely and tells you something is actually wrong. I hadn't done that work when I muted it, I'd just turned off a light instead of fixing the wiring.

A similar thing happened with a heartbeat check on an unattended job. I'd made the mistake of treating heartbeat paging as something to enable case by case rather than as a default, and it stayed off by default across most of the automation I run. That meant a stalled job could sit stalled for a long time before anything human noticed it, because the notification path that would have caught it was never turned on in the first place. That's a slightly different failure than muting, but it comes from the same root: treating alerting as opt-in extra work rather than as a baseline you build in from the start and then tune down, not tune up.

The pattern behind the mutes

Looking back at what I muted, almost none of it was random. It clustered into three categories.

The first was alerts on conditions that are normal most of the time and abnormal rarely, with no way for the alert itself to tell the difference. Disk usage on a cleanup cycle. Queue-empty between batches. Retry counts against a flaky but non-critical API. These need a duration or frequency component, not a raw threshold, or they will always be noisy.

The second was alerts that duplicated a stronger signal somewhere else. If a job posts a completion log and a separate monitor also checks for that job's output file, you effectively have two alerts for one condition, and when one gets noisy you might mute it while trusting the other is still watching, only to find out later they weren't actually redundant, they were checking slightly different things.

The third, and the one I'm least proud of, was alerts I muted because I didn't trust my own threshold. I picked a number without much evidence, it fired constantly, and instead of going back to figure out the right number I just turned it off. That's not tuning, that's giving up on the check entirely while telling myself I'd revisit it later. I usually didn't.

What changed in how I set these up

I stopped treating "mute" as a resolution. Now if something fires too often, the response is to fix the condition, not silence the channel. That usually means adding a time or count threshold instead of a raw state check, so the alert reflects "this has been true for a while" rather than "this is true right now," which is what actually distinguishes a real failure from a normal fluctuation.

I also stopped leaving paging off by default on unattended jobs. The default now is on, with muting or downgrading done deliberately per job once I've watched it run long enough to know its normal noise pattern. That's slower to set up than flipping everything on and forgetting about it, but it means silence is a choice I made on purpose, not a gap I never noticed.

And I keep a short list of exactly what's muted and why, reviewed occasionally rather than never. That sounds like a small thing, but the actual failure mode of alert fatigue isn't any single mute decision, it's that those decisions accumulate quietly and nobody, including the person who made them, remembers the full list six months later.

None of this makes the monitoring smarter than it is. An automated check still only knows what you told it to check for, and no amount of tuning turns a threshold alert into judgment. The goal isn't a system that catches everything, because that system doesn't exist and claiming otherwise would just be a different flavor of the same problem: alerts you can't actually trust, dressed up as ones you can. The goal is fewer, better-specified alerts, and being honest with yourself about which silences you chose on purpose.

If you're building or running automation of your own and want to see how this kind of pipeline work actually looks in practice, you can find more on the [home page](/).

Get new guides and videos first — join the Telegram channel.