What I stopped measuring and why

The dashboard that grew until it was useless

I run AI automation for real work every day: content pipelines, local rendering, agents wired into MCP tools that touch actual files and actual APIs. For a long time I did what most operators do when a system gets complicated. I added a metric every time something went wrong. A job failed silently, so I added a counter for silent failures. A render queue backed up, so I added a gauge for queue depth. After a year of this, my dashboard had more than thirty panels and I was checking maybe five of them.

The other twenty-five were not wrong, exactly. They were just noise dressed up as signal. Each one had made sense the day I added it, in response to one specific incident. None of them told me, on any given morning, whether the system was actually doing its job. So I started cutting, and the cutting taught me more about how these systems fail than the metrics ever did.

Token counts told me nothing about output quality

The first thing I stopped tracking was raw token throughput. It is an easy number to get, every API response includes it, and it feels like it should matter because tokens are what you pay for. But token count answers a billing question, not a quality question. A script that runs long because the model is padding with filler uses the same tokens as one that runs long because it is doing careful, structured work.

I only noticed this because I compared two batches of generated content with nearly identical token totals. One batch needed almost no editing. The other needed a full rewrite. If I had kept staring at the token dashboard, both would have looked equally “productive.” What actually separated them was prompt structure and the source material feeding the model, neither of which shows up in a token count. I still watch token spend for cost control, because that is a real budget line, but I stopped treating it as a proxy for whether the output was any good.

GPU utilization is not the same as work getting done

Running rendering locally instead of in the cloud means you stare at hardware graphs a lot, because you own the hardware and you want to know it is earning its keep. I used to keep GPU utilization pinned in a corner of my second monitor, and for a while I treated a utilization dip as a problem to fix.

It took a stuck render queue to show me the flaw in that. The GPU was sitting at low utilization because it was waiting on a disk read from a drive that was doing something else at the same time. The GPU graph looked bad. The actual bottleneck was I/O, and no amount of tuning the render settings would have shown up on that chart. Utilization tells you the GPU is busy. It does not tell you the GPU is the thing you should be optimizing, and on a mixed local setup with rendering, storage, and background jobs sharing the same box, that distinction matters constantly. I moved on to watching queue depth and job completion time instead, because those numbers move when the actual work moves, regardless of which resource is the current bottleneck.

Agent call counts rewarded the wrong behavior

This one stung a bit more because I built the counter myself and was proud of it. When I started wiring agents into MCP tools to do multi-step work, I tracked the number of tool calls per task as a proxy for how much the agent was “doing.” More calls felt like more thoroughness.

What it actually rewarded was agents that took the scenic route. A task that a well-scoped agent could finish in four tool calls would sometimes take twelve, because the agent was re-checking things it already knew, or breaking a single logical step into three smaller ones out of caution. When I looked at outcomes instead of call counts, the four-call runs and the twelve-call runs succeeded at roughly the same rate. The call count was measuring verbosity, not competence. I dropped it and started looking at whether the task’s actual end state was correct: did the file get written, did the record update, did the downstream step pick up clean input. That is slower to check because it means looking at real output instead of a number in a log, but it is the only version of “did this work” that means anything.

What I watch now

Cutting metrics is only useful if what remains actually predicts problems before they become expensive. What I kept, after all the trimming, is a short list, and every item on it is something that has directly caught a real failure at least once:

Job completion rate over a rolling window, not a single run. One failure is noise. A rate that is drifting down over a week is a system telling you something changed.

Time between a job starting and a human noticing something is wrong. This is the metric that actually matters for silent failures, the kind where a process reports success but the output is truncated or empty. If that gap is measured in days instead of hours, the monitoring is not doing its job regardless of how many other panels exist.

Cost per completed unit of work, not cost per API call or cost per hour of compute. This forces me to connect spend to something I can defend, rather than watching a number that goes up whenever I run more jobs, which tells me nothing about whether those jobs were worth running.

Everything else I check by hand, occasionally, when I have a specific question. I do not need a permanent dashboard for a question I ask twice a year.

The uncomfortable part of dropping a metric

The hard part of cutting a metric is not deciding it is useless. It is admitting that watching it made me feel like I was managing the system when I was actually just watching a number move. A big dashboard is reassuring in a way that has nothing to do with whether it is telling you the truth. Every panel I removed took a small hit to that feeling of control before it gave anything back.

None of this means AI automation runs itself once you strip the dashboard down. It does not, and I am not going to pretend a leaner set of metrics replaces someone actually checking output. What changed is where I spend the checking time. Less of it goes to staring at graphs that move whenever the system is merely active, and more of it goes to sampling actual output and actual failures, which is the only place quality problems ever show up first.

If you are running your own local or cloud AI setup and your dashboard has grown past what you actually look at, that is usually a sign worth taking seriously. Not every number you can collect is a number worth keeping.

More on how I build and run this stuff is on the home page.

Get new guides and videos first — join the Telegram channel.