· 5 min read
What I learned from reading my own automation logs for a week
A week of actually reading automation logs, instead of glancing at dashboards, turned up quiet failures and habits worth changing in a local AI automation setup.
Why I started reading logs instead of just watching dashboards
I run a handful of automations that touch content, rendering, and a few agents wired into local tools through MCP. Most days I check a dashboard, see the numbers are roughly where they should be, and move on. That’s fine for catching outages. It’s terrible for catching the stuff that quietly works “well enough” while doing something dumb underneath.
So for a week I stopped trusting the summary view and just read the raw logs. Not all of them, and not line by line for eight hours a day. I picked an hour each evening, opened the log files for whatever ran that day, and scrolled. No dashboard, no aggregation, just the timestamps and the text.
It was slower and less satisfying than watching a green checkmark. It also told me things the dashboard never would have.
The first thing I noticed: retries hiding as normal runs
The dashboard reports a job as “succeeded” the moment it finishes, regardless of how many times it had to try. Reading the logs, I saw jobs that failed on the first attempt, retried automatically, failed again, and only succeeded on a third pass. From the outside, that’s a green checkmark like any other. From inside the log, it’s three separate runs, three separate error traces, and a chunk of wasted compute time that never shows up anywhere I’d normally look.
None of these retries were catastrophic on their own. But they clustered around specific times of day, which told me something the success rate never would: a resource contention issue that only showed up under load, not a random flake. I wouldn’t have found the pattern from a pass/fail count. I found it because I could see the same error message repeating at the same hour, three days running.
Agents that looked productive but weren’t
I have a couple of agent workflows that call out to local tools through MCP to do repetitive work: checking file state, pulling status from a service, deciding on a next step. Watched from a distance, these look busy. Tool calls go out, responses come back, the agent moves to the next step.
Reading the actual tool call and response pairs, I found a few cases where the agent was calling a tool, getting back an empty or malformed response, and then proceeding anyway as if it had gotten a real answer. The agent wasn’t wrong to keep going, technically, because nothing told it to stop. But it was making a decision based on nothing, and the log was the only place that was visible. The dashboard just recorded that the agent “completed” its task.
This is the sort of thing that’s easy to miss because the agent doesn’t crash. It doesn’t throw an error. It just quietly does something less useful than intended, and everything downstream treats the output as fine. Fixing it wasn’t dramatic: I added a check so a malformed tool response gets logged as a distinct failure instead of getting silently absorbed. But I only knew to add that check because I’d actually read the exchange, not because anything alerted me to it.
Local vs cloud showed up in the timestamps, not the summaries
I split work between models running locally and calls out to cloud providers, depending on the task. The dashboard tracks which jobs used which, but it doesn’t tell you much about the texture of the difference. The logs do.
Local runs had a very consistent rhythm: predictable start-to-finish times, no network-related errors, and failures that were almost always about the input, not the infrastructure. Cloud calls had a wider spread of timing, and a small but real number of failures that were about the connection or the response itself rather than anything I’d fed in, things like truncated responses or a request that just never came back in a reasonable window.
Neither is better across the board. Local means I own the failure modes and they’re usually about what I gave the model. Cloud means I inherit someone else’s failure modes in exchange for capability I don’t run myself. Reading a week of logs made that trade concrete instead of theoretical. It’s the difference between knowing a fact and having watched it happen a dozen times in a row.
MCP tool calls fail quietly if you don’t ask
The MCP-connected tools were the biggest source of surprises. A couple of my tool integrations return a “success” status even when the underlying operation didn’t do what I expected, because the tool itself doesn’t have a way to express “technically I ran, but the result is empty” as distinct from “I ran and here’s your data.” From the log’s point of view, both look like a clean return.
I only caught this because I cross-checked a few tool responses against what I knew should have been there. That’s not something I can do for every call, every day. It’s a reminder that “the tool didn’t error” and “the tool did the right thing” are two different claims, and logging usually only tells you the first one unless you’ve specifically built in a way to check the second.
What changed after a week of reading
I didn’t rebuild anything major. Most of what came out of this was small, unglamorous fixes: distinguishing a genuine failure from a masked one, adding a log line at a decision point that previously logged nothing, tightening what counts as “success” for a couple of tool calls so an empty result doesn’t get treated the same as a real one.
The bigger shift was in what I trust. A dashboard tells you the shape of things at a glance, and that’s genuinely useful for noticing when something is clearly broken. But it can’t tell you when something is working in a way you wouldn’t have chosen if you’d watched it happen. For that, there’s no substitute for reading the actual output, at least often enough to know what “normal” looks like at the level of individual runs, not just aggregate rates.
I also noticed my own habits. I check dashboards constantly and logs rarely, which is backwards for catching the quiet failures. Dashboards are good for “is it up.” Logs are good for “is it doing what I think it’s doing.” Those are different questions and I’d been asking only one of them.
A few habits I’m keeping
I’m not going to read every log line every day going forward, that’s not sustainable. But I’m keeping a smaller version of this habit: a short, regular pass through raw logs for whatever ran recently, not just the summary. I’m also adding more explicit logging at the points where an agent or a tool call makes a decision based on a response, so the log can tell me directly when that response looked thin, instead of me having to notice it by eye.
None of this makes the automations smarter on its own. It just means the failures that used to hide inside a green checkmark now have somewhere to show up. That’s a small change, but a week of actually reading the logs was enough to convince me it’s worth keeping around.
If you want more of this kind of practical, no-hype look at running AI automation day to day, you can find more on the site home page.
Get new guides and videos first — join the Telegram channel.