← all posts

How I Keep A Year Of Automation Logs Searchable

The problem nobody warns you about

When I started wiring AI agents into real workflows, content generation, local rendering jobs, MCP tool calls, I didn't think about logging at all. I just wanted the automation to run. Six months in, I had a problem that had nothing to do with the AI itself: I couldn't figure out why a job that worked in March broke in June. Somewhere in between, I'd changed a prompt, swapped a model, or updated a tool definition, and I had no record of which one, when, or what the output looked like before and after.

That's the part people don't mention when they talk about running agents in production. The automation isn't the hard part. Knowing what your automation actually did six months ago is the hard part. This is how I built a system to answer that, without paying for a hosted logging platform.

What actually goes into a log entry

Every run, whether it's a content agent drafting a script, a render job processing a video, or an MCP tool being called by an agent, writes one JSON line per event. Not a paragraph of prose, a structured object. Each line carries:

• a timestamp
• a run ID so I can group every event from one job together
• the name of the agent or tool that produced it
• a short summary of the input (not always the full input, since some of those are large)
• a short summary of the output
• duration in seconds
• exit status (success, failure, or a specific error code)
• token counts and rough cost, when the step went through an API call

The key decision was making these fields structured rather than free text. Free text log lines are fine to read once. They're miserable to search across a year of history, because "the render failed" and "render job errored out" mean the same thing to a human and nothing to a keyword search.

The folder structure I settled on

Logs live on disk, organized by date: a folder per year, a folder per month, a folder per day, and inside that, one JSONL file per run ID. Nothing clever. This matters more than it sounds like it should, because it means I can always find last Tuesday's jobs by walking a path, without querying anything.

After 24 hours, a cron job gzips anything older than a day. Raw JSONL compresses well since the field names repeat on every line, so this cuts disk usage substantially without deleting anything. I never delete raw logs. Disk is cheap. The cost of not being able to answer "what did this agent output on this date" is not.

Why plain grep stopped being enough

For the first few months, this was the whole system. I'd use ripgrep to search across the uncompressed files, and zgrep for anything already gzipped. That's a legitimate approach and it's genuinely free, no account, no service, just a fast search tool running against files I own. For a while it was enough.

It stopped being enough around the time the log volume passed a few hundred thousand lines. Two problems showed up. First, zgrep across months of compressed files is slow, because it has to decompress everything it touches. Second, and more importantly, grep only understands text. It doesn't know that "agent" is a field name and "content-writer-v3" is a value. So a search for a specific agent name would also match that string if it happened to appear inside someone's output text, which it sometimes did.

Adding a small index instead of a big platform

The fix wasn't a logging platform. It was a script that runs nightly, reads any new JSONL files, and inserts the structured fields into a local SQLite database. SQLite has a full text search extension called FTS5 built in, so I get real text search on the summary fields, combined with normal SQL filtering on the structured columns like date, agent name, and status.

That combination is what actually solved the original problem. Instead of scanning files by hand, I can ask one question: show me every run from the content agent, between two dates, where the status was a failure, and the output text contains a specific phrase. That query runs in well under a second against a year of history, because it's hitting an index, not decompressing files.

I wrapped this in a small command line tool with a handful of flags for agent, date range, status, and keyword. No web dashboard. No login. It runs on the same machine that runs the automations, because that's also where I do the debugging.

Why local instead of a hosted service

I've looked at hosted log aggregation tools before, the kind built for teams running distributed systems. They're built for a different problem than mine. I run automations from one machine, for one person, and the logs contain things like drafts of unpublished content and internal notes. Sending that to a third party service adds a dependency I don't need and a data exposure I'd rather not create, for a problem that a few hundred megabytes of SQLite already solves.

This is also just consistent with how I approach the local versus cloud question generally. Cloud services make sense when you need shared access, scale past what one machine can hold, or need someone else to operate the infrastructure for you. None of that applies here. My log volume for a year fits comfortably on a laptop drive, and I'm the only one querying it.

What this setup can't do

I want to be honest about the limits, because it would be easy to oversell this. The FTS5 search is keyword based. It matches words and phrases, not meaning. If I search for "timeout," I won't find a log entry that says "the request never came back" unless I also search that phrase. There's no semantic search layer here, and I haven't added one, because that would mean embedding every log entry and either running a local model for it or sending log text to an API, and neither has felt worth the added complexity yet.

It's also a single machine solution. There's no access control, no multi-user support, nothing resembling the audit trail a compliance team would want. If I were building this for a team instead of for myself, I'd need a different architecture entirely, probably a real database server and some kind of access layer. For one person debugging their own agents, that would be solving a problem I don't have.

And it doesn't watch anything in real time. This is a forensic tool, not a monitoring one. If an automation is actively failing right now, I still need to check the terminal or the process output directly. The log index answers "what happened," not "what's happening."

What I'd change if I started over

Honestly, not much about the core structure. Structured JSONL per event, organized by date, gzip after a day, index nightly into SQLite, that combination has held up. If I were starting today I'd probably add a couple more structured fields up front, particularly the specific prompt or config version used for each run, since reconstructing which prompt produced which output after the fact is still the most tedious part of debugging a regression. Everything else, I'd keep.

If you're running your own AI automations and want to see how the logging and search side actually works in practice, not just described, I walk through builds like this on the channel.

Find more breakdowns like this on the [home page](/).

Get new guides and videos first — join the Telegram channel.