The automation I rebuilt three times

The first version worked. That was the problem.

I run a handful of content pipelines that use AI agents to research, draft, and queue up videos and articles across a few brands. The first version of one of these automations took an afternoon to build. A script called a model, got a script back, dropped it into a folder, and a render job picked it up. It worked the first time I ran it, and it worked the second time, and I moved on to something else.

It broke on the ninth run. Not loudly. It just started producing scripts that were too short for the video template, and nothing downstream noticed until a render queue sat empty for a day and I went looking for why.

That’s the pattern I want to write about, because I think a lot of people who are new to running agents expect the hard part to be getting a model to do something useful. It isn’t. The hard part is the automation surviving contact with its own output over weeks, not minutes.

Version one: a script and a prayer

The first build was a single Python script. Call the model, take whatever comes back, write it to a file. No validation, no retry logic, no logging beyond a print statement. This is fine for a demo. It is fine for the first ten runs, because the model tends to behave consistently when you’re testing it by hand and watching every output.

It stopped being fine once it ran unattended. The failure mode wasn’t the model refusing to answer or throwing an error I could catch. It was the model succeeding, technically, at a task that didn’t match what the rest of the pipeline assumed. A script that came back a few hundred words short of what the video template needed wasn’t an error. It was a valid response to the prompt. Nothing in the code knew to reject it.

That’s the first lesson worth stating plainly: an automation built around an AI call needs to check the shape of the output, not just whether the call succeeded. A model returning 200 OK tells you nothing about whether what it returned is usable.

Version two: I added checks, and broke something else

The second version added a floor. If the script came back under a word count threshold, don’t send it downstream, flag it instead. That fixed the immediate problem. It also created a new one I didn’t see coming until later: the flagging step wrote to the same status field the render step read from, and under moderate load the two writes could land close enough together that one clobbered the other. A script would get flagged and then, a few seconds later, silently get treated as ready anyway.

This wasn’t a model problem. It was a plain concurrency bug, the kind you’d get in any pipeline with two processes writing to shared state without coordination. But it only showed up because the automation was now running often enough, and long enough, for two processes to overlap. The AI part of the system worked exactly as expected. The plumbing around it didn’t.

I want to be specific about why this happened, because it’s a mistake I see other people make too: it’s tempting to treat the AI call as the whole system and everything else as glue. In practice the glue is where automations fail. The call to the model is usually the most reliable part of the pipeline, because you’re the one who wrote the prompt and you’ve seen its outputs. The state tracking, the queueing, the “did this actually finish” logic around it is where the bugs live, because that’s the part nobody stress tested.

Version three: separating “the model did something” from “the pipeline is correct”

The rebuild that actually stuck split the automation into two concerns that had been tangled together from the start. One process is responsible for getting a usable script out of the model: it validates length, checks that the output matches the expected structure, and retries with a clarified prompt if it doesn’t, up to a fixed number of attempts before giving up and flagging for a human. The other process is responsible for tracking what state each piece of content is in, and it owns that state exclusively. Nothing else writes to it.

That sounds like a basic separation of concerns, because it is. I didn’t need a cleverer prompt or a bigger model to fix this. I needed to stop treating “the model responded” and “the pipeline can move forward” as the same event. They aren’t. A model can respond correctly and the pipeline can still be wrong about what to do next, if the thing reading that response doesn’t have a clear, single source of truth for state.

The other change in version three was logging that a human can actually read later. Not debug-level noise, but a line per run that says what came in, what came out, and what decision the pipeline made about it. When something goes quiet, the first thing I do now is check that log, not re-run the script and hope. That habit alone has saved more time than any prompt tweak I’ve made.

What this says about running agents generally

None of the three rebuilds involved changing which model I was calling, and none of them involved a smarter prompt. All three were about the code around the AI call: what it assumes, what it checks, and what happens when the model’s output is technically valid but not what the next step needs. This is the part that doesn’t show up in a five-minute demo of an agent doing something impressive. It only shows up when the same automation has to keep working on the fortieth run, the four hundredth run, at 3am, with nobody watching.

If you’re setting up something similar, whether it’s an MCP-connected agent doing research, a local model handling a batch job, or a cloud API generating drafts, the question worth asking isn’t “does this work.” It’s “what does my code do when the model gives me something that’s valid but wrong for this context.” If the answer is “nothing, it just passes it along,” that’s the version that’s going to fail quietly in a few weeks, the same way mine did.

I also want to be honest about the limits here. None of this made the automation smarter than it was. It didn’t start catching factual errors or judging quality the way a person would. What it does now is refuse to move forward on output that doesn’t meet a structural bar I defined, and it tells me clearly when that happens instead of letting a downstream step fail silently. That’s a narrower win than it might sound like, and it’s the realistic one. The AI still does the same job it always did. The pipeline around it just stopped lying to me about whether that job got done correctly.

Where this leaves the setup now

The automation runs unattended most days. When it flags something, I look at it, and usually the fix is a prompt adjustment or a one-off manual pass, not another rebuild. That’s the actual measure of whether an automation is done: not that it works once, but that when it fails, it fails in a way you can see and fix without tracing it back through three layers of silent state.

If you’re building something similar and want to see how the pieces fit together in practice, browse the rest of the site for more on local versus cloud tradeoffs and how I wire agents into real tools: xavierfok.com

Get new guides and videos first — join the Telegram channel.