The automation that only breaks when I am asleep

The 3am pattern

I’ve run enough automated pipelines by now to notice a pattern that has nothing to do with bad luck. The jobs that fail overnight are disproportionately the ones nobody is watching. Not because midnight has some special curse on it, but because overnight is exactly when a pipeline is running unattended for the longest uninterrupted stretch, chaining the most steps together, and touching the most external systems without a human in the loop to catch the first small thing that goes wrong before it becomes five things.

During the day, I’m poking at the same scripts, restarting a stuck process within minutes, noticing a weird log line before it turns into a queue backing up. At night, none of that happens. The pipeline just keeps going, one step feeding into the next, and if step three quietly fails, step four still fires on bad input, step five renders garbage, and by morning I’ve got a folder of broken output and a queue that never advanced.

This isn’t really an AI problem. It’s an unattended systems problem that AI pipelines happen to be especially exposed to, because they tend to be longer chains than a normal script.

Why it’s rarely the model’s fault

When people imagine an AI automation breaking, they usually picture the model itself doing something wrong. A hallucinated answer, a weird output, a refusal. In my experience running actual pipelines, that’s not where the overnight failures come from. The model layer is usually the most predictable part of the whole thing.

What breaks is everything around the model: a disk filling up because renders weren’t cleaned up, a rate limit on an API that only bites once volume passes a certain point, a text-to-speech step that silently produces a truncated file instead of erroring out, a queue file that gets written to by two processes at once and ends up corrupted, a network blip that a script doesn’t retry correctly. None of that is exotic. It’s the same class of failure that has always haunted cron jobs and batch processing. AI pipelines just have more steps, more external calls, and more places where one small hiccup can cascade.

The parts of a pipeline that don’t sleep

A content pipeline that runs overnight usually looks like a chain: fetch source material, generate a draft, run it through a review or formatting step, generate audio, render video, publish or queue for publishing. Each of those steps is a separate process, often on a separate schedule, sometimes on a separate machine.

During the day I can watch this chain step by step. At night it runs as a chain of assumptions. Step two assumes step one produced valid output. Step three assumes step two didn’t time out halfway through. None of these assumptions are unreasonable, they just don’t hold 100% of the time, and the time they don’t hold is disproportionately going to be the long unattended overnight run, simply because that’s when the most total steps execute without anyone glancing at a terminal.

What actually breaks overnight

The specific failure modes I run into aren’t dramatic. They’re mundane in a way that’s almost embarrassing to write down, but that’s the honest picture:

A rendering job that produces a file, but the file is empty or truncated because a dependency crashed partway through instead of failing loudly. The pipeline doesn’t check the file, so it moves on to the next step as if everything worked.

A queue or state file that two scripts touch at the same time, because their schedules happened to overlap that night. One of them wins, the other’s changes vanish, and the pipeline’s idea of what’s “done” no longer matches reality.

A script that treats any error as fatal and stops the whole batch on the very first line of stderr, instead of logging it and continuing. I’ve had jobs die at item one of fifty because of a single warning that didn’t actually mean anything was broken.

A network call that works fine at low volume during the day but starts getting rate-limited once a script fires off requests back to back for hours with nobody around to notice the pattern and space them out.

None of these are AI failures. They’re the ordinary failure modes of unattended software, showing up in an AI context because that’s where I happen to have long chains of steps running without supervision.

Local vs cloud: different failure clocks

Running things locally versus in the cloud changes what breaks overnight, but it doesn’t make the problem go away.

Local automation fails on resource limits: disk space, USB devices that drop off the bus, a machine that loses its network connection and doesn’t reconnect on its own, a process that leaks memory over a long run and eventually gets killed by the OS. These are physical, tangible failures. You can usually trace them to a specific piece of hardware or a specific script that didn’t clean up after itself.

Cloud-dependent automation fails on a different clock: API rate limits, quota resets, a provider having a rough night on their end, a network path that gets flaky for twenty minutes and then recovers. You don’t control that clock, and unlike a local resource limit, you often can’t predict it from your own usage patterns alone.

Neither is inherently more reliable than the other. What matters is that the two failure modes are different enough that a script written to survive one won’t automatically survive the other. A retry loop that handles a rate limit gracefully does nothing for a full disk. A disk space check does nothing for a stalled API call. If your pipeline mixes local and cloud steps, which most real pipelines do, you need to handle both classes of failure explicitly, not just the one you happened to think about first.

Agents make this worse, not better

Adding an agent into an overnight pipeline sounds like it should make things more resilient. An agent can reason about a weird situation, right, instead of just crashing on it. In practice I’ve found the opposite is often true for unattended runs.

An agent given a tool and a goal will keep trying to make progress even when the right move is to stop and flag the problem. If a script hits an error, it usually just stops, which is annoying but at least visible: you wake up to a stopped job. An agent hitting the same error might retry, work around it in some way you didn’t anticipate, or take an action that technically “succeeds” but isn’t what you wanted, because its instructions told it to keep going and it doesn’t have the judgment to know that “keep going” wasn’t actually a good idea for this specific failure. A stuck script is a known state. An agent that improvised its way through the night is an unknown one, and unknown states are worse to wake up to than a plain crash.

This isn’t a reason to avoid agents in automation. It’s a reason to be deliberate about where their autonomy stops. I give agents narrow, well-defined tools for specific steps, not open-ended authority to “figure out” a whole overnight run. The more autonomy a step has, the more I want a human checking its output before it feeds into anything downstream.

What I’ve changed because of this

The single biggest change I’ve made isn’t a smarter script, it’s stricter boundaries between steps. Every stage in a pipeline now checks its own output before marking itself done, rather than assuming success because it didn’t crash. An empty file is treated as a failure, not a completed step. A stage that hits an error logs it and stops that item, not the entire batch, so one bad item overnight doesn’t take down forty good ones behind it.

I’ve also gotten more disciplined about separating “this ran” from “this worked.” A script exiting with no error tells you almost nothing on its own. What tells you something is a check on the actual output, a file size, a duration, a status the next step can verify before trusting it. That single habit has caught more overnight problems for me than any retry logic or alerting system.

The real fix isn’t more automation

There’s a tempting instinct to solve automation failures by automating the failure handling too: more retries, more self-healing, more layers watching the other layers. Some of that helps. But the honest lesson from actually running this stuff every day is that the fix is usually simpler and less glamorous: fewer assumptions between steps, explicit checks instead of implicit trust, and pipelines designed so that one bad step fails loudly and stops, instead of quietly poisoning everything after it. That won’t stop things from breaking while you sleep. It just means what’s waiting for you in the morning is a clear, fixable problem instead of a mess you have to reverse-engineer over coffee.

If you want more of this kind of grounded, no-hype look at running AI automation in the real world, come check out the rest of the site at xavierfok.com.

Get new guides and videos first — join the Telegram channel.