The backup I never tested until I needed it

The script said ok every single night

I run a small fleet of servers that handle content automation, customer infrastructure, and rendering. Every one of them has a nightly backup job that pushes to cloud object storage. Every one of them logged ok after the run. For months.

The problem was that ok meant “the script finished without throwing an exception,” not “the data made it to the bucket.” Those are different claims, and the gap between them is where this kind of failure lives.

What was actually happening

The backup script would tar up the target directory, hand it to a CLI uploader, and check the process exit code. If the exit code was 0, it wrote ok to the log and moved on. That’s a completely reasonable pattern on paper. The failure mode was that on a subset of machines, the upload step was silently truncating or dropping partway through, and the uploader still returned exit code 0 because from its point of view the connection closed cleanly, it just closed before the transfer finished.

An exit code tells you the process didn’t crash. It doesn’t tell you the bytes you expected are the bytes that arrived. Those are two separate verification steps, and the script only did the first one.

There was a second, smaller issue on one machine: a tar warning (the kind you get from a file changing while it’s being archived, or a permissions notice on a socket file) was being treated as noise and ignored, but on that host it was actually killing the backup step entirely. The log still said the job “ran.” It just didn’t say the job’s output was usable.

Why nobody noticed for months

Nobody looked. That’s the honest answer. The dashboard for this system was “did the cron job fire,” not “can I restore from what it produced.” Those are different questions, and I was only monitoring the first one.

This is a common trap with automation generally, not just backups. When you wire something up to run unattended, it’s tempting to treat “it executed” as the success signal, because that’s the easiest thing to check programmatically. Checking “it executed and produced correct output” usually requires either a second, independent verification step, or a periodic manual audit. Both cost more to build than a simple exit-code check, so they get skipped, especially under time pressure when the system was originally built to solve a more urgent problem than “what if this silently degrades.”

I found this one because I was doing an unrelated audit across the fleet and happened to spot-check bucket sizes against what the source directories should have produced. The numbers didn’t match on several machines. Nothing had alerted. Nothing had failed loudly. It had just been quietly wrong.

The fix, concretely

The fix wasn’t complicated, which is part of why it stings that it wasn’t there from the start:

• After the upload step, the script now does a separate verification pass: it lists the object in the bucket and compares its reported size against the local archive size before declaring success.
• The tar warning that was being swallowed on that one host is now treated as a hard failure for that run, not a log line.
• The nightly log now distinguishes “job ran,” “upload completed,” and “upload verified” as three separate states instead of collapsing them into one.

I rolled this out to the affected machines and confirmed each one is now producing verified, restorable backups. That’s the extent of what I can claim here. I did not run a full disaster-recovery drill across the whole fleet as part of this, and I’m not going to pretend that verifying upload size is the same as proving every backup restores cleanly end to end. It’s a real improvement, not a complete guarantee, and I want to be specific about which one it is.

There’s also still at least one machine in the fleet holding data as a single, unbacked copy, which I know about and haven’t resolved yet. Writing that down here mostly because it’s true, not because it’s flattering.

What this has to do with running AI agents

I think about this failure the same way I think about giving an AI agent write access to something real. An agent that reports “task completed” is making the same kind of claim the backup script was making: I finished the steps I was told to run. It is not automatically the same claim as the output is correct, or the side effect you wanted actually happened.

If you wire an agent into a tool that can send a message, push code, or modify a database, and the only thing you check is whether the agent’s final response looks confident and well-formatted, you’re in the same position I was in with ok in the log. The agent isn’t lying to you any more than my exit-code check was lying to me. It’s reporting the thing it actually verified, which is narrower than the thing you actually care about.

The practical version of this, for anyone running agents against production systems: build a verification step that’s independent of the agent’s own report. If an agent says it pushed a file, check that the file is there with the content you expect, using a separate call, not by trusting the agent’s summary of what it did. If an agent says it sent a message, check the outbox or the delivery log. This is more work up front and it’s exactly the kind of work that gets skipped when you’re moving fast, which is precisely why it’s worth naming as a specific habit rather than assuming you’ll remember to do it under pressure.

What I’d actually tell someone setting this up today

Don’t treat “no error was thrown” as your success condition for anything that writes data somewhere else, whether that’s a backup, an agent action, or a rendering pipeline dropping a file to storage. Pick a verification check that’s cheap to run and independent of the thing you’re verifying. For a backup, that’s comparing sizes or checksums against the source. For an agent, that’s checking the actual state of the system it touched, not its self-report.

And periodically, do the thing you hope you never have to do for real: pull a backup down and open it. Have an agent’s claimed output checked by something other than the agent. The value of that exercise isn’t the confirmation when it works. It’s that it’s the only way you find out, on your schedule instead of an outage’s schedule, when it doesn’t.

If you want more of this kind of writeup, on running AI agents, MCP tooling, and local versus cloud tradeoffs, from someone actually running it day to day, you can find the rest on the xavierfok.com homepage.

Get new guides and videos first — join the Telegram channel.