The dry run mode that lied to me
What dry run mode is supposed to do
Dry run mode exists to answer one question before you commit to anything: what would happen if I actually ran this. You flip a flag, the script walks through its logic, prints what it would have done, and stops short of touching anything real. No files written, no API calls that mutate state, no messages sent, no money moved. It's supposed to be a preview of the live run with the consequences switched off.
That's the promise. In practice, a dry run only tells you what the code path you're dry-running actually exercises. If the dry run and the live run share the same code up until the last step, the preview is trustworthy. If they diverge earlier than that, the preview is a guess wearing a report.
I run a fair amount of AI-driven automation day to day: content generation across a few brands, local rendering for video, agents wired into MCP tools that touch email, file systems, and a handful of internal dashboards. Dry run flags show up constantly in this kind of work, because the failure modes are expensive enough that you want a look before you leap. One of them lied to me in a way that was instructive enough to write down.
The pipeline I run
The setup was a publishing script. It pulls a batch of drafted articles, checks them against a set of gates (word count, required fields, a lint pass, a duplicate-topic check), and then pushes the ones that pass to the live site. Nothing exotic. The dry run flag was supposed to run every gate exactly as the live run would, then stop before the actual publish call.
I ran it against a batch of ninety-nine articles. It reported that all ninety-nine were clean and ready. Given that number, I expected the live run to publish ninety-nine articles with no surprises. Twenty-six of them failed on the live run instead, some outright rejected, some published with fields that were technically present but wrong, empty strings that had passed a "field exists" check but not a "field has content" check.
Where the lie crept in
The gap wasn't in the gates I could see. It was in a validation step that only executed on the live code path, not the dry run path, because it lived inside the same function that made the publish API call. The dry run short-circuited before that function ran, which meant it short-circuited before that validation ran too. The script wasn't lying on purpose. It was reporting exactly what it checked. It just hadn't been built to check the same things the live run checked, and nothing in its output told me that.
This is the general shape of the problem with dry run modes, especially once AI is involved somewhere in the pipeline. A dry run is only as honest as the overlap between its code path and the real one. The moment someone adds a step that's conditional on "are we actually doing this," that step becomes invisible to every dry run that comes after, and nobody goes back to check whether the preview still means what it used to mean.
Why dry run and live diverge more in AI pipelines specifically
With plain deterministic scripts, this kind of drift is annoying but bounded. Once you find the missing check, you know exactly what was wrong and you can fix it once.
AI-driven steps make the divergence harder to spot, because the thing you're previewing often isn't fully deterministic in the first place. A content pipeline that calls a model to generate or rewrite text can behave differently on a dry run versus a live run even with identical inputs, simply because the same prompt doesn't guarantee the same output twice. If your dry run caches or mocks the model call to avoid burning a real request, and your live run does not, you've now built two different pipelines that happen to share a name. The dry run tells you the shape of what could happen. It doesn't tell you what will.
Agent-based tooling adds another layer. An agent wired into MCP tools decides which tool to call and in what order based on the state it's given at that moment. A dry run that stubs out the tool calls isn't testing the agent's actual decision-making, it's testing what the agent would try to do against a fake environment. If the real environment has a rate limit, a stale auth token, or a field that's null in production but always populated in your test fixture, the agent's real behavior can differ from its dry-run behavior in ways that have nothing to do with a bug in your code. It's the agent correctly adapting to a different reality than the one you showed it.
None of this means dry runs are useless or that the tooling is broken. It means a dry run answers a narrower question than people assume it does. It's a preview of intent, not a guarantee of outcome, and that gap widens whenever probabilistic decisions sit anywhere in the path.
The fix wasn't a flag, it was a smaller live loop
After the twenty-six failures, the instinct is to patch the dry run so it exercises the exact same validation the live run does. I did that, and it helped, but it wasn't the real fix. Even a perfectly mirrored dry run can't preview a model call it isn't allowed to make, or a rate limit it isn't allowed to hit.
What actually closed the gap was shrinking the live run itself. Instead of trusting a dry run across ninety-nine articles and then firing all ninety-nine live, I started publishing in small live batches, five or ten at a time, and checking the real result before moving to the next batch. That's slower. It also means the first batch is doing the job the dry run was supposed to do, except honestly, because it's not a preview, it's a small real run with real consequences that I can still afford to unwind if something's off.
For anything touching an agent or a model call, I stopped trusting a mocked dry run as a stand-in for behavior and started treating the first few live executions as the actual test, watched closely, with a way to stop the batch if something looks wrong. The dry run still runs first, because it catches the cheap, deterministic mistakes, missing fields, obvious lint failures, duplicate entries. It just doesn't get the final word anymore.
What I actually trust now
I still use dry run flags. They're worth keeping because they catch a real class of mistakes cheaply, the ones that don't depend on anything probabilistic. What changed is how much weight I put on a clean dry run result. A clean dry run tells me the deterministic checks passed. It doesn't tell me the model will generate what I expect, that the agent will pick the tool I'd pick, or that an API will behave the same way at 2pm as it did during the test.
The practical rule I use now: if a step in the pipeline involves a model call, an agent decision, or anything hitting a live external service, the dry run for that step doesn't get to make the go or no-go call by itself. A small, watched live batch does. It costs a bit more time up front. It's cost a lot less than finding out about twenty-six broken articles after the fact.
The general lesson
Dry run mode isn't a lie by design. It's a preview built by someone, at some point, who decided which parts of the real path were worth simulating and which parts were too expensive or too risky to include. Every step added after that decision either gets folded into the preview or quietly excluded from it, and the exclusion never announces itself. The output still says "clean," because as far as the dry run knows, it is.
If you're running AI anywhere in a pipeline, that gap is wider than it looks, because the parts you'd most want previewed accurately, model output and agent decisions, are exactly the parts a dry run struggles to simulate honestly. Trust it for what it actually checks. Verify the rest with a small live run you can watch.
If you want more of this kind of writing on running AI automation without the hype, you can find the rest of it on the [home page](/).
Get new guides and videos first — join the Telegram channel.