What I Changed After a Model Update Broke Three Jobs

The morning I noticed something was off

I run a handful of jobs that lean on AI models to do real work: summarizing source material before it gets turned into a video script, tagging content for routing, and formatting output for a downstream renderer. None of this is exotic. It’s the kind of plumbing anyone running AI in production ends up building, whether they call it an “agent” or not.

One morning I noticed the output queue looked thin. Not empty, just thin. A few items had gone through with obviously wrong formatting. A tagging job had started leaving fields blank instead of filling them in. Nothing had crashed. No error logs, no alerts, no stack traces. The jobs reported success the whole time.

That’s the part that gets people. A broken API call is loud. A broken model response is quiet.

What actually happened

The provider I use had pushed a model update. Not a new model I opted into, an update to the same model ID I was already calling. Same name, same endpoint, different behavior underneath.

Three jobs were affected, and each one failed in a different way, which is worth walking through because “the model changed” isn’t a diagnosis, it’s a category.

Job one: summarization. The prompt asked for a summary under a certain length, formatted as plain text with no headers. The updated model started adding markdown headers and bullet points by default, even when told not to. The summaries were still accurate. They just didn’t match the format the next step in the pipeline expected, so the renderer choked on stray asterisks and hash symbols showing up as literal characters.

Job two: tagging. This one asked the model to output a small set of category tags from a fixed list, one per line. The old model behavior was consistent about staying inside that list. The new version started paraphrasing categories that were close but not exact, things like writing “AI tools” instead of the expected “ai-tools” tag. My matching logic was doing exact string comparison, so those items just fell through with no tag applied. No error, just silence.

Job three: structured output for the renderer. This one asked for JSON. The update changed how the model handled edge cases in the prompt, specifically ones where source material was thin or ambiguous. Instead of returning a JSON object with an empty field, it started returning an explanatory sentence about why it couldn’t complete the field, wrapped in the same JSON structure but with prose where a value should be. My parser expected a string or null, not a paragraph, so it stored the explanation as if it were the actual content.

Three different failure shapes, one root cause: the model changed and nothing in my pipeline was set up to notice.

Why this is easy to miss

None of these are edge cases you’d catch by staring at a diff, because there’s no diff to stare at. You didn’t change your code. The provider changed the thing your code calls. If you’re treating a model like a stable function that takes input and returns predictable output, that assumption is the bug. Models are not stable functions. They’re closer to a vendor you don’t have a support contract with, whose product spec shifts on their schedule, not yours.

The other reason it’s easy to miss is that “success” and “correct” aren’t the same signal. All three jobs returned a 200, took a normal amount of time, and produced text. Nothing about the technical layer of the request looked wrong. The problem lived entirely in whether the content of the response still matched what the downstream code expected, and that’s a layer most people don’t instrument at all.

What I changed

I didn’t rebuild anything from scratch. I added checks at the specific points where these three failures happened, and I made a couple of process changes to catch the next one faster.

Format validation, not just success validation. For the summarization job, I added a check that runs after the model call: does the output contain markdown syntax when it shouldn’t? Does it exceed the length ceiling? If either check fails, the job doesn’t silently pass through, it gets flagged and held instead of going straight to the renderer. This is a five-line regex check, not a new system. It just needed to exist.

Strict matching with a fallback path, not a silent drop. For the tagging job, I kept the exact-match logic but added a step before the drop: if a returned value doesn’t match the fixed list exactly, log it to a review file instead of discarding it. I’m not trying to make the matcher smarter or add fuzzy matching that guesses what the model meant. I’d rather see the mismatch and decide manually than have the system quietly reinterpret categories I never approved.

Type checking before storage. For the structured output job, I added a check that the JSON field is actually the type it’s supposed to be before it gets written anywhere. If a field that should be a short string or null comes back as a long paragraph, the job stops and flags it instead of saving it as-is. This one is basic input validation that honestly should have been there from the start. The model update didn’t create the need for it, it just exposed that I hadn’t done it.

A weekly spot check on live output, not just logs. Logs told me these jobs succeeded. They didn’t tell me the output was wrong. So I added a short manual step to my week: pull five to ten recent outputs from each automated job and actually read them, not just check that they exist. This isn’t automatable in a way I trust yet, and I’m not going to pretend a script can judge “does this read right” as well as a person glancing at it for two minutes.

Pinning where the provider allows it, and accepting the tradeoff where it doesn’t. Some providers let you call a specific model snapshot instead of the general model ID that gets updated underneath you. Where that option exists, I use it for jobs where consistency matters more than getting the latest improvements automatically. Where it doesn’t exist, I accept that the model can shift under me and lean harder on the validation steps above instead. Pinning isn’t a fix on its own. It just buys you control over when the change hits, not whether one eventually will.

What I didn’t do

I didn’t add a general-purpose “AI output sanity checker” layer that tries to catch anything unexpected across every job. That’s the kind of abstraction that sounds appealing after an incident and turns into its own maintenance burden six months later. Each of these three checks is specific to what that job actually expects back. A generic layer would have to guess at correctness across wildly different tasks, and guessing is exactly the failure mode I was trying to get away from.

I also didn’t switch models or providers over this. The update caused real problems, but the fix wasn’t “distrust the vendor,” it was “stop assuming the contract between my code and the model’s output is more stable than it actually is.” That assumption was mine to fix, not theirs to guarantee.

The actual lesson

If you’re running AI as part of a real pipeline, treat every model call as an external dependency that can change its behavior without telling you, because that’s what it is. Validate the shape of what comes back, not just whether the call succeeded. Keep a human glance in the loop somewhere, even a small one. None of this is glamorous, and none of it required rebuilding the pipeline. It just required admitting that “the model returned something” was never the same claim as “the model returned the right thing.”

If you want more of this kind of practical rundown on running AI tools day to day, without the hype, you can find the rest of what I write and record over on xavierfok.com.

Get new guides and videos first — join the Telegram channel.