Structured output from LLMs: turning prose into data you can trust
# Structured output from LLMs: turning prose into data you can trust
Wire a language model into a real program and you hit this wall almost immediately. You ask a simple question and want back one clean piece of data. The model hands you a friendly paragraph instead: sure, here is the information you asked for, followed by three sentences of chat with your one number buried somewhere inside. A human reader shrugs and moves on. A program chokes.
The program wants a value, in a known shape, every single time. Getting that from a model reliably is a solved problem, mostly, but the solution has layers, and the order you learn them in matters. Here is how I get structured data out of models in pipelines that run unattended, and the parts that still go wrong when everything is set up correctly.
Prose is the fragile part of a pipeline
In a chat window, prose is the right output. You are a person; you read it. The moment a model's answer feeds the next step of a pipeline, that same prose becomes the most fragile component in the system.
Say the next step needs a price. Now you are writing brittle string searches to fish a number out of a sentence the model is free to phrase a hundred ways. One day it writes the price is twelve dollars. The next day, that will run you about twelve bucks. Your search misses, nothing complains, and something downstream quietly does the wrong thing. The format itself has become a moving target.
What a pipeline needs is a contract. Give me your answer in this exact shape, with these exact field names, and nothing else. No greeting, no explanation, no trailing pleasantry. Just the data. Real ways to get that contract exist. Most people start with the worst one.
Asking nicely works until it matters
The first thing everyone tries is a line in the prompt: respond only with JSON, add no other text. With a good model this works most of the time, and that is precisely what makes it dangerous. It works in testing. It works in the demo. It ships. Then one day, on an input you never tried, the model helpfully wraps the JSON in a code fence, or adds a sentence explaining what it did, and your parser falls over.
Most of the time is a death sentence at scale. A step that succeeds ninety nine times in a hundred, running a thousand times a day, fails ten times a day, quietly. And the timing is the insidious part. It never breaks while you are watching. It breaks weeks later, on a strangely shaped input from a real user, long after you stopped thinking about this code. I have lost more than one afternoon to a bug that turned out to be a single stray sentence the model felt like adding, and those afternoons are what taught me to never let format depend on good behavior.
Hand it a schema instead of a description
The first real upgrade: stop describing the shape in words and hand the model an actual schema. A schema is a precise statement of what you expect. This field is text. This one is a number. This one may only hold one of three listed values, and these fields are required.
The difference is telling a contractor make me a nice form versus handing over the form with the boxes already drawn. Intent stops being guesswork. A schema does two jobs at once: it tells the model exactly what you want, in a language with little room for creativity, and it gives you something to check the answer against later, which becomes the whole story in a minute.
When I build one of these steps, the schema absorbs most of my effort and the prompt shrinks to a pointer that says fill this in. There is a quiet long-term bonus too. Six months on, when I have forgotten how a step works, the schema is the documentation. A fuzzy paragraph of instructions rots. A precise schema stays true, because the code downstream depends on it and would break the day it drifted.
Constrained decoding closes the loophole
Here is the piece most tutorials skip, and the one that changed my results. The strongest approach never lets the model produce free-form text in the first place. It goes by several names, JSON mode, structured output, constrained decoding, and the idea under all of them is the same: instead of letting the model pick any next word and hoping the words add up to valid JSON, the system restricts its choices at every step so that only output matching your shape is even possible.
A greeting physically cannot appear, because a greeting is never a valid next token in the structure. A closing bracket cannot be forgotten, because the structure refuses to stop without it. Plenty of providers and plenty of local tools now support some version of this, and where it exists I turn it on without debate. Asking nicely is requesting that someone color inside the lines. Constrained decoding is handing them a stencil.
One careful caveat. The shape being guaranteed says nothing about the content. The model can still put a wrong value in the right box, confidently. What disappears is the chatter, the code fences, the malformed structure, and that removal takes a huge amount of fragility out of the system.
Validate anyway, every time
Even with constrained decoding on, no output touches the rest of my pipeline unchecked. The next step is always validation: take what came back and check it against the same schema, in code. Is the price actually a number, or the word twelve spelled out? Is the category one of the allowed values, or an invention? Are the required fields all present?
Validation is cheap to write and pays for itself the first time it catches something, and it will catch something. The model is good and imperfect at the same time, and a system's whole job is continuing to work when a component misbehaves. I picture the model as a fast, clever assistant who occasionally hands back a form with one box filled in wrong. Nobody feeds that form into the machine without a glance. The validator is the glance, performed every time, immune to boredom.
When validation fails, and it will
Three honest options exist, and I use all of them depending on how much the answer matters.
Retry first. Models carry a little randomness, so the same request often comes back clean the second time.
Repair next. Hand the broken output back with the error: this did not match the shape, here is what was wrong, fix it. The second pass succeeds surprisingly often, since pointing at a specific mistake is a far easier task than getting everything right cold.
Fall back last. When an answer keeps failing, crashing helps nobody, and passing garbage along is worse. Flag that record for a human and let the rest of the batch keep moving. The goal was never a system with zero failures. The goal is failure that stays small and visible, instead of one malformed row silently corrupting a thousand others.
A real example: invoices
This stopped being abstract for me with a pile of messy invoices in wildly different layouts, from which I wanted a few fields: the vendor name, the total, the date, and the line items. The old approach was brittle templates and a lot of swearing.
The new way: define a schema with exactly those fields, switch on structured output mode, feed the model the raw text of each invoice one at a time. Back comes a clean object, the same shape every time, however wild the source layout. Then the validator runs. The total must be a number, the date must look like a date, the vendor cannot be empty. Whatever passes flows straight into my records, and whatever fails goes into a small pile for my eyes, usually two or three oddballs out of a few hundred.
The result is that a boring, error-prone reading job became a quiet background process, and my attention gets spent only on the genuine edge cases that need a human.
The limits, stated plainly
Structured output makes the shape reliable and leaves the truth of the content entirely open. A perfectly formed object can carry a misread total, and confusing a valid structure with a correct answer is how people get burned. Values that matter still get sanity checks.
Constrained decoding also carries a small tax. It can run slightly slower, and some setups lack support for it entirely, which drops you back to asking nicely plus heavy validation. Deeply nested schemas confuse models the way they confuse people, so my shapes stay as flat and simple as the job allows. None of this argues for skipping structured output. It is the difference between using the tool with open eyes and trusting a magic box.
The order to learn it in
If you only chat with models, none of this applies; just talk. The moment a model's answer feeds a program instead of a person, structured output stops being optional, and it is what separates a clever demo from a thing you can leave running.
The path is short. Ask nicely first, to understand the problem. Add a real schema, so intent becomes precise. Switch on constrained decoding wherever your tools support it, so shape becomes guaranteed. And validate everything that comes back, always, because the model will eventually surprise you, and code is where you want to find out, well before production data does.
Get those four habits down and a model can sit inside almost any pipeline without keeping you up at night wondering what it quietly mangled. I write about this kind of unglamorous plumbing, the parts the flashy demos skip, at [xavierfok.com](/).
Get new guides and videos first — join the Telegram channel.