Fine-tuning vs prompting: when each is actually worth it
# Fine-tuning vs prompting: when each is actually worth it
The question I get more than any other is some version of "should I fine-tune a model for my use case". I understand the pull. The phrase sounds serious, technical, like the thing real practitioners do. But I have watched a lot of people go down that road who did not need to, and my honest answer, almost every time, is no. Or at least: no for now, and probably no for a long while.
The decision sounds complicated. Strip the hype away and it is simple.
Prompting changes the conversation, never the model
Prompting is everything you do with words at runtime. The system prompt that sets the role and the format. The examples you paste in to show what good output looks like. The context you attach so the model knows what it is working with. None of it touches the model itself. The weights, the billions of numbers doing the actual thinking, stay exactly as they were. The model forgets your instructions the moment the conversation ends, and next time you simply set them again.
The payoff is iteration speed. Write, test, adjust, run again: a full cycle takes minutes, with no dataset and no training job and no waiting. Being wrong costs almost nothing, so you can be wrong a hundred times on the way to the version that works. That speed is one of the most underrated properties of working with these models, and people give it up far too casually.
Fine-tuning changes the weights, and your maintenance load
Fine-tuning is a different animal. You take a pretrained base model and run additional training on examples you supply, and your patterns get pushed into the model's internal parameters. After that, the model produces your formats and your tone without being told. The patterns are simply in there.
That is genuinely powerful when the situation calls for it. Here is the machinery it costs. You need a dataset, usually hundreds of carefully formatted examples at minimum and often far more. You need a training run, which is metered compute. And the part people miss: when the provider ships a better base model, your tuned model is suddenly behind it, and catching up means redoing the training on the new base. The dataset stops being a one-off investment and becomes a recurring maintenance item. That is the honest price of admission.
The overhead gap
Side by side, the difference is stark. Prompting costs the time it takes to write and refine instructions, and retesting happens in the same afternoon. Fine-tuning costs a dataset that has to be assembled and labelled, plus the training compute, plus the retraining whenever the base moves.
Prompting has real limits too. Context is finite, attention spreads thin as prompts grow, instructions past a certain complexity stop being followed reliably, and a giant context window brings real latency. Fine-tuning can relieve some of that by baking behaviour in. But the threshold where that relief outweighs the overhead sits much higher than most people expect when they first ask the question.
Where a prompt is simply enough
For most of what people actually do, a well written prompt covers it. A task you can describe in words is a task prompting can usually handle. A task that changes next month is handled by editing the instructions. A task you are still exploring is exactly where you want an approach with free iteration.
One-off jobs are the obvious case: summarise this document, classify this data, draft this message in this tone. Nobody should build a training dataset for that. But recurring tasks stay in prompting territory far longer than people assume. If a clear system prompt produces the output you want reliably, you are done. Modern frontier models produce genuinely strong output from a well crafted prompt, and most people asking about fine-tuning have never come close to that ceiling.
Where fine-tuning earns its cost
A few situations genuinely justify it. The clearest is a narrow, repeated task at serious volume, where cost or latency matters. Run the same extraction or classification millions of times and the economics shift: a small fine-tuned model can match the behaviour you need at a fraction of the price of a large model dragging a long prompt behind it on every call.
Strict format requirements are the second case. If output absolutely must land in a specific schema, every field, every time, tuning can push reliability past what any prompt manages. The third is a very specific voice held consistent across huge volumes of content, where prompts alone start to wobble at scale.
Notice what the three share: scale, repetition, and a tightly defined task. Fine-tuning adjusts behaviour, and only behaviour. Expecting it to make a model smarter about your domain leads straight to disappointment, because adding knowledge is a different problem with different tools.
The middle ground people jump over
Between plain prompting and full training sits a band of options that gets skipped surprisingly often.
Few-shot examples first. Rather than describing what you want, show it: two to five pairs of input and exact desired output, sitting right in the prompt. The model reads the pattern directly instead of inferring it from a description, and for many tasks this closes most of the gap between a mediocre prompt and a great one. Examples cost a little context length, update as easily as editing a text file, and need no pipeline at all. Try them before anything more expensive.
Retrieval is the other one. If the model needs to know facts about your product or your documents, look the facts up at runtime and place them in the prompt. They stay current without retraining, and the model can point at what it used. Baking facts in through fine-tuning works poorly; models tend to hallucinate unpredictably around trained-in facts. For factual grounding, retrieval is the more reliable tool by a wide margin.
Why people reach for training too early
The pattern I see is consistent. A task is underperforming with a basic prompt. Instead of spending two more hours improving that prompt, the builder concludes the model is the limitation and training is the cure. Occasionally that is true. Far more often the prompt was mediocre, and fine-tuning on top of a mediocre prompt yields a model that reproduces mediocre output with great consistency.
The diagnostic I would run before any tuning project: have you written a genuinely detailed system prompt? Added well chosen examples? Broken the task into smaller steps? A no to any of those means the ceiling has never actually been reached. A day on the prompt is cheaper than a week on a dataset, and the returns from prompt work on a modern model are larger than almost anyone expects before trying.
What fine-tuning will not do
Worth being blunt here, because expectations run wild. Fine-tuning does not make a model smarter. It cannot add world knowledge or new reasoning ability; the base model's intelligence and cutoff are fixed. It cannot cure hallucination either, and a tuned model can pick up new failure modes of its own, over-applying your training patterns in places they were never meant to go. You are shaping style and behaviour. The underlying tendencies of the base model remain.
The order I actually follow
Start with a prompt and make it genuinely good. Add examples. Break hard tasks into steps. If facts are the gap, add retrieval. If format is the gap, try structured output constraints. If tone wanders, add more style examples. Only when all of that has been done, and a real documented ceiling remains, and the volume justifies the overhead, does fine-tuning go on the table.
By the time people work through that sequence, most find the problem dissolved somewhere earlier than expected. And the ones who do end up training are better for the detour: the dataset and the evaluations come out sharper when you understand the task from all the prompt work that came first. The teams getting the best results are the ones who got very good at prompting, used retrieval for facts, and reached for training only at a proven wall. That order is simply the efficient path.
I write more breakdowns like this at [xavierfok.com](/).
Get new guides and videos first — join the Telegram channel.