How prompt caching cut my AI bill
# How prompt caching cut my AI bill
A few months ago my model bill started creeping upward and I could not explain it. Usage had not changed shape. No new flood of users, no switch to a pricier model, the same kinds of requests at roughly the same volume, and the number at the bottom of the invoice kept climbing anyway. Eventually I did the boring, necessary thing and traced where the money was going, line by line. The answer was almost embarrassing. I was paying, thousands of times a day, for the model to read the same enormous block of text it had already read thousands of times before. The fix is a feature called prompt caching, and I want to lay out what it is, why it cut my bill so hard, and the honest catches, because it helps some workloads enormously and others none at all.
Where the money was going
A call to a hosted model carries more than the user's question. In front of that question sits the setup: the instructions, the rules to follow, the examples to imitate, and the background reference the model needs to behave consistently. In my app that block ran to a couple of dense pages of carefully written material, and it almost never changed from one call to the next.
Here is the thing I had simply never stopped to consider. Every call re-sent that entire block from scratch, as though the model had never laid eyes on it. The user's question, the part I actually thought about, was a line or two. The stable prefix in front of it made up most of every request, and I was transmitting the whole of it, at full price, on every single call.
What you actually pay for
To see why that matters, you have to know what the meter measures. Hosted models bill by the token, and on both sides of the exchange. You pay for everything you send in, setup included, and separately for whatever comes back out. For a lot of applications, mine very much included, the input side dominates, because a long instruction block dwarfs a short question and a short answer.
Two pages of setup ahead of a one line question means the bill is mostly the two pages. Each individual call looked cheap on its own, which is exactly why I stayed blind to it for months. Summed across a day, that repeated input was the single largest line on the bill.
What prompt caching actually is
Prompt caching lets the provider remember the processed form of a prompt prefix, so the model does not redo that work on every call. The first time the big setup block arrives, the model processes it in full, like normal, and stores the result. On the next call, if the prompt opens with exactly the same text, character for character, the stored work gets reused instead of recomputed.
The part that saves money is the pricing that follows from it. Cached tokens are billed at a much lower rate than fresh ones, because the provider is doing far less work to handle them. So the stable prefix that used to cost full price on every call drops to a small fraction of that price once it is cached. You still send it across the wire. You stop paying to have it read fresh each time.
Why the trick works
Underneath the feature is a split that holds for almost every production app. A prompt has a part that never changes across calls, the instructions and examples and reference material, and a part that changes every time, usually the user's specific input sitting at the very end. The unchanging part is very often the bulk of the prompt by weight, and it is identical from call to call.
Caching was built for precisely that shape. Remember the fixed front, do fresh work only on the small variable tail. Once you start seeing your prompt as one large constant block followed by a tiny variable suffix, the waste of reprocessing the constant at full price on every call becomes hard to unsee.
Structuring the prompt for it
This is the actionable part, because caching only pays off if the prompt is laid out to take advantage of it. The rule is simple. Everything stable goes at the very front. Everything that varies per call goes at the very end. The cache matches from the first character forward and stops at the first difference, so the longer the identical run at the start, the more you save. A single changing value near the top, a timestamp or a session id, breaks the match immediately and ruins caching for everything after it.
So I went through my prompt carefully and reorganized it. Stable instructions moved up to the front. Per request details moved down to the bottom. The edit was small and boring, it changed nothing about how the model behaved, and it changed the bill a great deal.
The honest result
I will describe the shape of the outcome rather than pretend to precision. The repeated setup block accounted for the large majority of my input tokens. With caching in place, the effective price of that repeated portion fell to a small fraction of what it had been, and on the calls that hit the cache, the overall cost of that workload came down by well over half. The model did no less work for users and served no fewer of them. I just stopped paying full price to re-read the same instructions thousands of times a day.
The entire change took an afternoon, and nearly all of that afternoon went into understanding the problem. The typed fix was a reordering and a feature switched on. That ratio is what has stayed with me since. A real optimization usually looks like this, a small, almost embarrassing tweak that was available from day one, unlocked by finally seeing where the money leaked. Spend your effort on understanding first. The clever fix is usually the minor part.
The gotchas
There are real catches, and they matter in practice.
A cache does not live forever. It expires after a stretch of disuse, so calls spread far apart in time keep finding it cold and paying full price to warm it back up. The feature rewards a tight rhythm of repeated hits against the same prefix.
Writing to the cache the first time usually costs slightly extra. A prefix you only ever use once nets out marginally worse with caching than without it. The win comes entirely from repetition.
And the matching is strict and order sensitive. One stray changing character near the front and the whole match silently misses, with no error message to tell you it happened. This is a feature you lay out your prompts around and then keep an eye on, never a switch you flip once and forget.
Where it does nothing
If every call is unlike the last, with no big shared setup running through them, there is nothing stable to cache and the feature does precisely nothing for you. If your prompts are short to begin with, the saving is too small to be worth the effort of collecting. Caching shines on exactly one shape of workload, a large unchanging prefix reused heavily, and if yours has a different shape you should expect little from it.
I would rather say that plainly than have you bolt it onto everything, watch the bill not move, and wonder what you did wrong. It is a sharp tool for a specific job.
The wider lesson
Caching makes nothing free. You still pay real money, less of it, and only on the repeated portion of the prompt. Treat it as an honest discount on repetition rather than a loophole you found, and it will not disappoint you.
The lesson sitting above the feature is the one I keep relearning. Before optimizing anything clever, look hard at where the money actually goes, because the biggest waste is almost always something dumb that familiarity has made invisible. Mine was paying to re-read the same two pages ten thousand times a day. I write up more of these unglamorous operational lessons from running AI in production on [xavierfok.com](/).
Get new guides and videos first — join the Telegram channel.