Why local models slow down and the levers that fix it
# Why local models slow down and the levers that fix it
A local model that was quick yesterday crawls today, with no obvious change in between. Or it answers short questions instantly and grinds to a halt the moment you paste in a long document. These slowdowns feel random. They almost never are. They come from a small set of predictable causes, and once you know them you can control your speed instead of being at its mercy. I tune models on an old home GPU until they genuinely keep up with daily production work, and these are the levers, ordered by how much each one actually matters.
Context is the usual suspect
Context is all the text the model is working with right now: your prompt, the document you pasted, the whole conversation so far. The cost of processing grows with how much of it you supply, and the growth is unkind. The model has to relate every part of the text to every other part, so the cross referencing work grows faster than the text itself. Double the context and you more than double the work.
That is why the same model feels fast on a quick question and painfully slow on a big input. Nothing broke. You handed it far more to think about. When speed suddenly drops, the first question to ask is almost always whether the context got long. This growth curve is fundamental to how these models work, so the practical answer is to be deliberate about how much text you feed in rather than waiting for faster models to save you.
Long chats slow down because of the KV cache
There is a second, slower kind of degradation that shows up over the course of a conversation. As the model generates, it builds a working memory of everything said so far, called the KV cache, so it does not recompute the whole exchange for every new word. That cache lives in VRAM, and it grows as the conversation does. More memory consumed, more time per response, and a chat that started snappy turns sluggish an hour in.
The fix is almost embarrassing: start a new conversation. The cache resets and the speed comes back. When a long back and forth has gone syrupy, a fresh chat is usually the fastest repair available.
Lever one: cap the context length
Most local AI tools let you set a maximum context size, and most people leave it maxed out. Dialing it down does two useful things at once. It limits how much memory the cache can eat, which keeps the model sitting comfortably on the card, and it puts a hard ceiling on how slow any single request can get.
The trade is that the model holds less text at a time. For most everyday tasks that costs you nothing, because most everyday tasks never needed an enormous window in the first place. Set the cap to what your work actually requires and you collect a speed win for free.
Lever two: make sure the model actually fits
If a model is chronically slow, slow from the very first request rather than slow over time, the likely cause is that it does not truly fit in your card's memory. When the model plus its context plus overhead exceeds VRAM, the extra spills onto the CPU and system memory, and performance falls off a cliff. Ten to twenty times slower is the normal penalty, which turns a usable tool into a useless one.
The fixes are all forms of making it fit: a smaller model, a more compressed version of the same model, or a tighter context cap to claw back room. Fitting comfortably beats fitting barely.
Loading delays, and warm versus cold
The first request after a long pause takes ages, then everything after it is quick. That delay is the model loading into GPU memory. Tools unload idle models to free the card, so the next request pays a reload. It reads as slowness and is really just the model waking up.
If the reloads annoy you, most tools can keep a model resident in memory. It stays ready at the price of holding your VRAM the whole time, which is worth it if one model does most of your work. The same logic matters for automation. A job that calls the model in bursts with long gaps keeps paying the reload cost on every burst, while a steady workload keeps the model warm for free. It also matters when you measure: benchmark the warm steady state, because a cold first run includes the load time and will convince you your setup is slower than it really is.
Skip batching at home
The advice to batch requests for throughput comes from data centers, where a server with abundant memory genuinely does process many requests more efficiently together. A home GPU running one model rarely has the spare memory to batch anything meaningful. Firing several heavy requests at once just makes them fight over the same limited card, and everything finishes later.
On home hardware the honest advice is the opposite: run tasks one at a time, let each finish, and total throughput comes out ahead.
Cap the output too
The model produces its answer one word at a time, and every word costs time. A request that produces a long answer is slower than one that produces a short answer for that reason alone. If you need a brief reply, say so, and set a hard cap on how much the model may generate.
I have sped up automated tasks dramatically with only this change. I was letting a model ramble for a paragraph when the job needed one sentence. Controlling output length is a direct, reliable lever that costs nothing and is forgotten constantly.
Work out which half is slow
Two different speeds get lumped together as slow, and they have different fixes. There is the wait before the first word appears, and there is how fast words stream once they start. The first is dominated by processing your input, so it grows with prompt length. A giant pasted document means a long silence before anything shows up, even when the eventual answer is short. The streaming speed after that is about generating each new word, and it stays fairly steady regardless of input size.
So diagnose by halves. Slow to start but streams fine: your input is too long, trim it. Starts fast but streams slowly: the model is not fitting well, shrink it. Knowing which half is broken tells you which lever to pull.
Prompt length deserves special suspicion here, because it is the lever people least expect. Long instruction blocks, oversized examples, and pages of pasted background all get processed on every single request. Tighten the instructions, keep only the examples that earn their place, and cut the background to what the task needs. I have made automations noticeably faster purely by editing the prompt down, and a lean prompt often produces better answers as well, since the model is no longer wading through clutter to find the actual question.
Three more places to look
Compression level is a speed lever, and people treat it purely as a size lever. A more heavily compressed version of a model moves less data per calculation, so it can run faster while also freeing memory, at the cost of a little quality. Past a point the quality loss stops being worth it, but trying one step more compression is a legitimate way to speed up a model that fits yet feels sluggish.
Whatever else is using the GPU is competing with your model. A game, video editing software, even browser tabs with heavy graphics will steal capacity from the card, and the model gets blamed for the resulting crawl. Checking what else is on the GPU takes seconds and people skip it. Close the heavy programs while the model runs, or accept that both run slower together.
And track one number: tokens per second, the rate at which the model generates words. Most tools will show it. Once you know your normal figure for a given model, any problem announces itself as a drop in that number, and you go investigate instead of guessing. A model that usually streams at reading pace suddenly crawling at a fraction of it means something concrete: the model spilled, the context ballooned, or something else grabbed the card. I glance at that number constantly.
Fix the configuration before buying hardware
When things are slow, the easy story is that you need a better card. Sometimes, eventually, that becomes true. But I have watched people spend real money on an upgrade and find the new card exactly as slow, because the actual problem was an oversized context or a spilling model, and the new hardware inherited it. Work the free levers first: cap the context, confirm the fit, trim the prompt, cap the output, close the competitors. A surprising number of people finish that list and discover their old card was fast all along.
My worked example: an automated classifier of short text snippets, running hundreds of times a day, felt slower than it should. The prompt turned out to be carrying a long block of instructions and examples pasted in months earlier, far beyond what the task needed, so I cut it to the essentials. Then I capped the output, since a classification needs a word or two and the model was producing sentences. Then I confirmed the model sat comfortably in VRAM. Same card, no purchases, and the task ran several times faster. That is the whole philosophy in one anecdote.
The order to check things
When something is slow, work down this list. First, check the context length, the most common cause and the biggest easy win. Second, confirm the model fits the card and is not partly running on the CPU. Third, if a long conversation has gone sluggish, start a fresh one to clear the cache. Fourth, cap the output so the model generates only what you need. Fifth, keep tasks serial on home hardware instead of batching. That sequence resolves the large majority of local slowdowns without spending a dollar.
The mindset underneath it all: fast local AI is mostly about managing context, and hardly about the card. I run genuinely fast models on hardware that is years old, because I am disciplined about how much text goes in and how much comes out. That discipline is worth more than an upgrade for most people, and it is free. More unglamorous knobs like these at [xavierfok.com](/).
Get new guides and videos first — join the Telegram channel.