Running AI on an eight year old GPU: what actually works
# Running AI on an eight year old GPU: what actually works
Eleven gigabytes of video memory, and about nine and a half of it are actually yours.
That is the first number worth internalising if you want to do AI work on old hardware. The card in my workstation is a GTX 1080 Ti, 250 watts, silicon from 2017. It encodes every video I publish, transcribes every voiceover, and keeps a small language model resident most of the day for tagging. Two years of using it as a working machine has left me with a fairly unromantic view of what it can do.
Most of what gets called an old hardware limit is a memory estimate that forgot about context, or a library default that assumed newer silicon.
Budget the memory before you pick the model
Three things compete for the same pool, and only one shows up on the model card.
The weights are the part everyone counts. At four bit quantisation a seven or eight billion parameter model lands around four and a half gigabytes. Fourteen billion fits but leaves you nothing to work with. Thirty billion is out at any quantisation worth using, since the weights alone are closing on twenty gigabytes.
The key value cache is the part that ruins your plan. Every token the model has already seen keeps an entry, and that entry lives in card memory alongside the weights. The per token cost depends entirely on the model's attention design. A recent model with grouped query attention might spend fifty odd kilobytes per token. An older design without it can spend ten times that for the identical context length. Two models both labelled seven billion can have completely different memory bills once you put a document into them.
Then the runtime allocates its own working buffers, which are not small, and Windows itself is holding a gigabyte or so for the desktop and whatever browser you left open.
So I plan against nine and a half gigabytes and size the model to leave headroom for the context I actually use.
Two different speeds, and only one of them gets benchmarked
This is the part I think is genuinely misunderstood, and it explains why the card feels fine in one situation and unusable in another.
Generating tokens is bound by memory bandwidth. The model streams its active weights through the chip once per token, so what limits you is how fast memory feeds the cores. This card manages 484 gigabytes a second against roughly a terabyte on a current flagship consumer card. Call it two times behind.
Reading your prompt is bound by compute. That stage processes the whole prompt in parallel, which is dense matrix work, which is exactly what modern cards have dedicated units for and Pascal has none of. No tensor cores at all. There the gap opens up to something like five or ten times.
Tokens per second on a short prompt, the number everyone publishes, is the measurement an old card looks best on. Time to first token on a long prompt, which nobody publishes, is where it falls over.
Practically: short prompt and long answer is the shape to aim for. Drafting, rewriting, expanding notes. Long prompt and short answer is the shape to avoid, and a coding agent that rereads half a repository every turn is the worst case this hardware has. Those go to a hosted model and I do not feel bad about it.
When I do have to send something long, I keep the long prefix byte identical between calls so the runtime can reuse the cache instead of recomputing it. That habit has bought me more real speed than any quantisation choice.
Before you blame the card
Four things have wasted my time on this machine, and all four look identical from the outside. Everything is just slow, and nothing errors.
The driver spilling into system memory. Ask for more card memory than exists on Windows and the driver does not refuse you, it silently pages across the PCIe bus at a tenth of the speed. Turn off Sysmem Fallback Policy in the Nvidia control panel and an oversized model gives you a clean out of memory error in one second, instead of a lie that takes an hour to spot.
The wrong device. This box has Intel graphics in the CPU, and to a lot of tooling that is device zero while the real card is device one. An upscaler I set up spent an afternoon on the integrated chip while the 1080 Ti sat idle.
Half precision. More on this below.
Something else already resident. A local model server that stays loaded after you stop using it will happily hold six gigabytes while your image job asks for eight.
Half precision is a trap on this generation
This is the mistake I actually made, and it ran for weeks before I caught it.
I was transcribing voiceovers with Whisper, on CUDA, requesting half precision, because that is what every guide tells you to do. It worked, and it was barely quicker than the CPU, and I assumed that was simply what an eight year old card gave you.
It was not. Pascal runs half precision at one sixty fourth the rate of full precision, which makes the fast path I thought I had selected the slowest path on the chip. The library did the sensible thing, fell back to the processor, and never said a word about it. I had been benchmarking my CPU and recording the result as a GPU number.
Switching to the eight bit integer path, which this generation genuinely does accelerate, fixed it in one config line. The general lesson is worth more than the fix: on older hardware, defaults can select the exact path your chip is worst at, and nothing will tell you.
When CPU offload is fine and when it is a waste
Every local runtime lets you split a model, keeping some layers on the card and running the rest on the processor. It gets sold as the way to run a model that does not fit.
The arithmetic is less friendly than that. The card reads its own memory at 484 gigabytes a second. A normal desktop reads system memory at around forty. Any layer you push off the card is now working against memory an order of magnitude slower, and every token waits for it. Move ten percent of a model and you can lose half your throughput.
Offload is fine when nobody is watching. Overnight batches, scheduled queues, anything that only has to be done by morning. There it costs nothing that matters.
Offload is a waste when the model wants twice the memory you have. At that point you are running on the CPU with extra steps, and a smaller model that genuinely fits will give you a better answer sooner. The exception is mixture of experts models, where only a fraction of the weights are active per token, so the part you exiled to system memory often is not the part being read.
The electricity is real, and hosted models are cheaper
Under sustained load the card pulls 250 watts and the whole machine sits near 370. Electricity here runs about thirty cents a unit, so that is roughly eleven cents an hour. In Singapore you pay for each of those watts twice, because all of it becomes heat in a room an air conditioner is being paid to cool.
An hour of steady generation from a small model is somewhere near a hundred thousand tokens, so my marginal cost is ten to fifteen cents per hundred thousand tokens before the card itself is counted. Hosted models in the same eight billion parameter class are cheaper than that, comfortably. For pushing a lot of text through a small model, the API wins on cost and it is not close.
Local buys me other things. No rate limit, so a job that wants forty thousand calls at three in the morning just makes them. No vendor in the loop, so a pipeline I wrote two years ago still runs untouched. Data that never leaves the room. And a card already sitting idle, which makes a four minute job cost a fraction of a cent.
What it still earns its place doing
The video encoder handles every final render I publish and has aged better than any other block on the board. It does not do the newest codec and its quality per bit trails current cards, but it is fast and it costs me no attention. Transcription on the eight bit path is quick enough that I stopped measuring. Thumbnail generation runs about forty seconds an image, which would be intolerable interactively and is fine as a batch of eight. Upscaling runs happily. The resident small model does titles, tags, chapter marks and classification all day, and that is the highest value job on the machine by a distance.
The last problem I hit had nothing to do with any of this. Two jobs wanting the card at the same moment gives you an intermittent crash that depends on what happened to be loaded a minute earlier. I tried tuning around it with shorter keep alive timers and smaller image batches. What fixed it was a queue with a single lock: one GPU job at a time, everything else waits. Contention was the bottleneck, and no setting solves contention.
Would I buy this card today? No. Spending real money now I would shop memory per dollar first and speed second. But between a new card with eight gigabytes and a used one with sixteen I take the used one every time, even when it benchmarks worse, because a model that does not fit does not run at any speed. And if you already own something old, run it hard for a month before you spend anything. The list of things it genuinely cannot do is shorter than the internet has told you.
Get new guides and videos first — join the Telegram channel.