Is CPU only AI usable? I timed it on a normal desktop
# Is CPU only AI usable? I timed it on a normal desktop
Nearly every guide to running AI models locally opens the same way: you need a GPU. If you do not have one, the tone drops into sympathy, as if the race ended before you laced your shoes. The machine I wanted to test on has no discrete graphics card at all. It is an ordinary desktop, the kind most people already own. So before spending real money on a problem I was not even sure I had, I asked the cheaper question: what actually happens when a language model runs on the CPU alone. I timed it on my own hardware, across three model sizes, with a stopwatch rather than a forum post as the source.
The answer has more texture than either camp admits, and the things it depends on are exactly what this post walks through.
Why the GPU advice exists
The GPU obsession comes down to memory and math. A language model is a giant pile of numbers called weights, and producing a single word means pushing your text through all of them: billions of small multiplications per word of output. A graphics card runs thousands of those multiplications at the same time, in parallel, which happens to be the exact shape of this workload. A CPU runs a handful of streams very quickly, one after another.
On raw throughput the GPU wins and it is not close. That part of the standard advice is completely true, and I am not arguing with it. What the benchmarks skip is how the gap actually feels at the keyboard for specific jobs, and that feel is what I wanted to measure.
What CPU only actually means
Run a model without a GPU and the weights load into ordinary system RAM instead of the fast video memory on a graphics card. Your processor cores then grind out every token.
Before speed even enters the picture there is a gate. The model has to fit in RAM. Too big and it either refuses to run or spills onto disk and crawls to uselessness. So the first check is always memory: how much RAM you have, and how small the model gets after compression. That compression is called quantization, and it turned out to be the most important lever in the whole test. Hold that thought.
The machine and the models
The test box is an ordinary desktop with a recent six core processor and thirty two gigabytes of RAM, no discrete graphics card doing any of the work. The software was a small local runner that loads a model and chats from the command line, the kind of thing anyone can install in minutes without being a programmer.
I picked three models to cover the range. A tiny one around three billion parameters. A middle one around eight billion. A large one around seventy billion, chosen mostly to watch it break. Each was quantized down so it would actually fit in memory. Then came the unglamorous part most demos skip: the same handful of everyday questions to each model, with every phase of the response timed.
The numbers
Two measurements matter here, and they capture two different kinds of pain. The first is how long you wait before the first word appears on screen. The second is how fast words arrive after that, in tokens per second, a token being roughly a piece of a word.
The three billion model put its first word on screen in about a second and then streamed slightly faster than I could comfortably read along. The eight billion model took a few seconds to start and settled into the pace of a slow but steady typist. Usable, just never snappy. The seventy billion model took the better part of a minute to produce its first word, and after that each word crawled out so slowly the whole exercise became almost funny to watch. It worked, technically. The answer that eventually assembled itself was correct and thoughtful. As a tool with a human waiting on the other end, it was useless.
That spread is the whole lesson in one picture. The big model was right too slowly to matter, while the tiny model most people dismiss as a toy was the one that felt good to use. On a CPU, the small model that answers now beats the big model that answers eventually, and speed you can feel is worth more than cleverness you wait a full minute to receive.
Where CPU only is genuinely fine
Plenty of real work never involves a person drumming their fingers at a screen. Batch work has a different shape: hand the machine a stack of inputs, walk away, come back later to finished results. Small and medium models on a CPU handle that shape perfectly well. Cleaning up messy text. Summarizing long documents. Sorting a pile of email into folders, or pulling structured fields out of a mess of unstructured notes.
A second win has nothing to do with speed. Everything runs on your own machine, with no API key, no per request meter, and none of your data ever leaving the building. For private documents you would never paste into someone else's cloud, that fact alone can justify a slower model.
Where it falls apart
Three places, all predictable.
Sheer size is the first. The models that genuinely reason better are simply too slow on a CPU for interactive use, as the seventy billion test showed.
Long context is the second and the sneakiest. The more text you stuff into the prompt, the more math there is per generated word, and that cost stacks up frighteningly fast on a CPU. A model that felt fine on a short question can grind nearly to a halt the moment you feed it a long document.
Real time is the third. A voice assistant that answers the instant you stop talking, or a coding helper keeping up as you type, needs a snappy back and forth that a CPU cannot deliver no matter how you tune it. For those jobs the GPU really is the right tool, and pretending otherwise would be lying to you.
Quantization is the lever that makes it possible
One idea from this test outranks the rest. The model is a pile of numbers, and quantization stores each number in fewer bits, shrinking the whole pile dramatically, often to a quarter of its original size or less. That shrinking is the entire difference between fitting in your RAM and never running at all.
The tradeoff is real and gentler than it sounds. Moderate quantization costs a sliver of quality you will struggle to notice. Aggressive quantization, the kind used to cram a huge model into too little memory, leaves you something that runs but answers like it took a knock to the head. The dial trades a little brains for a lot of room, and most of the actual skill here is knowing when to stop turning it. My rule now: find the biggest model that fits comfortably at a sensible quantization level, and stop there, rather than chasing a bigger model squeezed until it can barely think straight.
The honest cost accounting
The open models I used are free to download and carry no per request fee, which is a genuine pleasure compared to metered pricing. The comforting lies start right after that sentence, so let me head them off. The hardware was not free. The electricity is not free either, and a CPU pinned at full load for a long batch job pulls real power and noticeably warms the room. Cheaper than most people assume, yes. Zero, no.
For the genuinely hard tasks, the big hosted models also remain clearly ahead, and nothing in this test replaces them. The point is knowing which tool fits which job.
So should you bother
If your whole use case is fast interactive chat with a large capable model, save up for the graphics card. That really is the right tool for that job.
If you own an ordinary computer with a decent amount of RAM, and the real work is summarizing, sorting, and tidying text quietly in the background, then CPU only is a perfectly good setup rather than a sad compromise. It costs nothing extra and keeps every byte of your data at home. The useful question was never whether you can run AI without a GPU at all. It is which model, at which size, for which job, and once you frame it that way, a plain CPU does far more than the guides ever give it credit for.
I keep poking at the limits of local AI on modest hardware, and [the write-ups live on xavierfok.com](/).
Get new guides and videos first — join the Telegram channel.