My local LLM stack: what runs on the box and what still goes to an API
# My local LLM stack: what runs on the box and what still goes to an API
People hear that I run language models at home and picture a rack of servers behind glass. The reality is a desktop in a spare room, one consumer graphics card from a previous generation, a normal wall socket. That is all of it. It handles a surprising amount of real work every week. What follows is the stack as it actually exists, with the limits left in, because the limits are what make the rest believable.
Three parts, one of them boring
Strip the terminology away and a local setup is three things. A runtime, a model, and some way to send text in and get text back.
The runtime is the piece nobody talks about at parties. It loads the model into graphics card memory, listens on a local address, and does the computation when a request arrives. Once it is running, the experience feels close to calling a remote API, except every byte stays on the machine. I run one of the popular open source runtimes, and my only strong opinion here is a dull one: pick something stable and actively maintained. That single choice matters more than most of the decisions people agonize over.
The interface layer is whatever suits you. A chat window, a script, a hook into other tools. Nothing about it is special, and swapping it later costs nothing.
The models I keep in rotation
My rotation is small, and I stay deliberately vague about version numbers because they go stale faster than anything written about them. The shape of the choices has held steady for a long time.
For general text work I run a mid size model, somewhere in the seven to fourteen billion parameter range. Models that size fit comfortably in the memory I have, answer at a usable speed, and get through everyday language tasks without embarrassing themselves. For code I switch to a model trained heavily on programming, because code models and chat models have noticeably different strengths and the difference shows up within minutes of use.
The largest open models never make the list. On my card they would run only with compromises severe enough to defeat the point, and I would rather run a smaller model well than a large one badly. The memory ceiling decides, honestly and without appeal.
The hardware, stated plainly
One consumer graphics card, a few generations old, with a hard limit on memory. That memory number governs everything downstream: which models load, at what quantization, with how much room left over for context. Every other spec is a footnote beside it. Nothing else comes close.
The upside of old consumer hardware deserves stating just as plainly. A used card from a few generations back costs little, draws household power, and lives in an ordinary desktop case. Nobody walking past the room would guess the machine does AI work. When someone insists you need exotic hardware for local models, they are usually pricing a job this machine was never going to be given. What you actually need is an honest match between the hardware you can afford and the tasks you plan to hand it.
Quantization, in plain terms
Modest hardware works at all because of quantization, which gets discussed like a dark art and deserves better.
A model is a huge collection of stored numbers, each recorded at some precision. Full precision is accurate and heavy. Quantization rounds the numbers down to a lower precision, which shrinks the model and speeds it up, in exchange for a small drop in output quality.
The right mental picture is compressing a photograph. Light compression is nearly invisible. Heavy compression turns the image to mud. Models degrade along the same curve, so I run moderately quantized versions, where the size saving is large and the quality loss takes effort to notice. That one choice is what lets genuinely capable models fit into memory that would never hold them at full precision.
Treat quantization as the normal way to run models at home rather than a compromise of last resort. Chasing full precision on consumer hardware buys almost nothing and costs you most of your options.
What the box is genuinely good at
Privacy comes first. When I am working with my own notes and drafts, nothing leaves the machine, and every question about where content goes or who might train on it simply stops existing. I never wonder whether a half formed idea, a draft agreement, or a private note ended up in some retention queue on a server I will never see, because none of it ever left the room. Some of my work is sensitive enough that this alone justifies the setup.
Cheap bulk work comes second. When a job means running the same small task many hundreds of times, the marginal cost of one more local request rounds to zero. No token meter, no monthly bill creeping upward while a batch runs overnight. For high volume repetitive text processing, the economics tilt hard toward local.
Constant availability closes the list. At two in the morning, mid project, the box is simply there. No outage page, no queue, no hourly cap. It just answers.
Where it honestly loses
The open models I run trail the frontier models badly on hard reasoning, and the gap widens as problems get harder. Hand the same difficult multi step puzzle to my local model and to a top paid API, and the API wins, often by a wide margin. The gap is large. Any article that pretends otherwise is selling something.
Long context is the second loss. The big hosted models hold enormous amounts of text in one request. Mine hold far less, so long documents get chunked and stitched back together, which adds friction and the occasional seam error.
The third loss is time itself. A local model is a snapshot, frozen at its training cutoff, while hosted models get refreshed. Ask my local stack about recent events and the answer arrives confident and out of date.
The routing rules I actually use
Local and hosted stopped being rivals for me long ago. Tasks get routed, and the rules are short.
To the local box: anything private, always. Repetitive volume work, because the cost argument settles itself: extracting fields from documents, reformatting text, classifying short content into a few categories. Throwaway drafts and quick lookups, where a good enough answer now beats a great answer later. And experiments, since iterating against a free endpoint lets me think out loud without watching a meter.
To the paid API: work where quality is the whole point and a wrong answer costs money or reputation. Anything that needs genuinely current knowledge. Documents too long to chunk sensibly. Rare one off tasks, where spinning up a local model would be ceremony for a single call.
Written out, the rules look almost too simple, and that is the point. Once you accept what each side is honestly good at, routing takes no thought at all.
Advice for a first setup
Start with the smallest model that does your job, then scale up only when you hit a reason. The instinct is to load the biggest thing that fits. Resist it. A small model gets the full stack working, shows you what real outputs look like, and fails cheaply while you are still learning.
Budget an evening and some reading for the setup, and treat that as normal. The first request that processes entirely on your own machine is a quietly satisfying moment. Everything after it goes faster, and the second install takes a fraction of the time.
Most of all, decide what local is for in your workflow before you judge it. Measured against the best hosted model on the hardest task, every home setup disappoints. Measured against the actual jobs I route to it, at the cost and privacy profile I want, mine earns its place every week.
If you want to follow along as I keep running and improving this stack, there is more at [xavierfok.com](/).
Get new guides and videos first — join the Telegram channel.