I ran a local model on a mini PC
# I ran a local model on a mini PC
There is a small silver box on the shelf above my desk, about the size of a paperback, bought for less than a decent phone. It has no graphics card. Right now it is running a language model, answering questions entirely on its own, with nothing going out over the internet. When I first got that working I sat and stared at it for a while, because it does feel like a minor magic trick. A lot of what gets written about local AI is breathless, and almost none of it covers the boring truth, so this is the honest version: what the box is, how fast it really runs, what I use it for, and where it falls short.
Why run a model locally at all
The fair first question is why bother, when a browser tab gets you a smarter answer in a second, for free or nearly free. My reasons have nothing to do with speed.
Privacy comes first. Nothing I type into the box leaves the room. No account, no logging, no terms of service that shift under me next year. For the half formed notes and personal material I throw at it, that matters.
Ownership comes second. The model on that box will run identically in five years, whatever gets deprecated, repriced, or shut down in the meantime. A login is never mine in that way.
Then there is availability. No rate limits, no daily quota, no message telling me I have hit a cap. It sits on the shelf waiting, and I can hammer it as hard as I want for nothing extra. And a last reason that sounds soft but has paid me back over and over: running the thing on my own hardware means I actually understand it. A box I can open and poke at teaches me more than a service behind a login ever has.
None of those reasons claims the model is better. A frontier cloud model out-thinks this little box on almost every hard question, and on the truly hard ones it is not close. Local AI trades away raw capability in exchange for control. For some jobs that trade is excellent, for others it is terrible, and most of the skill is learning to tell which is which.
What is actually in the box
The hardware is where people picture the wrong thing. Nobody should imagine a tower with an expensive card glowing inside it. This is a mini PC, the kind of tiny computer usually bought to plug into a television. It has a laptop class processor, no separate graphics card, and a pool of memory shared between the processor and its built in graphics. That shared memory is the whole story, because a language model has to fit inside it to run at all. Mine has thirty two gigabytes, which sounds generous until you remember the model wants to load itself entirely into that space and still leave room for the machine to do anything else.
There is no screaming fan and no special cooling. The box draws about as much power as a couple of light bulbs. For years the accepted story said this class of workload needed serious, expensive, power hungry hardware, and that story is quietly expiring. The chips got better while the models got smaller, and the two met somewhere in the middle, on a shelf, in a box the size of a sandwich. Most people still carry the old picture in their heads, and the gap between what they imagine and what is now possible is a large part of why I wanted to write this down.
Picking a model that fits
You cannot take the largest, smartest model available and cram it onto hardware like this. It will refuse to fit, or it will fit and crawl. The real skill is matching the model to the box, and two numbers govern the match. Parameter count is roughly the size of the model's brain. Bigger generally means smarter, and always means heavier. Quantization is the squashing step, storing each parameter in less space at the cost of a small amount of precision, and in practice the answers barely change.
For this machine the sweet spot is a model in the seven to eight billion parameter range, quantized down. It loads in a few seconds, leaves the rest of the machine usable, and is smart enough to earn its keep. Out of stubbornness I once squeezed a much larger model on. It ran, technically, at about one word every two seconds, roughly the pace of a person typing with one finger, and I gave up on it within the hour.
How fast it really is
Speed is the question everyone actually cares about, and almost nobody answers it with numbers. The measure is tokens per second. A token is a chunk of a word, produced one at a time, so the figure is effectively a reading speed. This box, with that seven billion parameter model, produces somewhere around ten to fifteen tokens per second. In human terms that is a little faster than comfortable reading pace, so answers stream out at about the speed of a person reading aloud to me.
A big cloud model would have finished before this one gets going, and for most of what I do the difference costs me nothing, because I am reading along as the text appears anyway. Where it hurts is long work. Ask the box to chew through a large document and those tokens add up to a real minute or two of waiting. So I have learned to match the job to the machine. Quick questions stay local. Grinding tasks go somewhere with more muscle. Knowing where that line sits is most of the game.
What I actually use it for
The honest answer is the quiet, private, repetitive jobs. Cleaning up my own messy notes. Turning a wall of bullet points into a few readable sentences. Summarizing something long that I do not want to read in full. Drafting the boring first version of an email so I have something to push against instead of a blank page. Sorting and labeling little piles of text.
None of that is glamorous, and that is exactly the point. These jobs need a competent assistant rather than a genius, and the box is always on, never charges me, and never sends my notes anywhere. I also wire it into small scripts on my own machine, so a workflow can do a little thinking without any network call. A script that needs to pull a clean title out of a messy line, or decide which bucket a note belongs in, just asks the local model and gets an answer in a moment. Because a call costs nothing, I never stop to weigh whether a small task is worth the spend, and that changes how freely I build things in a way that is hard to appreciate until you have lived with it.
The limits, plainly
I promised the honest version, so here it is. A small local model is visibly weaker on anything that needs real reasoning or deep knowledge. It makes more mistakes. It states wrong things with complete confidence. It loses the thread on long, complicated requests in a way the big cloud models mostly no longer do, and its memory for context is much shorter, so you cannot pile material on and expect it to hold everything at once.
So I never treat its output as the final word on anything that matters. First drafts, grunt work, and facts I can check myself all suit it well. Anything where being wrong is expensive gets verified by hand or sent to a bigger tool. The trap with local AI is falling for the privacy and the ownership so hard that you start trusting the answers more than they deserve. I have caught myself doing exactly that, leaning on an answer because I liked that it came from my own box, then finding it was simply wrong. Fondness has no effect on the model's accuracy. Treat it as the useful, limited tool it is, and it will serve for years.
The real cost
People always want the numbers, so here they are. The box was a one time purchase, under the price of a mid range phone, and that is the entire hardware bill. The model cost nothing to download and nothing to run, with no subscription hiding behind it. The only ongoing cost is electricity, and at a couple of light bulbs of draw, leaving it on around the clock comes to a few dollars a year. After the up front spend, running it sits close to free, and I use that word carefully because I usually distrust it.
The honest framing cuts the other way too. The cloud is also cheap. A few dollars a month buys a far more capable model with none of this fuss, so the box was never a money play. The payoff is that my data stays home and the tool cannot be taken away from me. I also just enjoy running it. If your only goal is escaping a subscription, the math probably does not work, and I would rather tell you that plainly than sell you a fantasy.
Should you try it
Would I tell everyone to go buy a box and do this? No. If typing into a browser chat suits you and nothing above itched, this is a hobby project solving a problem you do not have. If you like owning your tools, feel a little uneasy sending everything to someone else's server, and enjoy understanding the machine on your desk, it is one of the most satisfying weekends you can spend right now.
Running a real model on hardware you own has never been cheaper or easier. No fortune required, no giant graphics card. A small box and an afternoon. And there is real value in knowing, first hand, that this technology does not have to live in someone else's cloud. It can sit on a shelf above a desk and be entirely yours. I keep writing up local AI experiments like this one, boring parts included, on [xavierfok.com](/).
Get new guides and videos first — join the Telegram channel.