How I choose a local AI model: a method that takes an hour
# How I choose a local AI model: a method that takes an hour
There are hundreds of local models you could download, new ones land every week, and the leaderboards that rank them will mostly mislead you. People spend days chasing whatever tops some chart, install it, and find it wrong for the job they actually have. I pick models for tasks that run in my pipeline every day, and the picking stopped being trial and error once I turned it into a short, repeatable method. Here is that method, start to finish.
The leaderboard trap
Public rankings measure performance on standardized tests, broad academic style benchmarks. That says something about general capability and almost nothing about how a model handles your specific task with your specific style of input. A model that tops a reasoning chart can be mediocre at the plain summarizing you need, while a humbler model turns out perfect for it. I have seen both happen.
So treat the leaderboard as a rough starting filter at best. The only ranking that matters is how a model performs on your real work, and the whole method below exists to measure exactly that.
Step one: define the job, then rate its difficulty
Before looking at a single model, write down the job concretely. "A good model" is a wish. "Summarize articles into three bullet points" is a spec, and so is "classify incoming messages into categories" or "draft replies in a particular tone." The more specific the definition, the easier every later choice becomes.
Then rate the difficulty honestly. Ordinary work that a small model handles well sits in one class, and genuinely hard reasoning that needs something bigger sits in another. That single judgment sets which model sizes you should even consider, and it protects you in both directions: no giant model for a trivial job, and no undersized model doomed to fail at a hard one.
Step two: find your hardware ceiling
Your card's memory eliminates most candidates instantly, which is a gift. The rough arithmetic: about half a gigabyte of VRAM per billion parameters at four bit compression, plus a buffer for context. Run that math against your card and everything too big to fit well disappears from the list.
There is no point falling in love with a model that will spill onto the CPU and crawl. This step turns hundreds of options into a shortlist of models that actually run fast on your machine, before you have had to think hard about anything.
Step three: general or specialized
Some models are generalists, decent at a bit of everything. Others are tuned hard for one kind of work: coding, a specific language, following instructions with precision. If your task is narrow and well defined, a specialized model that matches it will often beat a larger generalist, because it was shaped for exactly that work. If your tasks are varied, a good generalist is the safer pick.
Match the model's specialty to the task's shape. A coding tuned model for code, an instruction tuned model for structured directions, a generalist for mixed everyday work.
Step four: trade size against speed
Within your hardware ceiling, bigger generally means more capable and slower, smaller means faster and weaker on hard cases. Pick based on the difficulty you rated in step one and the speed the job demands. A task running hundreds of times a day wants the smallest model that still produces good results. An occasional task where quality dominates and speed is irrelevant deserves the biggest model that fits.
One extra lever inside this trade: a bigger model at heavier compression often beats a smaller model at light compression for the same memory. Keep that swap in mind when the shortlist is tight.
Step five: build a tiny test set
This is the step almost nobody does, and it is worth more than everything else combined. Take five or ten real examples of your actual task. Your real inputs, never benchmark questions. For each one, write down what a good answer looks like. That is all.
That little set is your personal benchmark. It takes maybe fifteen minutes to assemble, and it converts model selection from a vibe into a measurement, because it tests the exact thing you care about and nothing else.
Step six: the runoff
Take the two or three models that fit your hardware and match your task type, and run every one through your test set. Compare the answers side by side against what you defined as good. A clear winner usually emerges fast, and it is frequently a model no leaderboard would have pointed you at. Sometimes the smaller, faster candidate proves plenty good and takes the job on speed. Sometimes one model simply gets your task in a way the others miss.
This head to head on your own examples is the heart of the method. Every earlier step existed to narrow the field down to a fair contest.
How people fumble the runoff
A few failure patterns show up during testing, all avoidable. Judging on one example is the first: models are inconsistent, and a single lucky or unlucky answer misleads you, so always run the full set. Ignoring consistency is the second. A model that is reliably good beats one that alternates between brilliant and broken, at least for anything you intend to automate. Third, weigh the speed you measured alongside the quality, because a slightly better answer at five times the latency is usually the wrong trade for high volume work. And fourth, write down which model won and why. A month later you will not remember, and the releases that appeared in between will tempt you to start over from nothing.
The most common way people sabotage the whole method is skipping step five because it feels like extra work. They narrow by hardware, pick a type, then choose the finalist on reputation. Then the model underperforms on the actual task and they conclude model selection is a lottery. The test set is the fifteen minutes that makes the other forty five worth anything.
Learn to read a model page
Model names look like noise and are actually compact spec sheets. The number is the parameter count, which drives capability and memory needs. Words like instruct or chat mean the model was tuned to follow directions. A named specialty, code or a particular language, tells you what it was shaped for. The compression label tells you the size and quality trade you are accepting. Reading those few signals lets you filter a long list down to the handful worth downloading, without downloading any of them.
One trap deserves its own warning: base models versus tuned variants. A model often ships in several flavors, and the base version is raw, without the tuning that makes a model respond helpfully to chat style requests. Grabbing the base by mistake and concluding the whole model is bad is a classic beginner error. For nearly every practical use, you want the variant labeled instruction tuned or chat tuned.
Obedience beats brilliance in production
For automated work, instruction following matters more than raw intelligence. Some models stick precisely to the format you asked for, every time. Others are smarter in the abstract and loose in practice, drifting from your format or appending things you never requested. When code has to parse the output, a reliably obedient model wins over a flaky genius every single time.
So when you test, score two things separately: whether the answer is good, and whether the model did exactly what you asked, in the shape you asked, consistently. That second score often decides the contest, and leaderboards do not measure it at all.
New releases, and when to switch
Once a model passes your test set and runs fast enough, stop. Use it. The field will keep producing celebrated new releases, and newer does not mean better for your particular narrow job. The time to revisit is when your needs change, when a task gets harder, or when a clearly meaningful model in your size class has emerged. Then the same little test set settles the question in minutes: run the candidate through it and switch only if it beats the incumbent on your work.
Two housekeeping notes from doing this repeatedly. Each downloaded model is several gigabytes, and a selection process leaves a folder of half tested candidates eating your disk, so delete the losers once you have a winner. Re downloading later is quick if you ever need one again. And accept that no single model has to win everything. One can be strong at your summarizing and weak at your classification while another is the reverse, and the practical answer is a small stable of specialists with each task routed to the model that earned it. Once a model is installed, routing a job to it is trivial.
The method, end to end
A real walkthrough, since the abstract steps land better with an example. The job: draft short polite replies to routine messages, automatically, dozens of times a day. That is ordinary work, so I look small. The card holds eight gigabytes, which rules out everything above the seven and eight billion parameter class. The task needs dependable instruction following and a steady tone, so instruction tuned candidates over raw generalists, giving a shortlist of three. I collect ten real messages, write what a good reply to each looks like, and run all three models through them. One of the three produces consistently on tone and format across all ten messages, so it takes the job, despite never having topped any chart I am aware of. I delete the other two and ship it. Total elapsed time, under an hour.
The lesson underneath the whole method is short: test on your own work, never on someone else's chart. The model that wins your little test set is the right model for you, wherever it ranks in public. More measure it yourself write ups like this at [xavierfok.com](/).
Get new guides and videos first — join the Telegram channel.