Xavier Fok
← All writing

· 6 min read

Running a small model as a filter in front of a big one

How a model cascade works: using a small, fast model to triage requests before they ever reach an expensive one, and where that pattern breaks.

What a model cascade actually is

A model cascade is just a pipeline with a bouncer at the door. A small, fast, cheap model looks at the request first. If it can handle the request on its own, or it can confidently say “this doesn’t need more,” it stops there. If not, the request gets passed up to a bigger, slower, more expensive model that actually does the hard reasoning.

That’s it. There’s no trick to the term. It’s a routing decision made by a model instead of an if-statement, sitting in front of another model.

I run this pattern daily across my own automation. Not because it’s clever, but because most of what an AI pipeline touches doesn’t need a large model at all. Feeding every request into the biggest model available is the equivalent of calling in a senior engineer to check if a light switch is plugged in.

Why bother filtering before you escalate

Three reasons come up over and over when I build these, and they’re the same three reasons that show up in any queueing system, not just AI ones.

Latency. A small model can return a classification or a short answer far faster than a large one can generate a full reasoned response. If most of your traffic is simple, you want most of your traffic answered fast, and only the hard tail waiting on the slow path.

Cost. Larger models cost more per call, whether that’s API spend or the compute you’re burning locally. A cascade means you’re only paying the big-model price for the requests that actually need big-model reasoning. Everything else gets handled at the small-model rate.

Load and throughput. If you’re running models locally, whether that’s on your own hardware or a rented GPU, a big model has a hard ceiling on how many requests it can serve concurrently. A small model in front of it means fewer requests ever reach that ceiling, so the system as a whole handles more volume without falling over.

None of that is hype. It’s the same logic as putting a cache in front of a database, or a load balancer in front of an application server. AI models aren’t special here. They’re just another layer with a cost and a latency profile, and cascades are how you manage that profile.

The mechanics: a gate, not a vote

The important design decision in a cascade isn’t which models you pick. It’s how the small model decides to hand off.

There are basically two ways to do it:

  1. Confidence thresholding. The small model produces an answer along with some signal of how sure it is, whether that’s a raw probability, a self-reported confidence score, or a simpler proxy like “did it produce structured output that parses cleanly.” Below a threshold, you escalate.

  2. Classification routing. The small model doesn’t try to answer the actual task at all. It just decides which bucket the request falls into: simple, needs context, needs multi-step reasoning, out of scope. The bucket determines whether it goes to the big model, a different tool entirely, or gets rejected outright.

I favor the second approach in my own pipelines, because confidence thresholding on a small model is genuinely hard to trust. Small models are often confidently wrong in exactly the cases where you most need them to hesitate. A dedicated classification pass, where the small model’s only job is “does this look like something I can handle,” tends to be a more honest signal than asking it to grade its own homework on an answer it just generated.

Either way, the cascade only works if the handoff decision is cheap and fast compared to the work it’s gating. If your filter model takes nearly as long as the big model would have, you haven’t built a cascade, you’ve built an extra hop.

Where I use this pattern

The clearest example in my own setup is content and metadata generation for video and article pipelines. A lot of what an automation pipeline does before it ever gets to “write this section” or “reason about this argument” is small, mechanical classification: is this topic a duplicate of something already published, does this input match an expected format, does this row of a queue actually have enough source material to act on. None of that needs a large model’s reasoning. A small model, or in a lot of cases a much cheaper classifier, handles it in a fraction of the time and gets escalated only when the answer is genuinely ambiguous.

The same shape shows up when I wire agents into tools through MCP. A tool-calling agent gets a lot of incoming requests that are really just “is this the right tool for this input at all.” Running a small filtering step before the full agent loop kicks in keeps the expensive, multi-step reasoning limited to cases that actually need multiple steps. Most requests don’t.

This isn’t unique to my setup. It’s the same shape as any triage system: cheap first pass, expensive second pass only when the first pass says so.

The failure mode nobody warns you about

The problem with cascades isn’t that they don’t work. It’s that when they fail, they fail quietly.

A big model that gets a hard question and struggles usually produces something that looks uncertain, or a bad answer that a downstream check can catch. A small model that misclassifies a hard question as easy doesn’t struggle. It just answers, confidently, and wrong, and the request never reaches the model that could have handled it. Nothing upstream sees an error. The pipeline reports success.

This is the actual engineering cost of a cascade. You’re not just picking a threshold once. You’re accepting that some fraction of requests will get silently mishandled by the filter, and you need a way to catch that beyond just watching for outright failures, because there won’t be any.

In practice this means building in spot checks: routing a sample of “handled by the small model” traffic to the big model anyway, and comparing outputs. If they diverge often, your threshold or your classification prompt is wrong, and you find that out from the comparison, not from an error log.

Building the threshold without guessing

The tempting shortcut is to pick a threshold that “feels right” and move on. It doesn’t hold up. What actually works is treating the threshold as a variable you tune against real traffic, not a constant you set once.

Concretely: log every decision the small model makes, along with what would have happened if the request had gone to the big model instead, at least for a sample. That sample is what tells you whether your filter is too aggressive (sending too much to the expensive path, which defeats the point) or too loose (letting through requests the small model genuinely can’t handle).

This is slower to set up than just wiring two models together and calling it done. But a cascade without that feedback loop isn’t actually saving you anything reliable. It’s just moving the failure from “expensive but correct” to “cheap but occasionally silently wrong,” and you won’t know the difference until something downstream breaks in a way that’s hard to trace back.

What this is not

A model cascade doesn’t make a small model smarter. It doesn’t let a 1B-parameter model reason like a much larger one. It’s a routing layer, not a capability upgrade. The small model in the cascade is never doing the hard work. Its entire job is deciding whether hard work is needed.

It’s also not a way to avoid paying for the big model when the task genuinely needs it. If most of your traffic is hard, a cascade buys you very little, because most requests will escalate anyway and you’ve just added a filtering step on top. This pattern earns its keep specifically when the bulk of real traffic is simple and only a minority is genuinely hard. Measure that before you build one, not after.

And it’s not a substitute for deciding what a task actually requires. If you don’t know whether your traffic is mostly simple or mostly hard, that’s the first thing to find out, before you spend time tuning a cascade you didn’t need.

If you want to see how this pattern shows up in the actual pipelines I run, day to day, take a look around the site.

Get new guides and videos first — join the Telegram channel.

Keep reading