How AI transcription works, and where it predictably fails
# How AI transcription works, and where it predictably fails
AI transcription feels flawless right up to the moment it confidently types the wrong word. You feed in an audio file, a wall of text comes back, and it is close enough that you stop checking line by line. Then one day a proper noun comes out mangled, or a technical term becomes a different word that merely sounds similar, and you catch it only because something felt off on the read back. That is the moment you realize the tool has been a black box the whole time.
This piece opens the box. What the model actually does with sound, why the whole field improved so quickly, the five places it still fails on schedule, and the trick I use in my own pipeline that sidesteps the worst of it.
What the model actually hears
The common assumption is that the model listens for words the way you do, matching sounds against a dictionary. What actually happens is more interesting. The recording is taken a short chunk at a time and converted into a grid of features: which frequencies are present at each instant, how they shift over time, where the energy spikes. It is the kind of information you would see on a spectrogram. The raw waveform never reaches the model's reasoning at all. The structured representation is what gets read.
From there the clever part begins. The model considers a whole window of audio at once and asks which sequence of words most plausibly produced that pattern of features, drawing on everything it learned about how likely one word is to follow another. It predicts the sentence as a whole. Context does as much work as sound, which is exactly why the errors it makes are fluent instead of random.
Why it stopped being a toy
Ten years ago automatic transcription was a demo you tried once. Today millions of people trust it without a second thought. The change came from three shifts landing at roughly the same time, each multiplying the others.
The architecture changed first. Older systems chained separate modules, each doing one narrow job, and errors compounded at every joint. Modern systems train a single neural network end to end, straight from audio features to text, and the joints disappeared along with the modules.
The data exploded second. Modern models train on tens of thousands of hours of real human speech in many languages and accents, recorded under all sorts of conditions and covering all sorts of topics. Variety in training is what buys resilience to conditions the model has never met.
And compute got cheap enough, third, to train and run these networks at scale. Any one shift alone would have moved accuracy a little. All three together made it feel like a leap. A modern transcription model has been exposed to more hours of speech than a person could hear in several lifetimes.
Failure one: names
The model predicts words by probability, and rare words carry low probability. A common first name appears millions of times in training data. An unusual surname, a small company, a product, a town spelled nothing like it sounds: these may barely appear at all. Faced with audio it cannot match confidently, the model reaches for the nearest word it knows, and the nearest word by sound can be a completely different word.
The dangerous part is how plausible the error looks. The output is never gibberish. It is a real word, phonetically close, sitting comfortably in the surrounding sentence, and you read straight past it. Names deserve a deliberate, separate check on every transcript that matters. Some tools accept a glossary of expected names, which genuinely helps; without one, the model has no way to know a rare name even exists.
Failure two: jargon
Domain terms fail for the same underlying reason. A drug name spoken by a doctor, a library name said aloud by a developer, a tool a security researcher mentions by its proper name: all low probability, all high risk. The wider the gap between the model's training data and your domain, the more corrections you will make.
Here is the heuristic I actually use. If the recording would sound at home on a mainstream podcast, the model will probably nail it. If it sounds like specialists talking to specialists, budget for corrections, and expect them to cluster in precisely the passages where accuracy matters most. One odd term in an ordinary conversation is a small risk. An hour of dense specialist language is a different job entirely.
Failure three: accents
This one is uncomfortable because the failure is uneven across people. Models train on whatever speech data can be gathered, and accents that are underrepresented in that data get transcribed less accurately. That is a data problem rather than a design choice, and the industry has been improving on it, but it remains real today.
The practical response is cheap: test before you commit. Run one minute of a speaker's audio and read the output carefully. That single minute tells you whether you are facing a light skim or a heavy correction pass, before you have transcribed an hour and discovered the problem at the end.
Failure four: overlapping speakers
Most models assume one voice at a time. Two people talk over each other, even briefly, and the model drops a voice, garbles the overlap, or merges both speakers into one nonsensical line. Separate diarization tools exist to split speakers before transcription, and they help, at the price of an extra step and their own fresh mistakes.
The best fix happens at recording time. Give speakers a clear turn structure. Even a short pause between speakers transforms what the model can do with the audio. You cannot always control the recording. When you can, it pays off all the way down the pipeline.
Failure five: noise
The most straightforward one. Models learned from clean speech, and every step away from clean speech degrades the prediction. A fan, a keyboard, a loud room, a quiet recording boosted afterwards, a cheap compressed microphone: each pushes the audio further from what the model expects. Modern models absorb mild noise well, then hit a floor where accuracy drops off sharply.
People keep hunting for software fixes to what is really an audio problem. Noise reduction helps at the margins. A better microphone in a quieter room helps more than every setting combined, and once the signal is lost at capture, no cleanup recovers it.
The trick my own pipeline uses
Everything I publish is built from a written script, which puts me in an unusual position: I almost never need the model to figure out what was said. I already know. What I need is timing, the exact moment each word lands in the audio, so captions appear in sync.
Predicting unknown words is the hard problem. Aligning known words to the audio they came from is a far easier one. So instead of asking for free transcription, I hand the model the script text and ask it to align that text against the recording. The model solves only the timing problem. Spelling is already settled, so names and technical terms come out exactly as written, every time, and the whole class of plausible wrong word errors disappears.
This generalizes well beyond video captions. Any time you hold a transcript you trust and only need timestamps, feed the text in rather than asking the model to guess it. You are deleting the hardest part of the problem instead of solving it.
How much to trust it, exactly
Within its home conditions, trust it for a first pass: clean audio, one speaker, mainstream vocabulary. It will save you real time and the error rate stays low. Outside those conditions the error rate climbs, and it climbs in the five predictable places this article just walked through.
That predictability is the useful part. You never need to audit a transcript evenly. Read the names. Read the stretches thick with jargon. Read the noisy patches and the moments where people spoke over each other. Skim the rest. Targeted checking keeps most of the time savings and catches the errors where they statistically live.
A tool with known failure modes is a tool you can actually rely on. This one earns a first pass on almost everything I record, and a careful second look exactly where the map says to give one.
More plain explanations of the AI plumbing I run every day are at [xavierfok.com](/).
Get new guides and videos first — join the Telegram channel.