I built a content pipeline that runs itself, and here is what broke
# I built a content pipeline that runs itself, and here is what broke
A render ran for fifty minutes one night. The process stayed up, the log kept growing, the dashboard walked the job from queued to in progress to complete. The file it produced opened fine and had exactly the duration I expected. It was the title card, held for eleven minutes, and then black.
Every monitor I had said that job was healthy. Every one of them was answering a question I did not care about.
I have been running an end to end publishing pipeline for a couple of years now. Scripts get written, a text to speech step voices them, a renderer cuts that against stock and filmed footage with captions burned in, and a scheduler releases finished pairs on a cadence. The architecture took a weekend. Everything since has been failure modes, and the failure modes are the only part of this worth writing down.
The pipeline itself is trivial
Five stages in a line: write, voice, render, verify, publish, each one reading a file and writing a file.
None of the pain has ever come from a stage. It comes from the space between stages, where two things have to agree about what already happened.
Shared scratch state ruined a week
I started serial, got impatient, and set the renderer to run four jobs at once because the machine looked idle between encodes.
Every render writes intermediate files: a loudness normalised audio track, a concat list of clip segments, a caption file, some generated diagram frames. All of that landed in one scratch folder, named for the step that produced it. Named by step, never by job.
So job two overwrote job one's audio while job one was still assembling its timeline. Both finished. Nothing errored and no log line hinted at it. What came out was a finished video with somebody else's narration on it, and I found that four days later by watching one back before it went out.
My first fix was the obvious wrong one. I put a job id into every temp filename and considered it handled. That killed the collisions I already knew about and did nothing about a step that swept the scratch folder clean at the start of every run, or two encodes fighting for one graphics card. I got a subtler corruption further down instead.
What actually fixed it was going back to strictly serial, with one queue, one worker, and a lock file that a second worker respects. The rule I added afterwards matters more than the lock: the queue is the only legal entry point. Kicking off a render by hand, outside the queue, is itself a bug now, because it has no idea what the worker is holding.
Total throughput barely changed. The encode was already saturating the machine, so four jobs at once were four jobs taking turns badly. Parallelism pays when the parallel parts do not share the resource that limits you, and mine shared every one of them.
Liveness tells you nothing
That render taught me the most expensive lesson in the system: a process can be extremely alive and completely useless.
Is it running, did it exit zero, has the log moved. Those are cheap and they answer nothing about whether the run produced work. Every check now asserts on the artefact instead:
- Decode the file all the way through, not just open it. A truncated render opens perfectly and dies at minute nine.
- Confirm an audio stream exists and its mean volume is something other than silence. Silent output is the single most common way this pipeline has lied to me.
- Compare duration against what the script predicted. Length here comes entirely from narration word count, so 2,400 words arriving as a six minute video means something got dropped.
- Check the first frame. The render puts a title card on the front and a later step trims it; a title card still present means the trim never ran, and a file five seconds shorter than expected means it ran twice.
- Watch ten seconds of it, at three points. I resisted this for about a year because it felt like conceding the automation had failed. It catches things no assertion of mine ever has.
That list took two years to accumulate and every item on it exists because something shipped that should not have.
The dry run that marked everything as shipped
Here is the one I got badly wrong, with nobody to hand it to.
Before pointing the scheduler at anything live I ran it with a dry run flag, which is the responsible thing to do. The flag worked exactly as designed: nothing uploaded, nothing published, nothing sent.
What it did do was write to the ledger. The send call sat inside the dry run check. The line marking an item as shipped sat outside it, at the bottom of the loop, where it read as bookkeeping rather than as an action. I had been through that function a dozen times without once noticing those two lines were on opposite sides of a fence.
An entire queue got marked as published in about eleven seconds, with nothing published. Since published items are skipped forever, that queue was now permanently empty of work that had never happened.
The cost was a day. No disaster and no angry customer, just a day spent diffing what the ledger claimed against what was live and rebuilding the gap by hand. The ledger was the only record of what had been queued, so the thing I had to repair was also my only map of it.
The fix I settled on is structural rather than careful. Test mode no longer means an if statement at each call site. It means the job gets handed a writer object that refuses every mutation and logs what it would have done. You cannot forget to wrap a call that has nothing to write through.
The job that only fails on a timer
This class wastes more hours per unit of real complexity than anything else here.
Everything works when I run it by hand, because running it by hand is how I test it. Then it goes on a schedule, and the schedule runs it as a different user, from a different working directory, with a different environment, and with my credential vault locked.
The daft version: a relative path. In my shell the working directory is the project folder, so `output/` resolved where I expected. On the timer it was somewhere under the system directory, so the same string resolved elsewhere and the code politely created the folder and wrote there for two weeks.
The expensive version: secrets come out of a password manager through a command line tool. In my session that vault is unlocked, so the fetch returns instantly. On a timer the same command waited two full minutes for a master password nobody was going to type, then gave up. The job reported success anyway, because the call sat inside a try block that swallowed the failure into a log I was not reading.
Every scheduled job now prints a header before doing anything: which user it is running as, which directory it thinks it is in, and whether each credential it needs resolved to a value. Four lines at the top of a log. It tells me within seconds whether I am looking at a bug or at an environment, and it has saved me more debugging time than any tool I have added since.
Four systems, four opinions
The thing I actually believe after all this: generation was never the expensive part. It is a fixed lump of compute that mostly just happens, and it gets cheaper every year.
The expensive part is that several systems each hold an opinion about what is done and they disagree.
The folder on disk has an opinion, because the file exists. The state file has an opinion, because an api call returned success at some point. The platform has an opinion, and it is the only one that decides whether a video is actually public. The article queue has its own opinion about whether the companion piece went out alongside the video it belongs to.
Any two of those can be individually correct and still contradict each other. The file is on disk and the upload half finished. The platform says published and the state file never got written, because the process died between the api call and the flush.
A reconciler runs once a day and compares all of them. It is the piece I trust most and it took the longest to get right, mostly because of one rule I arrived at late: for any given fact, exactly one system is allowed to be the truth. Disk decides rendered. The platform decides published. The state file is a cache and it is allowed to be wrong.
Before that rule I had reconcilers that repaired state by writing to whichever side looked stale, which is how you get two systems flipping each other back and forth every morning while you work out which one started it.
It also throws false alarms constantly, because a video uploaded ten minutes ago looks missing while the platform is still processing it. So the reconciler reports and never acts. I read the report. Keeping detection and repair on opposite sides of a human being is the only reason I let it near anything.
The part I will argue with people about
Most automation advice says add monitoring. I think monitoring is where the majority of the effort gets wasted, because nearly all of it watches the wrong noun. Processes are easy to watch and almost meaningless. Artefacts are annoying to watch and tell you everything.
Six months of a dashboard full of green taught me less than ten seconds of watching an output file.
And the title of this piece is doing some work, so let me be straight. The pipeline runs unattended. It does not run unwatched. I read the reconciler report daily and still watch part of every video before it goes public, because the day I stop is the day something ships with the wrong narration on it and a viewer tells me before I notice.
Get new guides and videos first — join the Telegram channel.