The spreadsheet that outlived three automations
The pattern I kept seeing
I've rebuilt the same tracking system three times. A queue of items that need processing, a status column, a place to note what went wrong. Each time I built it as a proper service: a database table, an API, sometimes a small web UI so I could glance at progress without opening a terminal.
Each time, within a few months, I stopped using the service and went back to a CSV.
Not because the service was broken. Because the spreadsheet never needed anything from me. It didn't need a migration when I added a column. It didn't need a deploy when the logic changed. It didn't have a schema that drifted out of sync with the code reading it. I could open it, see the whole state of the system in one screen, edit a cell by hand when something went wrong, and close it. The automations built around it came and went. The file stayed.
What a flat file workflow actually means
A flat file workflow is just this: your source of truth for state lives in a plain file, usually CSV or TSV, that any tool can read and write without a client library, a connection string, or a running process. No server has to be up for the data to exist. No API has to respond correctly for you to see what's in it.
This is not a rejection of databases. I run Postgres in production for things that need concurrent writes, relational integrity, or query performance at scale. A flat file workflow is for the other category of problem: a queue, a checklist, a batch job's input list. Things a single person or a single script works through, where the value of the format is that you can open it in anything.
The core property that makes this work is that a CSV is legible without tooling. Open it in a text editor and you see the actual data, not a serialized blob. Open it in a spreadsheet app and you get sorting and filtering for free. Grep it from a shell script. Load it into a pandas dataframe in three lines. Every one of those paths works on the same file with zero translation layer.
Why the automations kept failing to replace it
Each automation I built added a layer between me and the data. The first was a small Flask app with a SQLite table behind it. It worked, but every time I wanted to change what a row looked like, I was writing a migration, restarting the process, and re-checking that nothing else broke.
The second was an agent-driven pipeline that wrote status updates to a database directly. This one failed differently. Because the agent was the only thing writing to the table, I couldn't just glance at what was going on. I had to write a query, or build a small dashboard, just to answer "what's stuck." The information was there, but it wasn't in front of me.
The third was closer to right: a Postgres table with a thin dashboard on top, similar to the pipeline flow dashboard I run today for video rendering. That one survived longer because it solved a real problem the flat file couldn't: multiple processes reading and writing the same rows at once without racing each other. But even there, I still export to CSV constantly when I want to eyeball something, hand a slice of it to another script, or check a single row's history without touching the running system.
The common failure mode across all three wasn't the technology. It was that I built infrastructure for a problem that didn't need infrastructure yet. A queue with a hundred rows that one person or one script works through sequentially doesn't need row-level locking or a query planner. It needs to be readable.
When the spreadsheet actually breaks down
I'm not going to pretend a CSV scales forever, because it doesn't, and pretending otherwise is how you end up debugging file corruption at 2am.
It breaks down the moment more than one writer touches it concurrently. Two processes reading a CSV, updating a row, and writing the whole file back will race, and one of them will lose its write silently. I've hit this exact failure with a render queue that got double-processed because two scripts pulled from the same file without a lock. If you have concurrent writers, you need either file locking discipline that most teams don't actually maintain, or a real database with transactions.
It also breaks down on volume. A CSV with a few thousand rows opens instantly in anything. A CSV with a few hundred thousand rows starts to choke tools that load the whole file into memory, and any operation that used to be "open the file" becomes "write a script to filter the file first."
And it breaks down when you need relationships between records. A flat file is one table. The moment you need to join a queue item against a customer record and a pricing table, you're either duplicating data across files or building the join logic yourself every time you read it. That's a database's job, not a CSV's.
Where AI agents fit into this, honestly
The reason this matters more now than it did five years ago is that AI agents are good at reading and writing flat files in a way they're much less reliable at with a live database. An agent can open a CSV, understand the columns from the header row, and make a sensible edit without you giving it a schema, a connection string, or write permissions to a production table. That's a real advantage when you're wiring an agent into a task and don't want to hand it database credentials for something it's going to touch three times a week.
I use this pattern for authoring queues and topic inventories: a flat file an agent can read to know what's pending, and write to when it finishes a batch. No API layer between the agent and the state. If the agent does something wrong, the fix is opening the file and editing a cell, not writing a rollback migration.
This doesn't mean an agent should be trusted with a flat file the same way you'd trust a validated form. A CSV has no schema enforcement, so nothing stops an agent from writing a malformed row, a wrong date format, or a duplicate ID. If you're letting an agent write to a file that other automation reads from unattended, you still need something checking the output before it flows downstream. The flat file didn't survive because it's more capable than a database. It survived because it's harder to break in ways that stop me from seeing what's actually there.
The actual decision
The question I ask now, before building any tracking system, isn't "what's the right technology." It's "who and what writes to this, and how often." One writer, sequential access, low volume: flat file, every time. Multiple concurrent writers, or real relational queries, or enough volume that a text editor chokes: that's when a database earns its complexity.
Most of the internal tooling I build sits in the first category longer than I expect it to. The spreadsheet outlived three automations because the automations were solving a problem the spreadsheet didn't have yet.
If you want to see how this plays out in an actual production setup, including where I did eventually move off flat files and why, the rest of the writing on automation and AI agents is on the [home page](/).
Get new guides and videos first — join the Telegram channel.