RAG explained by building one over my own notes
# RAG explained by building one over my own notes
The model I use every day has no idea my notes exist. The document I wrote last week, the research I filed months ago, the half-formed ideas sitting in a folder on this machine: none of that was in its training data. Ask it about my own work and it either guesses or admits it has nothing.
So I built a retrieval augmented generation system over the notes folder. Partly to fix the problem, partly because building one is the fastest way to understand what the term really means. This is the full walkthrough, all five steps, plus the failure modes I hit and the point at which I would tell you to skip the whole idea.
The name is scarier than the idea
Retrieval augmented generation gives a model temporary access to material it was never trained on. The model stays untouched. You do no retraining and change no weights. When a question comes in, you search your own data first, pull the most relevant passages, and place them in the prompt beside the question. Then you tell the model to answer from what it just read.
That is the entire trick. Retrieval is the search step. Augmented describes the context you added. Generation is the model writing the answer, which is the only step the model handles.
Why I skipped fine-tuning
The obvious alternative is training a model on the notes directly. I ruled that out early, for reasons that apply to most personal knowledge bases.
Fine-tuning means running a training job, which costs real compute and real time and leaves you with a new model to serve. Add one document tomorrow and you are queueing another run. The knowledge also gets baked in anonymously: a fine-tuned model usually cannot tell you which sentence of your notes an answer came from.
Retrieval keeps everything as plain documents. Update a note, rebuild the index, and the cost rounds to zero. Every answer traces back to the exact passages you handed over, because you chose them yourself. For data that changes and grows, retrieval wins by a wide margin. Fine-tuning earns its keep when you want to change a model's style or behaviour. Keeping a model current on facts is a job it does badly.
Step one, cut the notes into chunks
A model has a context window, a hard cap on how much text it can read at once, so the whole notes collection cannot ride along on every prompt. You split the documents into passages instead. Mine landed around two to four hundred words each, with a little overlap between neighbours so a sentence never gets cut in half and stranded.
Chunking sounds trivial. It caused more subtle problems than any other step. Chunks that run long waste context space on filler. Chunks cut too short stop making sense on their own. The rule that worked for me was one coherent thought per chunk, one argument or one explanation, rather than any fixed word count. The right size depends on your documents, and you will probably have to experiment.
Step two, turn the chunks into embeddings
Keyword search cannot find a note you phrased differently two years ago. Meaning-based search can, and embeddings are how you get it. Run a piece of text through an embedding model and you get back a long list of numbers, a point in a high dimensional space. Passages with similar meaning land near each other in that space. Passages about unrelated things land far apart.
Every chunk goes through the embedding model once, and the stored results become your index. You can pay a hosted API a small metered fee per call, or run a smaller open source embedding model locally if everything has to stay on your machine. I used the hosted route, because the index only rebuilds when the notes change, so the calls are rare and the cost stays small.
Step three, store the vectors somewhere
The stored embeddings need a home that supports one specific query: given a new vector, find the vectors closest to it. Vector databases exist for exactly this. For a personal notes collection you do not need one.
I keep mine in a flat file on disk and load it into memory at query time. That handles thousands of chunks without complaint. Somewhere around hundreds of thousands of chunks, or multiple users querying at once, a real vector database starts earning its place. Below that, the flat file has fewer moving parts and fewer ways to break. The storage layer is a lookup table. The interesting part happens at query time.
Step four, the query
Say I ask what I wrote about shrinking Docker images. The question gets embedded with the same model the chunks went through, which gives me a vector for the meaning of the question. A nearest neighbour search against the index pulls the five or ten chunks whose vectors sit closest. Those are the passages most likely to hold the answer.
The whole search takes milliseconds once the index is in memory. And the results come from my own notes, ranked by how closely their meaning matches the question. No model has been asked anything yet. This is still just a very good search.
Step five, hand the context to the model
The retrieved chunks go into the prompt with the question. The shape is plain: here are some notes that may be relevant, here is the question, answer from the notes. The model reads the passages and writes an answer grounded in them rather than in training memory.
Because I know exactly which passages went in, I can show my sources. This answer came from these three passages in these two documents. Fine-tuning almost never gives you that, and for knowledge work it matters a lot.
Where it breaks
RAG is search stapled to a reading model, and the answers are only as good as the search. Retrieve the wrong passages and the model writes a confident answer from wrong context, with no idea anything is off. It will also sometimes ignore the context entirely when the question matches no passage well, or when its prior training disagrees with your notes.
Four failure modes cover most of what I hit. Bad chunking is the most common root cause: split one thought across two chunks and neither half scores high enough to be retrieved. Retrieval mismatch is second, where the question is phrased far enough from the note, especially around jargon, abbreviations, or personal shorthand, that the right chunk misses the top results. Third is a stale index, when notes changed and the embeddings were never rebuilt, so every search runs against an old snapshot of the data. Fourth, and subtlest, the model occasionally answers as if it never read a perfectly good retrieved passage, especially a short or ambiguous one.
None of these are fatal. They are things you test for instead of assuming away.
How I tested it
I picked twenty questions I already knew the answers to, questions where I knew a relevant note existed. Each one went through the pipeline, and I checked two things: did the right chunks appear in the top five, and did the final answer match what I had actually written.
Almost every failure traced upstream. The chunking or the retrieval was at fault; the generation step rarely was. That matches the general pattern with these systems. When a RAG build gives bad answers, the search is usually the broken part, and the model gets the blame.
When to skip all of this
If your data fits in one prompt, paste it in. Ten pages of notes and a modern context window need no retrieval layer at all, and adding one buys you an index to keep fresh plus every failure mode listed above.
Retrieval pays off when the collection is too large to paste, keeps growing without a ceiling, and needs answers that cite the exact passages behind them. Below that threshold, simpler wins. A big context window and a paste operation is a perfectly good architecture for a lot of real use cases.
What I would tell a first-time builder
Spend more time on chunking than feels reasonable. Then test retrieval before touching anything else: for your hardest questions, does the right chunk even reach the top results? If it misses, no prompt wording will rescue the answer.
Leave the storage layer boring. A flat file or a sqlite table covers most personal knowledge bases, and complexity added now is maintenance owed forever. I got a working system over my own notes with five plain steps, and the code was never the hard part. The hard part was honest testing, and admitting the cases where the whole thing was more trouble than pasting the notes into a prompt.
If you want more walkthroughs of things I have actually built and run, there are more at [xavierfok.com](/).
Get new guides and videos first — join the Telegram channel.