Every language model has a peculiar kind of amnesia. It has read half the internet, yet it has never seen your refund policy, your product docs, or the email your support team sent last Tuesday. Ask it about those and it does the one thing it's trained to do: produce a plausible-sounding answer anyway.
Retrieval-augmented generation — RAG — is the standard cure. Before the model answers, a search system fetches the few paragraphs of your data that matter, and staples them into the prompt. The model stops guessing and starts reading.
To make this concrete, this whole essay runs on a toy knowledge base: the internal docs of an imaginary company, Peak Bean Coffee Co. — four documents, twenty-five paragraphs. Let's watch what happens without RAG first.
A customer asks something only Peak Bean's policy can answer. Press both buttons.
Customer“Can I return my espresso machine after 40 days?”
§1First, the library gets sliced
Search engines don't retrieve whole documents — a 30-page PDF would drown the model. So every document is cut into chunks: bite-sized passages that live or die on their own. Chunk size is the first knob every RAG engineer turns, and it's a genuine trade-off, not a formality.
Below is Peak Bean's refund policy. Drag the slider to re-slice it. Watch the highlighted fact.
§2Meaning becomes geometry
Here's the beautiful part. Each chunk is fed through an embedding model, which converts its meaning into a list of numbers — a coordinate. Chunks about similar things land near each other, regardless of the words they use. Your question gets converted into the same space, and “search” collapses into something almost childish: find the nearest dots.
Below is Peak Bean's entire knowledge base as a map. It's a toy 2-D version of a real embedding space, but the behavior is honest.
Pick a question (or type your own), then drag the ✦ star around the map and watch retrieval react.
view this map as a table
| # | chunk | document | similarity |
|---|
§3Choosing how much to grab
Nearest-first gives us a ranking. But how many chunks do we actually hand to the model? That number is called k, and it's the second knob. Too few and the answer is missing its evidence. Too many and the good passage drowns in noise — while every extra chunk burns context-window budget.
This ranking is live-linked to Exhibit C — whatever you asked up there is ranked down here. Step k up and down.
§4The staple gun
The last step is almost anticlimactic. The retrieved chunks are pasted into the prompt above the user's question, with one instruction: answer from these passages only. The model doesn't “learn” your data — it just reads what retrieval put in front of it, that one time. Flip it off and watch what the model falls back on.
The prompt below is assembled live from your Exhibit C question and Exhibit D's k. Flip the switch.
§5Where it quietly breaks
Everything above also explains the classic RAG failures — you've already caused all three of them on this page:
The convincing distractor
“Delayed parcels qualify for a shipping-fee refund” sits between two clusters — it's about refunds and shipping. Ask a money question and it sneaks into the top-k, half-relevant, wholly misleading.
→ you saw it drift in Exhibit CThe severed fact
When chunking split “full refund within 30 days,” no retrievable passage contained the rule. The model can only read what survives the slicer.
→ you did this in Exhibit BThe out-of-vocabulary query
Ask about something the embedding model barely understands and your query lands in no-man's-land — retrieval returns the nearest wrong things with full confidence.
→ type gibberish in Exhibit C§6What to keep
- RAG = slice → embed → nearest-k → staple into the prompt. Four steps, two knobs.
- Embeddings match meaning, not keywords — that's the superpower and the failure mode.
- Chunk size and k are trade-offs you tune with evals, not constants you copy from a tutorial.
- A grounded answer is only as good as what retrieval put on the desk — measure retrieval first when RAG misbehaves.
If you can drag a star around a map, you now understand the architecture behind most production AI assistants. The rest is engineering: real embedding models, vector indexes, rerankers, and evals — the parts I build for a living.