How RAG finds the answer

A model can explain what a refund policy is. That does not mean it knows your refund policy. If you ask it a question about a document it has never seen, a confident answer is not much use.

RAG gives the model something to work with. The full name is retrieval-augmented generation. You search your documents first, then send the relevant passages along with the question. The model can use those passages when it answers.

I made this example around a fictional coffee company called Peak Bean. It has four documents and 25 short passages. Start with the same question answered with and without a policy in front of the model.

Compare two answers

A customer asks something only Peak Bean's policy can answer. Press both buttons.

Customer“Can I return my espresso machine after 40 days?”

Without a policy

With retrieved policy

The first answer invents a 60-day return window. The second uses the policy and points to the relevant passages. These are scripted examples, but the difference is what I want you to notice. Can you check the answer against a source?

Split the documents into chunks

A long document can contain several unrelated topics. Retrieving a smaller passage usually makes it easier to find the part that answers a question. These passages are called chunks.

The boundaries matter. A chunk that is too short can lose half a rule. A chunk that is too long can carry a lot of text you do not need. Move the slider and watch what happens to the refund deadline.

Split a policy

What might happen if the chunks get too small?

Below is Peak Bean's refund policy. Drag the slider to re-slice it. Watch the highlighted fact.

target chunk size 120 characters

– chunks · avg – chars

At small sizes, “full refund within 30 days of delivery” gets split across chunks. Larger chunks keep it together, but also include other policies. There is no single size that works for every document. Sentence boundaries and overlap are worth testing.

This slider counts characters and keeps whole words. Real chunkers can use tokens, sentences or document sections.

Find related passages

An embedding model turns text into a list of numbers. You can compare those numbers to estimate how closely a passage relates to a question. That lets a search find related wording even when the exact words differ.

The passages are drawn as dots so you can see that comparison. Nearby dots get ranked higher. This map is an illustration of the idea, not the output of an embedding model.

Explore the search space

The phrase “money back” appears in zero Peak Bean documents. If you ask “where's my money back?”, what does the search find?

Pick a question or type your own. Drag the star to move the query. You can also focus it and use the arrow keys.

“Money back” is mapped close to the refund passages even though those exact words are absent. Real embeddings can connect related wording in a similar way. They can also rank a passage highly when it is related but does not answer the question.

view this map as a table

#	chunk	document	similarity

The map uses two dimensions so it fits on a page. Real embedding models use many more, with the number and similarity measure depending on the model.

Choose how much to retrieve

Once the passages are ranked, you still need to choose how many to include. Here that number is k. Too few can leave out a useful detail. Too many can add irrelevant passages and use up space in the prompt.

The ranking below follows the question you chose on the map. Change k and look at the passages that make the cut.

Choose the top results

Does increasing k always improve the answer?

These results follow your question on the map. Change k to include more or fewer passages.

chunks retrieved top-k of 25

Choose a question on the map to see its ranked results.

context window

0 / 170 tokens

Watch which passages enter as k increases. More retrieved text does not always help. The useful setting depends on the question and the documents. The token count here is only a rough estimate.

Put the passages in the prompt

The retrieved text goes into the prompt with the question. This does not train the model on your documents. It gives the model information to use for this answer.

Asking it to use only those passages helps set the task, but the instruction is not a guarantee. You still need to check whether the answer follows the sources.

Build the prompt

The prompt below uses your question and the passages selected by k. Flip the switch.

retrieval ON

system: answer ONLY from the passages below. cite them.

⟨ no passages retrieved yet, visit the map ⟩

user: …

Example answersources included

Waiting for a question…

The question stays the same. The available context changes. Try k = 1, then increase it and check whether the example has the sources it needs.

Where retrieval goes wrong

A related passage answers a different question

The shipping-fee refund sounds relevant to a refund question. It is still the wrong policy if the customer wants to return a machine.

Try a different question

A chunk loses part of the rule

If the deadline and refund condition end up in separate chunks, one search result may leave out the detail that matters.

Change the chunk size

The query is poorly represented

This demo only recognises a small vocabulary. An unknown word gives it no useful signal. Real embedding models are much broader, but unfamiliar language can still hurt retrieval.

Try your own wording

What I would check first

Did the retrieved passages contain the information needed to answer the question?
Did chunking preserve the conditions, dates and exceptions?
Did increasing k add useful context or just more text?
Does each claim in the answer follow from its cited passage?

If an answer is wrong, I would look at what was retrieved before changing the prompt. A model cannot reliably fill in a policy detail that never reached it.

About this example

The company and policies are fictional. Chunking uses character counts, the map uses hand-placed coordinates and a small word list, and the answers are scripted. No model API runs on this page. In a real system, chunk boundaries, embedding models and retrieval settings need to be tested on the questions people actually ask.