The idea: look it up, then answer
A large language model knows what was in its training data and what is in the prompt, nothing else. It has never read your refund policy. Retrieval-augmented generation (RAG) fixes that without retraining: before the model answers, a search step finds the few pieces of your documents that matter for this question and pastes them into the prompt. The model then answers from that text and says where it found it.
The page has three panels: four small documents (a refund policy, shipping, warranty and a marketing page for a laptop), the vector index with one row per chunk, and the question side with the prompt and the answer. Pick a question and press Ask, change the chunking or top_k, or run a demo. Each answer is checked against the documents as they are now: green for correct, red for wrong and why.
Chunking: chunk_size and overlap
Whole documents are too long to paste into every prompt and too mixed to match one question, so they are cut into chunks. The page cuts by words with a sliding window: chunk_size words per chunk, and each window starts chunk_size − chunk_overlap words after the previous one. Real systems count tokens and often split on sentences or headings first, but the trade-off is the same:
- Small chunks are precise, but they can cut a fact in half and lose the words that say what it is about. With 8-word chunks, "e-books cannot be" and "refunded." end up in different chunks, so no chunk says it (Demo 4).
- Big chunks keep facts whole, but their vector is an average of many topics. A short, focused chunk can then outrank them: with 32-word chunks the one-line X1 marketing page beats the warranty paragraph for the warranty question.
- Overlap repeats the edge words in the next chunk, so a fact that falls on a boundary is still whole in one of them (Demo 5). It costs more chunks to store and search.
Embeddings and cosine similarity
An embedding is a list of numbers that stands for the meaning of a text. A real embedding model is a neural network trained so that texts with similar meaning get vectors that point in similar directions; it has hundreds or thousands of dimensions that no one labels. The page uses a toy model with 7 named dimensions (refund, shipping, warranty, time, cost, damage, digital), each counting that topic's words, so you can read the vectors. The same model must embed both the chunks and the question, or their vectors are not comparable.
How close two vectors are is measured by cosine similarity: the cosine of the angle between them, q·c / (|q| |c|). It is 1 when they point the same way, 0 when they share nothing. Length does not matter, so a long chunk is not favoured just for being long. In Demo 1 the question "How long do I have to return something?" becomes refund 1, time 1. The first refund chunk has refund 1, time 3: cosine 4 / (√2 · √10) = 0.89, the best match.
Retrieval: top_k
The index returns the top_k chunks with the highest similarity. Real vector databases (pgvector, FAISS, Pinecone and others) use approximate nearest-neighbour indexes such as HNSW so they do not have to compare against millions of rows, but the result is the same idea: the k closest vectors.
A small k keeps the prompt short and focused, but one off-topic chunk that happens to score high can push the right one out (Demo 2: the X1 marketing line "a tough laptop" scores 0.99 for the warranty question). A larger k makes it more likely the fact is in there (Demo 3) but adds noise and cost. Similarity search always returns k results, even when nothing is relevant (Demo 7: every score is 0.00), so many systems also set a minimum score. Production systems often add keyword search (BM25) next to the vectors and a reranker model that re-scores the top 20–50 before keeping a few.
Prompt assembly and citations
The prompt has three parts: instructions, the retrieved chunks, and the question. The instructions do two jobs. "Answer only from the context" keeps the model from mixing in what it half remembers, and "if it is not there, say so" lets it refuse instead of guessing. Tagging each chunk with its id lets the answer cite its source, so a reader can check it.
System: Answer only from the context. Cite the chunk id.
If the answer is not in the context, say you can't find it.
Context:
[c1] Since day one we want customers happy, so refunds are accepted within 30 days of …
[c2] 30 days of purchase. Items must be unused and in the original box. Digital goods …
Question: How long do I have to return something?
When RAG goes wrong
The model can only be as right as the chunks it is given. When they are wrong, it usually does not notice: it answers confidently from whatever context it got.
| symptom | cause | what helps |
|---|---|---|
| "I can't find that" although the docs say it | the right chunk is not in the top_k | bigger top_k, better chunking, hybrid search, a reranker |
| a confident wrong answer | an off-topic chunk scored higher (Demo 2), or the fact was cut in half (Demo 4) | overlap, chunking by sentence or section, a reranker, more specific instructions |
| an old answer | the document changed, the index did not (Demo 6) | re-index on every change |
| an invented answer | nothing relevant was retrieved and the model guessed | "say you can't find it", a minimum similarity score |
Keeping the index fresh
The index is a copy of the documents, made when they were indexed. In Demo 6 the refund window changes from 30 to 14 days. Until someone re-indexes, the search still finds the old chunk and the model, doing exactly what it was told, answers "30 days" and cites it. Real systems re-embed changed documents on every save or on a schedule, delete the chunks of removed documents, and store the document version with each chunk.
RAG, fine-tuning or a long context?
| RAG | fine-tuning | whole docs in a long context | |
|---|---|---|---|
| new facts | yes, re-index and done | poorly, and needs retraining | yes |
| cites sources | yes | no | possible |
| cost per question | a search + a short prompt | a short prompt | all the documents, every time |
| good for | large, changing knowledge | style, format, a skill | a few documents that fit |
To see what the model does with the prompt once it has it, see How a Large Language Model Works. An agent can also do retrieval itself, as a tool it calls with its own search query: see How an AI Agent Works.
See also: finding the top_k chunks is a nearest-neighbor search; K-Nearest Neighbors shows the same idea on a 2D plot.
See also 2D vectors and the dot product, where cosine similarity comes from: a·b / (|a||b|).
The vector index lookup is usually an HNSW graph search; see HNSW vector search.
Retrieval here is vectors only; for keyword BM25 vs vector search and hybrid fusion with RRF, see BM25 vs Vector Search vs Hybrid.
What the page leaves out
A learned embedding model (the page uses 7 hand-made dimensions), tokens instead of words, approximate nearest-neighbour indexes, keyword and hybrid search, rerankers, query rewriting, metadata filters, chunking by document structure, and the LLM itself: its answer here is scripted from the retrieved chunks, so every run is the same.