What Is RAG? Retrieval-Augmented Generation Explained

Plain-English definition

Retrieval-augmented generation (RAG) answers a question with a language model in two steps. First it retrieves the most relevant pieces of text from a knowledge source, such as documents, wikis, tickets or manuals, then it generates an answer using both the question and that retrieved text. The model reads the evidence you hand it, instead of relying only on what it memorised during training.

The approach was formalised in a 2020 paper by Lewis et al., which paired a pretrained sequence-to-sequence model with a dense vector index of Wikipedia. The authors describe this as combining parametric memory, meaning what the model learned in training, with non-parametric memory, meaning the searchable index. They report setting the state of the art on three open-domain question answering tasks.

Why it matters

Language models are trained once on a snapshot of text, so anything that happened after training is invisible to the model, and so is anything private to your organisation. Those are exactly the questions businesses need answered: our policies, our product documentation, this week's data.

RAG solves this without retraining, because updating the knowledge means updating documents rather than the model. It also produces answers that can cite their sources, which makes outputs checkable in a way that pure generation cannot offer.

How it works

A typical RAG pipeline has four stages.

1. Indexing. Documents are split into chunks, converted into embeddings that represent meaning as numbers, and stored in a searchable index, often a vector database.
2. Retrieval. When a question arrives it is embedded the same way, and the index returns the chunks whose meaning sits closest to the question.
3. Augmentation. The retrieved chunks are placed into the model's prompt, inside its context window, alongside the question and the instructions.
4. Generation. The model writes an answer grounded in the supplied chunks, ideally citing which ones it used.

A minimal RAG loop you can run

Most explanations of RAG stop at the diagram, but this one runs. It needs no libraries and no API keys, because the embedder and the generator are deterministic stand-ins, so the output is identical every time. Save it as mini_rag.py.

"""A minimal RAG loop with no dependencies and no API keys.

The embedder and the generator are deterministic stand-ins so the output
is identical on every run. Swap them for real providers (see SWAP below).
"""
import hashlib
import math
import re

DOCS = {
    "refunds": "Customers can request a refund within 30 days of purchase. "
               "Refunds are issued to the original payment method within 5 business days.",
    "shipping": "Standard shipping takes 3 to 7 business days. "
                "Express shipping takes 1 to 2 business days and costs extra.",
    "warranty": "Hardware is covered by a 12 month warranty. "
                "The warranty does not cover accidental damage.",
}

DIM = 256


def tokenize(text):
    return re.findall(r"[a-z0-9]+", text.lower())


def embed(text):
    """Deterministic stub: hash each word into a 256-dim count vector, then normalize."""
    vec = [0.0] * DIM
    for word in tokenize(text):
        idx = int(hashlib.md5(word.encode()).hexdigest(), 16) % DIM
        vec[idx] += 1.0
    norm = math.sqrt(sum(v * v for v in vec)) or 1.0
    return [v / norm for v in vec]


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


def chunk(text, size=1):
    """Split into sentence chunks (size = sentences per chunk)."""
    sentences = [s.strip() for s in re.split(r"(?<=[.!?])s+", text) if s.strip()]
    return [" ".join(sentences[i:i + size]) for i in range(0, len(sentences), size)]


# 1. Index: chunk every document and embed every chunk.
index = []
for doc_id, text in DOCS.items():
    for piece in chunk(text):
        index.append({"doc": doc_id, "text": piece, "vec": embed(piece)})


def retrieve(query, k=2):
    q = embed(query)
    scored = sorted(index, key=lambda c: cosine(q, c["vec"]), reverse=True)
    return [(round(cosine(q, c["vec"]), 3), c) for c in scored[:k]]


def generate(query, hits):
    """Stub generator: answer with the best retrieved chunk, citing its source.
    SWAP: replace with a real LLM call that receives the same context."""
    best_score, best = hits[0]
    if best_score < 0.2:
        return "I don't know based on the provided documents."
    return f"{best['text']} [source: {best['doc']}]"


def answer(query):
    hits = retrieve(query)
    print(f"Q: {query}")
    for score, c in hits:
        print(f"   {score:.3f}  ({c['doc']}) {c['text'][:60]}")
    print(f"A: {generate(query, hits)}
")


answer("How many days do I have to request a refund?")
answer("Can I get my money back?")

The output, identical on every run:

Q: How many days do I have to request a refund?
   0.500  (refunds) Customers can request a refund within 30 days of purchase.
   0.224  (shipping) Standard shipping takes 3 to 7 business days.
A: Customers can request a refund within 30 days of purchase. [source: refunds]

Q: Can I get my money back?
   0.129  (refunds) Customers can request a refund within 30 days of purchase.
   0.000  (refunds) Refunds are issued to the original payment method within 5 b
A: I don't know based on the provided documents.

All four pipeline stages are present here: the index loop is indexing, retrieve is retrieval, the context passed into generate is augmentation, and generate is generation.

Swapping in a real embedder. Replace the embed function with a call to an embeddings API. Using OpenAI's current Python client, per OpenAI's embeddings guide:

from openai import OpenAI

client = OpenAI()

def embed(text):
    response = client.embeddings.create(input=text, model="text-embedding-3-small")
    return response.data[0].embedding

The vector is read from response.data[0].embedding. The cosine function above assumes unit-length vectors, which suits OpenAI: its guide states embeddings are normalised to length 1, so cosine similarity can be computed as a dot product. Normalise the result yourself if your provider does not do this. Then replace generate with a call to your model, passing the retrieved chunks in the prompt.

How retrieval fails

The second question in the run above is the most useful part of it, because "Can I get my money back?" means the same thing as a refund request while sharing almost no words with the refund policy. The stub scored it at 0.129, below the cutoff, so the system correctly said it did not know, and a real embedding model handles that case far better than word hashing does. The underlying lesson still holds: the model can only answer from what retrieval returns.

Failure	What happens	Common fix	Evidence
Vocabulary mismatch	The question and the answer use different words, so the right chunk scores low	Use semantic embeddings, add keyword search alongside them, or rewrite queries	Demonstrated by the run above
Chunks lose their context	A chunk such as "revenue grew by 3% over the previous quarter" does not say which company or period	Add context to each chunk before indexing	Anthropic's contextual retrieval results, below
Relevant text sits mid-context	The model uses information at the start or end of its input better than information in the middle	Retrieve fewer, better chunks and rank the best first	Liu et al., Lost in the Middle
The model misuses good context	Retrieval is fine, but the answer is not faithful to what was retrieved	Measure faithfulness separately and tighten the prompt	Ragas names faithfulness as its own dimension
The index goes stale	Documents change and the index does not	Re-index on change and log document versions	ModelRefs reasoning; the original paper lists updating knowledge as a motivation
Retrieved text carries instructions	A document contains text that tries to steer the model	Treat retrieved content as untrusted input	See what prompt injection is

On chunk context. Anthropic reports that prepending chunk-specific context before embedding cut the top-20 retrieval failure rate from 5.7% to 3.7%. Combining that with contextual BM25 took the failure rate to 2.9%, and adding a reranking step took it to 1.9%.

Read those numbers with their setup in mind. The headline results used Gemini Text 004 embeddings and Cohere's reranker, the chunk context came from Claude 3 Haiku, and the evaluation averaged across codebases, fiction, arXiv papers and science papers. Anthropic recommends running your own evaluations on your own data, and so do we.

On position. Liu et al. found that performance is often highest when relevant information sits at the beginning or end of the input context, and degrades when a model must use information in the middle. They tested multi-document question answering and key-value retrieval, and report the effect even on models built for long contexts.

How to tell whether your RAG works

Evaluate retrieval and generation separately, because the two fail for different reasons and call for different fixes.

• Retrieval. Take a set of test questions with known source documents, and check whether the right chunk appears in the top results at all.
• Faithfulness. Check whether the answer is supported by the retrieved text, rather than by the model's memory.
• Answer quality. Check whether the answer actually resolves the question that was asked.

Those three map directly onto a published framework. Ragas, short for Retrieval Augmented Generation Assessment, proposes metrics for retrieval quality, faithful use of the retrieved passages, and generation quality. Its useful property is that it is reference-free, so you do not need human-written ground-truth answers to get started, which makes it a practical first harness rather than a research project.

Our tutorials on evaluating prompts programmatically and using an LLM as a judge show how to build the harness around it.

RAG vs. fine-tuning at a glance

Question	RAG tends to fit	Fine-tuning tends to fit
Does the knowledge change often?	Yes: update the index, not the model	No: the behaviour or style is stable
Do you need to show sources?	Yes: retrieved chunks can be cited	Harder: the knowledge sits inside the weights
What are you trying to change?	What the model knows	How the model behaves or formats output

This is a ModelRefs framing rather than a benchmark result. The full decision framework, with cost maths and a constraints checklist, is in fine-tuning vs. RAG vs. long context.

Common misunderstandings

• RAG teaches the model new things. It does not, because the model is unchanged. It reads what you retrieved, the way you can read a reference sheet without memorising it.
• RAG eliminates hallucination. It reduces it, but the model can still misread the evidence or answer past it. See what LLM hallucination is.
• Bigger context windows make RAG obsolete. Long context helps, but stuffing everything into the prompt is slower and costlier. It is often less accurate than retrieving the right few passages, as the position effects above show.
• RAG is one thing. It is a family of designs, in which chunking strategy, search type, reranking and citation handling all vary, and each materially changes quality.

Where RAG appears in real systems

RAG shows up in customer support assistants grounded in help centres, enterprise search over wikis and drives, and coding assistants that retrieve from your repository. It also underpins legal and medical research tools that must cite sources, and chatbots over product documentation.

On ModelRefs, RAG implementation blueprints live in the workflow catalogue, and choosing a model for RAG has its own model selection guide for RAG.

Limitations and caveats

• Retrieval is the weakest link. If search returns the wrong chunks, the model answers fluently from wrong evidence.
• Citations can drift. Models sometimes cite a retrieved source that does not actually support the sentence, which is why faithfulness is measured separately.
• Chunking can hurt context. Splitting documents can separate a rule from its exceptions.
• It is not a security boundary. Retrieved documents are untrusted input, so anything that reaches the prompt can try to steer the model and should be handled accordingly.

Next steps

Start with the building blocks, how embeddings represent meaning and what a vector database does. When you are ready to build, work through the hands-on RAG tutorial, then the enterprise RAG workflow guide.

Sources and further reading

Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020). The paper that introduced RAG, pairing parametric memory with a dense Wikipedia index, and reporting state of the art on three open-domain question answering tasks, accessed 19 Oct 2026.

Anthropic, Introducing Contextual Retrieval. The chunk-context problem and the reported top-20 failure rates of 5.7%, 3.7%, 2.9% and 1.9%, with the embedding model, reranker and data types stated, accessed 19 Oct 2026.

Liu et al., Lost in the Middle: How Language Models Use Long Contexts. Position effects across multi-document question answering and key-value retrieval, accessed 19 Oct 2026.

Es, James, Espinosa-Anke and Schockaert, Ragas: Automated Evaluation of Retrieval Augmented Generation. Reference-free evaluation across retrieval, faithfulness and generation quality, accessed 19 Oct 2026.

OpenAI, Vector embeddings guide. The embeddings call, the text-embedding-3-small model, and the statement that returned embeddings are normalised to length 1, accessed 19 Oct 2026.

Methodology: every figure attributed to a paper or vendor page was read from that source on 19 October 2026. The minimal RAG script was executed on Python 3.13.15 before publication, and the output shown is the output it produced, byte-identical across repeated runs. The RAG versus fine-tuning table is a ModelRefs framing, not a benchmark result.