The RAG Pipeline — Big Picture

Indexing happens once, offline; retrieval and generation happen live, on every question

Why

RAG is really two pipelines wired together: one that prepares knowledge ahead of time (indexing), and one that uses it at answer time (retrieval plus generation). Knowing both — and exactly where they connect — is what makes it possible to debug a RAG system that's returning bad answers, instead of just reciting the acronym. Nearly every production RAG bug traces back to one specific stage; being able to name the stage is the difference between a 10-minute fix and a multi-day investigation.

Intuition

Indexing happens once, or on a periodic schedule, per document. Retrieval and generation happen on every single user query. A slow or broken indexing step corrupts everything retrieved from it later, silently, until someone notices the answers are wrong — it's the more dangerous failure mode precisely because it's invisible at write time. A slow or broken retrieval step only affects the current query and is much easier to notice and roll back.

Explanation
Indexing phase — offline, one-time or periodic
  • Document loader — load raw documents (PDFs, Word docs, database rows, web pages) into a standard format (see Document Loaders)
  • Text splitter (chunking) — break each document into small pieces (see Text Splitters)
  • Embedding — convert each chunk into a vector representing its meaning (see Embeddings)
  • Vector database — store all these vectors so they can be searched later (see Vector Databases)
Retrieval + generation phase — live, per user question
1. User Query → the question comes in 2. Query Embedding → convert the question into a vector using the SAME embedding model used for indexing 3. Semantic Search → find the chunks whose vectors are most similar (closest in meaning) to the query vector 4. Prompt Construction → combine the retrieved chunks + the original question into one prompt 5. Generation → send that prompt to the LLM, which now answers using the retrieved information as grounding
Worked example: one document, start to finish

Trace a single sentence from a fictional employee handbook through the whole pipeline, with concrete (illustrative) numbers at every stage:

SOURCE FILE: handbook.pdf, page 12 RAW TEXT: "Employees are entitled to 15 days of paid leave per calendar year, prorated for new hires based on start date." [Document Loader] → Document( page_content="Employees are entitled to 15 days of paid leave...", metadata={"source": "handbook.pdf", "page": 12, "section": "Leave Policy"} ) [Text Splitter, chunk_size=300, overlap=30] → this sentence fits inside one chunk alongside 1-2 neighboring sentences; metadata is copied onto every chunk produced from this document [Embedding model, e.g. 1536 dimensions] → chunk vector: [0.0134, -0.0871, 0.0442, ... ] (1536 numbers) [Vector Database] → stored as one record: {id, vector, page_content, metadata} alongside millions of other chunk records from other pages/documents --- Weeks later, a user asks: "How much PTO do new hires get?" --- [Query Embedding] → same embedding model → query vector: [0.0129, -0.0855, 0.0439, ...] (close to the stored chunk vector above — similar MEANING, different words) [Semantic Search, top_k=5] → this chunk is retrieved as one of the top 5 closest vectors, along with its metadata: {"source": "handbook.pdf", "page": 12, ...} [Prompt Construction] → "Context: [this chunk + 4 others]\n\nQuestion: How much PTO do new hires get?\n\nAnswer using only the context above." [Generation] → "New hires get 15 days of paid leave per calendar year, prorated based on their start date. (Source: handbook.pdf, page 12)"

Notice the query never shared exact words with the source text ("PTO" vs. "paid leave," "new hires" vs. "prorated for new hires") — this only worked because of the embedding step, which is why Embeddings is the component that makes semantic (meaning-based) retrieval possible at all, as opposed to plain keyword search.

Naive RAG vs. advanced RAG — where the extra machinery goes

The pipeline above is "naive RAG" — one embed, one search, one generate. Every technique in the rest of this guide inserts itself at one specific point in that same flow:

Before retrieval
Query rewriting, multi-query expansion, HyDE, self-query filter extraction — all transform the query before it ever touches the vector database.
During retrieval
MMR, hybrid search (BM25 + dense), metadata filtering — all change how the vector database is searched, not what happens before or after.
After retrieval, before generation
Reranking, contextual compression, parent-document expansion — all reshape the retrieved set before it's handed to the LLM.
Around the whole loop
Agentic RAG, Corrective RAG, and multi-hop retrieval wrap the entire retrieve-then-generate cycle in an outer reasoning loop that can run it more than once.
Where things actually break — one failure per stage
  • Loader: a scanned PDF with no extractable text yields an empty page_content — the document is "indexed" but invisibly contributes nothing.
  • Splitter: a chunk boundary lands inside a table or code block and both halves become meaningless on their own.
  • Embedding: the query is embedded with a different model version than the documents were — every similarity score becomes meaningless (see Embeddings).
  • Vector DB: a stale index still returns a deleted document because the delete never propagated (see Production Concerns).
  • Retrieval: the top-5 chunks are all near-duplicates of each other, so the model never sees the one chunk that actually answers the question (see Retrieval Basics).
  • Generation: the model ignores the retrieved context entirely and answers from memory anyway (see Generation Failure Modes).
Where the name comes from
R — Retrieval
Pulling relevant chunks from the vector database for this specific query.
A — Augmentation
Inserting those chunks into the prompt alongside the user's question (see Augmentation & Prompting).
G — Generation
The LLM produces the final answer, conditioned on the augmented prompt instead of on memory alone.
The connection
Retrieval quality caps generation quality — the model can only ground its answer in what augmentation actually handed it.
← Previous Next →