Instead of always running retrieval on every single query, the LLM acts as an agent that first decides whether retrieval is even necessary for this particular question, and can call the retriever as a tool — potentially multiple times, reasoning between each call, deciding what to search for next based on what it's already learned. This is typically implemented with the ReAct (Reason + Act) pattern: the model alternates between writing out a reasoning step ("I need to find out X before I can answer this") and taking an action (calling the retriever, or another tool), observing the result, and reasoning again — rather than committing to a single retrieval call up front.

Useful because not every user message needs retrieval — "hi, how are you?" shouldn't trigger an unnecessary vector search, but "what's our refund policy?" should. The cost is real: an agentic loop can call the retriever (and the LLM) multiple times per user turn, so both latency and cost scale with how many reasoning steps the agent decides to take, which is harder to bound than a fixed single-retrieval pipeline.

After retrieval happens, an additional LLM call — or a lightweight classifier — grades whether the retrieved chunks are actually relevant and trustworthy enough to answer the question. If they're graded relevant, generation proceeds normally. If they're graded irrelevant or low-confidence, the system falls back to a broader search strategy (like a live web search) or explicitly asks the user a clarifying question. This adds a self-correction loop, so bad or irrelevant retrieval doesn't silently produce a confidently wrong answer — directly addressing the over-reliance failure mode described in Generation Failure Modes.

Self-RAG takes the "should I retrieve" and "was this retrieval good" decisions and trains the model itself to make them explicitly, using special reflection tokens the model generates alongside its answer — one token type indicates whether retrieval is needed at all for the current step, another critiques whether a retrieved passage is actually relevant, and another critiques whether the generated output is fully supported by that passage. Where Agentic RAG and CRAG bolt reasoning and correction on top of a general-purpose model via prompting, Self-RAG bakes the same self-critique capability directly into the model's training — a different point on the same spectrum of "let the model reason about its own retrieval," trading a training investment for lower per-query prompting and latency overhead.

Instead of relying purely on vector similarity, this approach builds a knowledge graph from the documents first — extracting entities (people, companies, concepts) and the relationships between them — then retrieves by traversing that graph rather than, or in addition to, doing vector search. Especially useful for questions that require connecting facts across documents through a relationship that isn't necessarily "semantically similar" in wording — e.g. "who is the manager of the person who wrote this report?" — a structural connection a pure semantic search wouldn't easily surface. The cost is building and maintaining the graph itself: entity extraction and relationship linking at ingestion time is a meaningfully heavier pipeline than chunk-and-embed, and it needs re-running (at least incrementally) as source documents change.

Standard chunking treats every chunk as equally granular, which struggles with questions that need a whole document's gist ("summarize this report's main findings") rather than one specific fact — no single chunk contains "the whole picture." RAPTOR addresses this by building a tree at indexing time: base-level chunks are clustered by similarity, each cluster is summarized by an LLM into a higher-level node, those summaries are themselves clustered and summarized again, and so on up multiple levels — ending with a small number of top-level nodes that summarize large sections of the corpus. At retrieval time, the search can pull from any level of this tree: a fine-grained leaf chunk for a specific-fact question, or a high-level summary node for a broad, gist-level question — letting one index serve both kinds of queries well, which a flat chunk-only index structurally cannot.

Some questions require combining information from multiple separate retrieval steps in sequence, where each step depends on what was learned in the previous one — a single retrieval pass isn't enough.

This is handled through iterative retrieval loops — retrieve, reason about what was found, retrieve again based on that reasoning — rather than one single pass, and is closely related to Agentic RAG above; in practice, an agentic ReAct loop is often the implementation vehicle for multi-hop retrieval, rather than a wholly separate architecture.

Rather than concatenating every retrieved passage into one long prompt (the "Stuff" strategy from Augmentation & Prompting), Fusion-in-Decoder encodes each retrieved passage separately alongside the question, then fuses all of those separate encoded representations together only in the decoder — the part of the model that generates the answer. This lets the model effectively condition on many more passages than would ever fit as raw concatenated text in a single context window, since each passage is compressed into a representation before fusion rather than kept as full raw tokens throughout. It's a research-and-specialized-systems technique more than an off-the-shelf tool most teams reach for directly, but the underlying idea — fuse compressed representations of many passages rather than raw-text-concatenate a few — shows up conceptually in how some production reranking and compression pipelines are designed.