Every model has a maximum number of tokens it can accept in a single request. As of this guide, production context windows commonly range from roughly 128k tokens (a few hundred pages) up to 1M+ tokens on the largest windows offered by some providers — but even the largest real knowledge base (a company wiki, a full codebase, years of support tickets, a legal document archive) routinely runs into the tens or hundreds of millions of tokens. The ceiling is real regardless of how large it is, and it doesn't move as your data grows — retrieval does.

Transformer self-attention computes a relationship between every pair of tokens in the input, which costs O(n²) in the sequence length n. Doubling the prompt length doesn't just double the compute — it roughly quadruples it (mitigated in practice by techniques like sparse and flash attention, but the underlying scaling pressure is real, especially for prefill/time-to-first-token). Concretely:

On top of the compute cost, providers charge per input token, and the model has to process every one of them before it can generate the first output token — so a bloated prompt is both slower to first token and more expensive per call, even when the extra tokens contribute nothing to the answer.

Even when a document technically fits inside the context window, dumping in irrelevant text hurts answer quality. This was measured directly in Liu et al.'s 2023 paper "Lost in the Middle: How Language Models Use Long Contexts": accuracy on a question-answering task followed a U-shaped curve as the position of the correct answer moved through a long context — highest when the answer was at the very start or very end of the prompt, and measurably worse when it was buried in the middle, even though nothing else about the task changed. This is exactly why retrieval isn't just "context window management" — narrowing to a handful of genuinely relevant chunks also sidesteps a real, measured attention-allocation weakness.

"Needle in a haystack" benchmarks — testing whether a model can recall one fact planted anywhere in a long document — have improved substantially on frontier models, and some workloads genuinely fit: a handful of PDFs, a single large but static reference document, or a small, tightly curated corpus. In those cases, "just paste it in" is simpler to build and operate than a retrieval pipeline, and it avoids retrieval's own quality problems entirely (see Retrieval Basics). The trade-off flips back toward RAG as soon as: the corpus won't fit even in a huge window, the corpus changes faster than you want to re-send it every call, cost per query matters at volume, or citations/traceability are required.

Any single question is usually answerable from a handful of paragraphs, not the entire knowledge base. Loading everything dilutes the few relevant lines in a sea of irrelevant text the model still has to read through. Retrieval acts as a relevance filter: it narrows a huge corpus down to the small number of chunks that are actually likely to answer the question, before the LLM ever sees any of it — turning an O(n²)-over-everything problem into an O(n²)-over-a-few-thousand-tokens problem.