The "A" in RAG — turning retrieved chunks into a prompt that actually keeps the model grounded
Retrieving the right chunks doesn't automatically produce a grounded answer. Without an explicit instruction, an LLM will often still answer from its own training knowledge even when handed irrelevant or empty context — and without a strategy for combining multiple chunks, formatting citations, and ordering context, either the prompt overflows or quality degrades. Augmentation is the step that makes retrieval actually pay off, and it's often under-invested in relative to the effort spent on retrieval itself.
Once relevant chunks are retrieved, the prompt needs to clearly tell the LLM to use only that information and not guess.
Without a grounding instruction like this, LLMs will often still answer from their own training knowledge even when given irrelevant or empty retrieved context — defeating the whole purpose of RAG.
Most chat-tuned models are trained to weight the system message as a standing instruction that governs the whole conversation, and the user message as the specific request. Production RAG prompts typically put the grounding rule, tone, and output format in the system message (it applies to every turn) and put the retrieved context plus the current question in the user message (it changes every turn). Mixing this up — burying the "don't guess" instruction inside a long user message alongside the context — makes it easier for the model to lose track of the instruction as context length grows, since the instruction now competes with retrieved text for attention in the same message.
A grounding instruction alone doesn't produce traceable answers — the prompt has to explicitly ask for citations and give the model a way to reference sources unambiguously, usually by numbering the retrieved chunks and asking the model to cite by number:
This also makes hallucinated citations easier to catch mechanically: if the model cites [5] and only 3 chunks were retrieved, that's a detectable, checkable failure rather than a subjective judgment call — a cheap automated guardrail worth adding before it ever reaches a user.
When the answer needs to be consumed by other code rather than read by a human — a downstream API, a UI component, an automated pipeline — the generation step should request structured output (JSON mode, or a schema-constrained response) directly, rather than free text later parsed with regex. A typical schema separates the answer from its evidence explicitly:
The insufficient_context flag is worth calling out specifically: asking the model to explicitly flag when the retrieved context didn't actually contain the answer turns "silently made something up" into a structured, loggable signal you can monitor (see Production Concerns).
Because models pay more attention to the start and end of a long prompt (see Generation Failure Modes), the order chunks are placed in the prompt is itself a lever, not an afterthought. A common pattern re-orders retrieved chunks by relevance score so the single most relevant chunk sits either first or last in the context block — not buried at position 3 of 5 — even though that's not the order they were originally ranked in for selection purposes. This costs nothing extra to implement and is one of the cheapest quality wins available in the augmentation step.
If retrieval returns several chunks, there are four common strategies for feeding them into the LLM.
If retrieved chunks are still too large to fit even after picking a chain strategy, two common fixes: truncate the lowest-ranked chunks first, dropping whatever scored lowest in retrieval and keeping only what fits; or use contextual compression — an LLM trims each chunk down to only the sentences actually relevant to the question, keeping more distinct sources while using less total space.