Without a systematic way to measure quality, every change is a guess
Without a systematic way to measure quality, there's no way to tell whether a change — a different chunk size, a different embedding model, adding reranking — actually helped or made things worse. Without evaluation, you'd just be guessing based on a handful of manual spot-checks. Evaluation in RAG splits cleanly into two questions that are easy to conflate: is retrieval finding the right chunks, and is generation using them well? Every metric below answers one or the other, never both at once.
Before RAG-specific frameworks existed, information retrieval already had a mature set of metrics for "did the search return the right things, in the right order" — and they still apply directly to the retrieval half of a RAG pipeline:
These require a labeled dataset of (query → which chunk ids are actually relevant), and they measure retrieval quality independent of whatever the LLM does with the result afterward — the cleanest way to isolate a retrieval regression from a generation regression when something breaks.
Notice the split: Faithfulness and Answer Relevance score generation; Context Precision and Context Recall score retrieval. A system with high context recall but low faithfulness means retrieval is doing its job and generation is the problem (see Generation Failure Modes); the reverse pattern means the fix belongs in Retrieval Basics or Advanced Retrieval, not in the prompt.
Metrics like Faithfulness and Answer Relevance aren't computed by a simple formula — they're computed by asking a separate LLM to read the question, the context, and the answer, and produce a judgment (often broken into sub-steps, like extracting individual claims from the answer and checking each one against the context independently, then aggregating). This is powerful because it can judge open-ended natural language the way a human would, but it inherits real weaknesses: the judge model has its own biases and blind spots, it can be inconsistent between runs, and it's meaningfully more expensive and slower than a formula-based metric. Best practice is to periodically spot-check the judge's verdicts against actual human judgment on a sample, rather than trusting an LLM-as-judge score blindly forever — the judge itself needs to be evaluated.
Automated metrics scale, but they don't replace human judgment entirely — especially for subjective qualities like tone, or for catching a failure mode the automated metric was never designed to detect. A practical human evaluation rubric usually scores each answer on a small number of independent dimensions rather than one overall "good/bad" call: correctness (is it factually right), groundedness (is every claim traceable to a retrieved chunk), completeness (does it cover everything the question asked), and helpfulness/tone (does it actually resolve the user's need). Scoring dimensions independently avoids the common trap where one bad dimension (e.g., overly verbose) drags down a rating for an answer that's otherwise perfectly correct and grounded.
Create a small set — even 20-50 examples to start — of realistic questions, along with the expected correct answer and, ideally, which source document or chunk should have been retrieved. A well-built eval set deliberately includes more than just "easy, clearly-answerable" questions:
k and context recall directly)Re-run this eval set every time something changes in the pipeline — chunk size, embedding model, retrieval strategy — to measure whether it actually improved things, rather than relying on gut feeling. Treat the eval set itself as a living artifact: add a new case every time a real production failure is diagnosed, so regressions can't silently reappear.
An offline eval set catches known failure patterns, but it can't fully represent the real, messy distribution of live user queries. Production RAG systems commonly run controlled A/B tests — routing a percentage of live traffic to a new retrieval strategy or prompt version and comparing outcome metrics (user thumbs up/down, follow-up question rate, escalation to a human) against the existing version — before rolling a change out fully. This catches regressions that only show up under real query volume and diversity, which is a different (and complementary) signal to what an offline eval set can tell you.
Beyond RAGAS itself, several tools specialize in tracing and evaluating LLM/RAG pipelines end to end: TruLens, Langfuse, and Arize Phoenix all provide tracing of every step of a RAG call (what was retrieved, what prompt was built, what came back), alongside built-in or pluggable evaluation metrics — useful because debugging a bad answer almost always requires seeing the full trace, not just the final output, to tell which stage actually failed.