Finetuning Vs RAG

LLMs interview question · Medium difficulty

When should you use fine-tuning versus RAG?medium

Status
✓
Type
conceptual
Topic
finetuning-vs-rag
Frequency
common
Tags
fine-tuning, RAG, production, architecture-decision
Answer

Fine-tuning changes the model's weights and is best for improving style, format, domain-specific behavior, and consistent response structure. RAG retrieves relevant external documents into the prompt at runtime and is best for fresh knowledge, private data, frequently changing facts, and citations. In many production systems you use both: fine-tuning for behavior, RAG for knowledge.

Explanation

Fine-tuning example: training a model to generate customer-support replies in a company-specific tone — that's a behavior change, and weights are the right place to encode it. RAG example: answering user questions from internal HR policy PDFs — that's knowledge that changes over time and shouldn't be baked into weights. The key distinction is fine-tuning changes model behavior while RAG adds external knowledge at runtime; fine-tuning is a poor fit for frequently changing facts because retraining or re-tuning weights repeatedly is expensive and unreliable compared to just updating a document store. Interview-ready framing: use fine-tuning to teach the model how to respond, and RAG to provide what it should know.

Follow-upWhy is fine-tuning a poor fit for facts that change every week?
Follow-upWhat would you do if you needed both a specific tone and frequently updated facts in the same system?
Follow-upWhy can't RAG alone fix a model's tendency to respond in the wrong tone or format?