RAG vs. Fine-Tuning: An Enterprise Decision Framework

"Should we fine-tune or use RAG?" is the wrong question, and it's the one we hear most. The right question is: what are you trying to change — what the model knows, or how the model behaves? Retrieval changes knowledge. Fine-tuning changes behavior. Most enterprise systems that struggle chose the wrong tool for the axis they cared about.

Here is the framework we use to make this call on client engagements, with the numbers that inform it.

The two axes: knowledge and behavior#

Knowledge is factual content: your product catalog, policy documents, contract clauses, ticket history. Knowledge has two critical properties — it changes (sometimes hourly), and it must be attributable (auditors ask "where did this answer come from?").

Behavior is form and procedure: output format, tone, domain-specific reasoning patterns, classification boundaries, tool-calling conventions. Behavior changes slowly and doesn't need citations.

RAG injects knowledge at inference time; the model reads it like an open-book exam. Fine-tuning adjusts weights; the model internalizes patterns like studying. Trying to inject knowledge via fine-tuning fails in a specific way: models trained on your documents approximate their content, and approximation of facts is hallucination with extra steps. A model fine-tuned on your pricing sheet in January will confidently quote January prices in June.

When RAG wins#

Choose retrieval when any of these hold:

1. The corpus changes faster than your release cycle. If documents update daily and you can only afford to retrain monthly, fine-tuned knowledge is stale by definition. A RAG index re-embeds changed documents in minutes.
2. Answers require attribution. Regulated industries need "this answer derives from Policy §4.2, revision 7." Retrieval gives you provenance for free; weights give you none.
3. Access control varies by user. Retrieval can filter chunks by the caller's entitlements at query time. You cannot ACL a weight matrix — a model fine-tuned on confidential documents will leak them to any user who asks well.
4. The corpus is large relative to what the task needs. A 4M-document corpus where each query touches five documents is the ideal retrieval shape.

The engineering reality: RAG quality is 80% retrieval quality. Before blaming the generator, measure recall@k on a labeled set. In our deployments, moving from naive cosine similarity to hybrid retrieval (BM25 + dense embeddings, reciprocal rank fusion, then a cross-encoder reranker over the top 50) improved answer accuracy more than any model upgrade — on one legal-search system, groundedness went from 71% to 93% with the generator held constant. Chunking strategy matters almost as much: structure-aware chunks (respecting headings and clause boundaries, 300–800 tokens with metadata) consistently beat fixed-size windows.

When fine-tuning wins#

Choose fine-tuning when:

1. You need consistent structured behavior. Emitting valid domain-specific JSON, following a house style, or classifying against a taxonomy with hundreds of labels. Few-shot prompting gets you to ~90% consistency; a LoRA fine-tune on 5–10K examples routinely gets to 99%+.
2. Your prompt has become a novella. If you're spending 3,000 tokens per request on instructions and examples, fine-tuning bakes those instructions into weights. One document-processing client cut per-request prompt tokens from 3,400 to 400 — at 2M requests/month, that paid for the training run in eleven days.
3. Latency budgets are tight. Shorter prompts and smaller specialized models are faster. A fine-tuned 8B model matching a frontier model on a narrow task cuts p95 latency from 4s to 800ms and inference cost by roughly 90%.
4. The reasoning pattern is domain-specific but stable. Claims adjudication logic, underwriting heuristics, medical coding conventions — procedural knowledge that doesn't change weekly.

The failure mode to respect: fine-tuning is a commitment. Every base-model upgrade requires re-tuning and re-evaluation. Budget for the eval harness before the training run — if you can't measure the behavior you're tuning for, you can't maintain it.

The usual answer: both, layered#

Most production systems we build land on a hybrid: a fine-tuned model for behavior, RAG for knowledge. The fine-tune teaches the model your output schema, your tone, and — importantly — how to use retrieved context: when to cite, when to say "not found in the provided documents," how to reconcile conflicting chunks. Retrieval supplies the volatile facts.

A concrete architecture from a financial-services deployment:

• Retrieval tier: hybrid search over 2.3M documents, entitlement-filtered, reranked, ~350ms p50.
• Generation tier: an 8B model LoRA-tuned on 12K curated examples of grounded question answering in the firm's format.
• Escalation tier: queries the small model flags as low-confidence route to a frontier model with the same retrieved context (~7% of traffic).

Total cost per query: about one-sixth of frontier-model-for-everything, with higher measured groundedness, because the fine-tune specifically trained refusal-when-ungrounded.

A decision checklist#

Ask, in order:

1. Does the task fail because the model lacks facts? → RAG first.
2. Does it fail because outputs are inconsistent in form or procedure? → Prompt engineering first; fine-tune when few-shot plateaus and volume justifies it.
3. Do you need attribution or per-user access control? → RAG is mandatory for that content, whatever else you do.
4. Is cost or latency the binding constraint at your volume? → Fine-tune a smaller model; keep RAG for knowledge.
5. Still failing with good retrieval and a reasonable fine-tune? → Your problem is usually evaluation and data quality, not architecture.

The teams that succeed treat this as an empirical question: they build the eval set first — 200–500 real queries with graded answers — and let recall, groundedness, and cost-per-query pick the architecture. The teams that struggle pick an architecture in a slide deck and build the eval set after the complaints arrive.