Fine-tune, RAG, or prompt? A decision framework
The three ways to adapt an LLM solve different problems, and teams reach for the expensive one first. Here's when each is right — and why 'the model doesn't know our stuff' almost never means fine-tuning.
When an LLM isn't doing what you need, there are three levers — prompting, RAG, and fine-tuning — and they are not interchangeable. The most expensive mistake I see is reaching for fine-tuning to solve a problem that was really a retrieval or a prompting problem. Here's how to tell them apart.
They solve different problems
- Prompting — change behaviour and format with instructions and examples. Cheapest, instant, no training; the first thing to exhaust.
- RAG — inject knowledge the model lacks (current, private, or too large to hold in the prompt) at query time. You update it by changing data, not by retraining (see the RAG-fundamentals post).
- Fine-tuning — bake a style, format, or behaviour into the model by training on examples. It changes how the model responds, not what facts it knows.
The mistake: fine-tuning to add facts
The most common error is fine-tuning to teach the model your documents. It doesn't reliably work: fine-tuning shapes behaviour, not a queryable factual store, and it bakes in a snapshot you then have to retrain to update. If the need is 'answer from our knowledge', that's RAG. The clean split: fine-tune for how to say it, retrieve for what to say.
The decision
- Need current or private facts? → RAG.
- Need a consistent style, format, or domain behaviour the prompt can't reliably produce? → fine-tune (LoRA/QLoRA make it cheap — see the QLoRA post).
- Neither? → prompt it, and stop there.
Often the real answer is a combination: a RAG system whose generator is lightly fine-tuned for the house style — retrieval for the facts, fine-tuning for the voice.
If you do fine-tune
The work is mostly data, not GPUs: curate a high-quality dataset of exactly the behaviour you want (this dominates the result), then supervised fine-tuning, optionally sharpened with preference tuning like DPO or RLHF. Distillation is a cousin — train a small model to imitate a bigger one's outputs in your domain (see the distillation post). For proprietary corpora, the hard part is data curation, licensing, and confidentiality, not the training run.
Prompt first, retrieve for facts, fine-tune for behaviour — and never fine-tune to teach the model something a database should have told it.