Fine-tuning, RAG or prompting? A decision guide
When to use prompting, RAG or fine-tuning for an LLM app, with a decision table, cost and data needs, failure modes and a step-by-step flowchart.
"Should we fine-tune?" usually comes up early in an LLM project, and the answer is usually "not yet". That isn't because fine-tuning is bad. The three approaches fix different problems, and most teams reach for fine-tuning before they know which problem they actually have.
This guide is how I decide. The short version: prompting changes what you ask, RAG changes what the model can see, and fine-tuning changes how the model behaves. Find out which of those is wrong first.
What each one actually changes#
Prompting changes the instructions and examples you send with each request. Nothing about the model changes. It's the fastest to iterate on, because you edit text and rerun.
RAG (retrieval-augmented generation) finds relevant documents at request time and puts them in the prompt. The model's weights don't change, but it can now see information it never learned: your company's documents, today's data, a customer's records.
Fine-tuning trains the model further on your examples, so its weights change. It's good at teaching a consistent behavior: a format, a tone, a classification scheme, a specialized style of reasoning. It's poor at adding knowledge that you need to update or cite.
The most common mistake is to use fine-tuning to teach facts. The model may learn to sound like it knows your documents while still getting details wrong. It also can't tell you where an answer came from, and you'd have to retrain every time a document changes.
The decision table#
| Question | Prompting | RAG | Fine-tuning |
|---|---|---|---|
| The model lacks information (private, recent, large) | No | Yes | Poor fit |
| Answers must cite sources | No | Yes | No |
| Information changes often | Somewhat (if it fits in the prompt) | Yes | No (retrain each time) |
| Output format or style is inconsistent | Try first | No | Yes, if prompting fails |
| A narrow task needs to run cheaply at high volume | Sometimes | No | Yes (a smaller tuned model) |
| Specialized behavior that's hard to describe in words | Partly (few-shot examples) | No | Yes |
| Time to first working version | Hours | Days | Days to weeks |
| Data you need | A few examples | Your documents | Hundreds to thousands of good examples |
| Ongoing cost | Tokens per request | Tokens + retrieval infrastructure | Training + hosting or a per-model premium |
| Easy to debug | Yes | Yes (inspect what was retrieved) | Harder |
Costs, latency and data in more detail#
Prompting. Long prompts cost tokens on every request and add latency. Prompt caching helps a lot when the long part stays the same, such as instructions or a reference document. The data you need is small: a clear instruction and a handful of good examples.
RAG. You pay for embeddings, a vector or search index, and the extra context tokens in each request. Retrieval usually adds tens to a few hundred milliseconds. The data you need is your documents, and how well RAG works depends mostly on how clean they are and how you split them into chunks. Most RAG quality problems are retrieval problems, so measure retrieval separately with recall@k on a labelled question set.
Fine-tuning. You need examples of exactly the input and output you want, and their quality matters more than their number. A few hundred clean examples can beat many thousands of noisy ones. With parameter-efficient methods such as LoRA, training a small open model is affordable on a single GPU. The larger costs are often building the dataset, hosting the tuned model, and redoing the work when the base model gets an upgrade. The benefit: a tuned small model can replace a large prompted model on a narrow task, often with lower latency and cost per request.
How each one fails#
Knowing the failure modes tells you what to measure.
Prompting fails when instructions conflict, when the prompt grows so long that the model ignores parts of it, or when the task needs knowledge the model doesn't have. You'll see the model following some rules and dropping others, or confidently making things up.
RAG fails when retrieval misses the right passage (the most common cause), when chunks cut an answer in half, when retrieved text contradicts itself, or when the model ignores the context and answers from memory. Look at what was retrieved before you blame the model.
Fine-tuning fails when the training data has errors (the model learns them faithfully), when examples don't cover real inputs, when the model overfits and loses general ability, or when people use it to inject facts that then go out of date. It's also harder to see why a tuned model gave a particular answer.
The flowchart#
Work through these in order. Stop at the first step that solves your problem.
1. Write a clear prompt with 3-5 good examples.
Build an eval set of 30-50 real inputs with expected outputs.
Good enough? -> Yes: ship it. Done.
-> No: look at the failures. What kind are they?
2. Failures are "doesn't know" (missing, private or recent facts)?
-> Add RAG. Measure retrieval recall separately from answer quality.
Fix chunking, add hybrid search or reranking before changing the model.
Good enough? -> Yes: ship it.
3. Failures are "knows, but wrong format, tone or behavior"?
-> First improve the prompt: structured output / JSON schema,
better examples, split into smaller steps.
Still failing consistently? -> Go to 4.
4. Is the behavior narrow, stable and important enough
to collect hundreds of good examples?
-> Yes: fine-tune (start with LoRA on a small model).
Compare against the prompted baseline on the same eval set.
-> No: keep prompting; consider a stronger base model.
5. Need cheaper/faster at high volume on a narrow task?
-> Fine-tune a small model on outputs you've verified,
and confirm on the eval set that quality holds.
Step 1 builds the eval set on purpose. Without it you can't tell whether RAG or fine-tuning helped, and every later decision becomes a guess.
They combine well#
These aren't exclusive choices. Common combinations:
- RAG with a good prompt. This covers most knowledge assistants. The prompt sets the rules (cite sources, say "I don't know" when the context doesn't contain the answer). RAG provides the facts.
- Fine-tuning plus RAG. Tune the model to use retrieved context well, for example to always cite and to follow a house format, while RAG supplies the current facts.
- A fine-tuned small model inside a larger system. For example, a tuned classifier that routes requests, or a tuned extractor, placed in front of a general model that handles the open-ended part.
Three examples#
An HR policy assistant for employees. The policies are private and change every quarter, and employees need to see which policy says so. That's RAG, with a prompt that requires citations and allows "not covered". Fine-tuning would get dates and amounts wrong and couldn't cite.
Turning support tickets into a fixed JSON format with 40 categories. Start with a prompt and a JSON schema. If the model keeps confusing similar categories after you improve the descriptions and examples, and you have a few thousand labelled tickets, fine-tune a small model. It will probably be cheaper and more consistent at high volume.
Writing product descriptions in a brand's voice. Try a prompt with several real examples first. If the voice is hard to describe and the prompted output drifts, and you have hundreds of approved descriptions, fine-tuning is a good fit. Product facts such as specs and prices should still come from the catalog at request time.
The short version#
Ask what's wrong before choosing the fix. If the model doesn't know something, use RAG. If it knows but behaves wrong, improve the prompt first and fine-tune only when the behavior is narrow, stable and well covered by data. Build the eval set before any of it, because it's the only way to know which approach worked.