RAG vs Fine-Tuning vs Prompting: Which One, When?

Three dollars. That is roughly what it costs in raw GPU time to fine-tune an eight-billion-parameter open-weight model with LoRA in 2026, and the job finishes in a few hours. The number looks so reasonable that plenty of companies head straight for it.

The problem is that three dollars is not even one percent of the real bill. The rest goes into collecting data, labelling it, building an evaluation loop, and the meetings six months later about why the model started making things up. And there is a harder truth underneath: for most of what companies want from fine-tuning, fine-tuning is the wrong tool. Getting accurate answers out of your own documents is not something it does.

What follows is a decision tree for choosing between three options: a well-built prompt, RAG, and fine-tuning. Which question sends you to which branch, what each branch costs, and where the money actually goes. We mapped the four big AI infrastructure decisions in our guide for non-technical leaders; this piece zooms in on one of them.

What does each approach actually change?

They operate on different layers. Fine-tuning changes how a model behaves: its tone, its output format, its habit of following instructions. RAG changes what a model knows, because it reads your documents before answering. Prompt engineering shapes both without touching infrastructure at all. The whole decision tree rests on that distinction.

Make it concrete. Wanting the model to answer customer emails in your company's registered tone, always in the same five-section structure, is a behaviour request. Wanting it to know the discount tier in your 2026 price list is a knowledge request. Fine-tuning is on the table for the first one and off the table for the second.

Microsoft's own developer guidance draws the same line: RAG enriches the model's input, while fine-tuning alters the model itself. Keep that sentence handy and you can rule out half the solutions pitched to you within the first minute of a vendor call.

Why "fine-tuning teaches the model your knowledge" is wrong

Because someone measured it, and it did not hold. Research from Technion and Google Research, published at EMNLP 2024, showed that models acquire genuinely new factual knowledge through fine-tuning only with great difficulty. Worse: as examples containing unfamiliar facts finally get learned, the model's tendency to hallucinate increases linearly.

Models learn facts mostly during pre-training. Fine-tuning does not teach them new facts; it teaches them to use the knowledge they already have more efficiently.

The mechanism is intuitive once you see it. Show a model a fact it does not know, wrapped in a confident sentence, over and over, and it does not learn the fact. It learns the habit of sounding confident. A separate study on models fine-tuned with scientific papers found exactly that pattern: the model absorbs the assertive style of scientific writing without the epistemic care underneath, and hallucination rates go up.

The nuance matters too. Fine-tuning on question-answer pairs about things the model already knows well improves factual accuracy. So fine-tuning is useful for getting existing knowledge out cleanly, not for putting new knowledge in. The industry shorthand has been right for years: fine-tuning is for form, not facts.

The decision tree: three questions, three branches

Three questions asked in order will get you to the right method. Each one either moves you up a cost tier or stops you where you are. The rule: you only climb a tier after you have measured that the cheaper one failed. The most expensive mistakes we see in the field come from skipping straight to the most complex option without that measurement.

Question one: does the model not know this, or does it know and say it badly? If it does not know (your price list, your contracts, your product catalogue), the answer is RAG, and fine-tuning will not fix it. If it knows but answers in the wrong format, rambling, or off-brand, you stay on the prompt branch.

Question two: how often does the information change, and do you need citations? If the information changes weekly or faster, fine-tuning is already illogical, since every update means retraining. And if you need to show which document an answer came from, fine-tuning is out too, because knowledge baked into weights cannot cite anything. Attribution and traceability are structural properties of a retrieval architecture.

Question three: do you have at least 50 real examples of a correct answer? OpenAI's own guidance puts the technical minimum at 10 examples and says meaningful improvement starts somewhere past 50 to 100; the practical range for business applications is 100 to 500. If you cannot carve out the time to produce those examples, the fine-tuning conversation ends here.

Squeeze one extra step in between: if your problem is inconsistent output format, try structured output (enforcing a JSON schema) before you try fine-tuning. It is incomparably cheaper and usually solves the whole problem on its own.

What does each option cost?

Roughly: the prompt branch is a few days of work; a RAG build is a mid-sized software project depending on scope; fine-tuning is cheap on compute and expensive on data. In none of the three does the money go into hardware. It goes into data and integration.

Some numbers. Training a 7-8B model with LoRA or QLoRA on a rented GPU lands in the 3 to 10 dollar range. Full fine-tuning of the same model needs over 100 GB of VRAM, meaning tens of thousands of dollars of hardware. The quality gap is narrower than you would expect: QLoRA reaches roughly 80-90% of full fine-tuning performance, standard LoRA 90-95%. The cheap method is now good enough.

On the RAG side, enterprise projects run from about 15,000 dollars (single source, simple interface) to 150,000 dollars (multi-source, role-based access control, audit logging). The instructive part is not the range but the split inside it. An analysis of 89 production RAG deployments found that 30-50% of total cost was data cleaning and preprocessing. The largest line item is not the language model bill; it is the access control layer and the integrations.

Speed carries a cost too. RAG adds 50-200 milliseconds for the retrieval step, and once you include processing the longer context, total latency can rise by 30-50%. A fine-tuned model skips that step, so it is cheaper and faster per query. But for that saving to pay back the setup cost, monthly query volume has to reach the millions. Most mid-sized companies never come close to that threshold.

Is the fine-tuning door closing in 2026?

Partly, at least on the closed-model side. OpenAI is winding down its self-serve fine-tuning platform: since May 2026 organisations that had never fine-tuned before cannot start new training jobs, and in January 2027 existing customers lose that ability too. Already-trained models keep running, but no new features are coming.

Anthropic never exposed a fine-tuning endpoint on its public API at all; a limited option exists only through Amazon Bedrock. What it promotes instead is telling: million-token context windows, prompt caching, context editing. The large providers are steering toward "give the model the right context" rather than "change the model".

The economics explain that choice. Reading from a prompt cache costs roughly a tenth of normal input pricing. A few years ago, "long system prompts are expensive, so bake the knowledge into the model" was a sensible argument. It has largely stopped being one. The fine-tuning door is not shut, but the way through it now runs via open-weight models and LoRA.

Which option survives a data deletion request?

RAG. The reason is technical but the consequence is legal. In a RAG architecture, personal data sits in a database; deleting a record is a query, and the next day the model no longer knows it. In fine-tuning, that data is spread across billions of parameters, and pulling one person's information back out means, in practice, retraining from scratch.

That gap matters wherever a right to erasure applies. Data protection regimes generally require deletion once the processing purpose ends and give individuals the right to demand it. Once you have baked personal data into model weights, you cannot technically comply with either obligation.

The practical rule: never turn a dataset containing names, contact details, health or financial records into training data. If your application genuinely needs that data, choose an architecture that keeps it outside the model; we walked through how in our non-technical guide to RAG. Treat this as general information rather than legal advice, and involve counsel when sensitive categories are in play.

Do you need a custom model for your language?

For most business applications, no. Current large models handle major non-English languages well enough, and reaching for fine-tuning because "it does not understand our language" is a reflex left over from 2023. But there is a real cost effect, and it comes from tokenization.

Morphologically rich languages, where words carry long chains of suffixes, get chopped into more pieces than English does. Measurements put tokens per word around 1.2-1.4 for English and roughly 1.8-2.5 for agglutinative languages, depending on the model. The same content costs 1.5-2 times more tokens, which is a direct line item on your API bill.

The effect on the decision is interesting. Because sending long context is more expensive in those languages, good retrieval (fetching few but correct documents) and prompt caching matter more, not less. The right answer is still not fine-tuning; it is a more disciplined RAG setup.

A concrete scenario: a 60-person insurance brokerage

Picture a brokerage with two complaints. Account managers constantly interrupt each other to find policy terms, and the quote emails going out look completely different depending on who sent them.

These sound like one problem and land on two different branches. Policy terms are a knowledge problem, the terms change constantly, and you need to show which clause an answer came from: RAG. Email tone is a behaviour problem, but you attack it with prompting first. Feed the brokerage's 30 best quote emails in as a template with a few examples and the consistency issue usually disappears.

When would fine-tuning enter this picture? Only when all three of these are true at once: prompting was tried and measurably fell short, 200-300 approved example emails have accumulated, and volume is genuinely high. In mid-sized companies we rarely see all three line up. Most projects get solved and closed on the first two tiers, which is good news, because those two are also the fastest to build.

Frequently asked questions

Can I use both together?

Yes, and most mature systems do. In the RAFT approach published jointly by Meta and Microsoft, the model is fine-tuned alongside both relevant and irrelevant documents, so it learns to use retrieved information, cite it, and ignore distractors. The typical mature architecture is RAG for content plus a light fine-tune for voice.

When I fine-tune, does my data go inside the model?

Yes, which is exactly why it is risky under data protection rules. Training data is distributed across the weights and cannot be extracted back out. With RAG, the data stays outside in a database.

Did long context windows make RAG unnecessary?

No. Million-token windows help, but "dump everything in" does not work; models tend to lose information sitting in the middle of a long context. The common 2026 pattern is hybrid: retrieve the relevant documents first, then reason over them in a wide context.

Does fine-tuning make responses faster?

Yes, since there is no retrieval step, so it is faster and cheaper per query. That advantage only repays the setup cost at very high volume.

Is fine-tuning an open-weight model worth it?

For a narrow, repetitive task, often yes; making a small model good at one job can be attractive on cost. If you are aiming for a general-purpose assistant, no.

So what should you actually do?

• Write the problem down in one sentence and ask: does the model not know this, or does it know and say it badly? That answer settles half the decision.
• Treat fine-tuning as the step after a measured failure, not as a starting option.
• For format and consistency problems, try structured output first; it fixes most inconsistency at close to zero cost.
• Never convert datasets containing personal data into training data, because deletion obligations become technically impossible to meet.
• Budget around data, not hardware. In both RAG and fine-tuning projects, most of the money is spent preparing data.

Choosing between these three is less a technology preference than a diagnosis. Once the diagnosis is right, the option usually picks itself, and more often than not the cheapest one turns out to be enough. If you are unsure which branch your problem sits on, half a day of diagnostic work will save you from a wrong build that takes months to unwind.