RAG vs. Fine-Tune: The Build Decision
The RAG-versus-fine-tuning debate gets argued as a capability question. In production it is a cost and maintenance-surface question, decided by three axes and what a team can actually operate.
Ask a team why they chose RAG over fine-tuning, or the reverse, and the honest answer is usually about which one they read about most recently, not which one fits the data. The debate gets framed as a capability question: can the model learn this well enough from the prompt, or does it need to be in the weights. In production the question that actually decides the outcome is different. It's a cost and maintenance question, and it has a fairly mechanical answer once you write down the right three variables.
How often does the knowledge change
A RAG index update is cheap. Add a document, re-embed it, and the next query can retrieve it. A fine-tune is not cheap in the same way: even with LoRA or QLoRA collapsing the parameter count that needs updating, you still pay for a training run, an evaluation pass to confirm nothing regressed, and a redeploy, every time the underlying knowledge changes. If the knowledge is a product catalog that updates hourly, RAG wins on this axis alone, before quality even enters the conversation. If the knowledge is a company's writing style or a fixed set of output conventions that almost never changes, the retrain cost is a one-time expense and fine-tuning stops being the more expensive option.
Does the answer need a paper trail
RAG gives you a citation by construction: the model's answer traces back to a retrieved chunk, and you can show a user or an auditor which document the claim came from. A fine-tuned model has no equivalent. The knowledge is somewhere in the weight matrix, and there's no operation that recovers which training example produced a given output. For a support bot answering questions about a blog's tone, that's irrelevant. For a system operating in a regulated domain, where a wrong or unverifiable answer is a compliance incident rather than an annoyance, the absence of a provenance trail is disqualifying regardless of how good the model's answers otherwise are.
The retrieval layer has its own cost structure
RAG's cost isn't just the query-time lookup, it's the vector database sitting underneath it, and that choice carries its own version of the build-versus-buy question. A hosted vector database like Pinecone charges for storage and query volume and removes the operational burden of running the index yourself. An extension like pgvector, if the team already runs Postgres, adds retrieval to infrastructure that's already being paid for and operated, at the cost of doing the index tuning and scaling work in-house instead of paying someone else to have solved it. Neither is a wrong answer, but it's a second decision inside the first one, and teams that pick RAG without pricing out the retrieval layer discover the real number after the vector database bill shows up next to the model bill.
Fine-tuning has an equivalent second decision: LoRA and QLoRA cut the trainable parameter count dramatically compared to full fine-tuning, which is what makes frequent retraining survivable at all, but the eval pass that has to run after every retrain, checking the new adapter against a held-out set before it replaces the one in production, is a cost that doesn't shrink just because the training run got cheaper. A team that automates the training step and skips the eval step is one bad update away from a regression nobody notices until a user does.
What a query actually costs
RAG's per-query cost is a retrieval lookup plus the extra context tokens in the prompt: real, but marginal and roughly constant no matter how often the answer changes. Fine-tuning front-loads its cost into training and pays close to nothing extra per query, but every knowledge update means paying the training cost again. The breakeven is a straightforward comparison: retrain cost times how often you retrain, against marginal retrieval cost times query volume.
def recommend(update_frequency_per_month, retrain_cost, marginal_query_cost, monthly_query_volume, needs_provenance):
if needs_provenance:
return "RAG"
fine_tune_monthly_cost = update_frequency_per_month * retrain_cost
rag_monthly_cost = monthly_query_volume * marginal_query_cost
return "fine-tune" if fine_tune_monthly_cost < rag_monthly_cost else "RAG"
This is a toy version, deliberately: the real inputs are specific to a team's actual retrain pipeline and query volume, and plugging in invented numbers here would be worse than useless. What the function is meant to show is the shape of the decision, not a number to copy.
Where flat retrieval still loses
None of this argues that RAG is free of its own failure mode. A flat vector index answers "what's the most similar chunk to this query" well and "what connects these three separate facts across documents" badly, because similarity search has no notion of the relationships between chunks, only their proximity in embedding space. That's a different problem than the one this framework solves, and it's worth naming instead of overselling flat RAG as if it covers every retrieval case. It's a big enough problem to deserve its own treatment rather than a paragraph here.
Knowing when either one has failed
The decision framework picks an architecture. It doesn't tell you when that architecture is quietly failing in production, and the two failure modes look nothing alike. A RAG system fails when retrieval returns the wrong chunk, or the right chunk but the model ignores it, and the fix is in the retrieval or prompting layer, not the model's knowledge. A fine-tuned model fails when the world moved and the weights didn't, and the fix requires knowing the model is stale, which nothing in the model's own output signals. Both failures need the same kind of instrumentation an evaluation gate provides elsewhere in the stack: log the retrieved context alongside the answer for RAG, log the confidence and correctness of answers over time for a fine-tuned model, so staleness shows up as a trend instead of a support ticket.
The common answer is both
In practice, the axes rarely all point the same direction, and the honest recommendation for a lot of production systems is to do both: fine-tune for the parts of the system that are stable, a consistent output format, a house style, a way of following instructions that the base model doesn't do well out of the box, and reach for retrieval for the parts of the system that are volatile, prices, policies, anything that changes on a schedule the training pipeline can't keep up with. Treating the choice as exclusive is what produces teams re-litigating the decision every quarter instead of building the system that actually matches how their knowledge behaves.
There's a fourth variable underneath the three above that the framework doesn't capture and shouldn't pretend to: what the team already knows how to operate. A team with a mature training pipeline and an existing eval harness pays a lower real cost for the fine-tuning path than the formula suggests, because the retrain cost that matters isn't the compute bill, it's the engineering time to run it safely, and that time is already sunk into tooling that exists. A team with neither built already pays a startup tax on fine-tuning that a spreadsheet won't show, and the pragmatic answer for them is to start with RAG regardless of what the update-frequency and provenance axes say, and build toward fine-tuning once the operational muscle exists to do it without an incident. The framework says what the right architecture is. It doesn't say what a specific team can build safely this quarter, and conflating the two is how technically correct recommendations turn into six-month outages.