RAG vs. Fine-Tune: The Build Decision

Most teams treat RAG versus fine-tuning as a capability question: whether the model can learn this from the prompt, or whether it needs to live in the weights. Ask a team why they picked one, and the honest reason is usually which writeup they read last. In production the scoreboard is update frequency, provenance, and the monthly cost shape a team can operate. If you write those three down, the choice is mechanical.

How often knowledge changes

A RAG index update is cheap. Add a document, re-embed it, and the next query can retrieve it. A fine-tune is not cheap in the same way. LoRA and QLoRA collapse the parameter count that needs updating. You still pay for a training run, an evaluation pass to confirm nothing regressed, and a redeploy, every time the underlying knowledge changes.

If the knowledge is a product catalog that updates hourly, RAG wins on this axis alone, before quality even enters. If the knowledge is a company's writing style or a fixed set of output conventions that almost never changes, the retrain cost is a one-time expense. Fine-tuning stops being the more expensive option.

LoRA makes the retrain look cheap enough that this axis should flip. The compute bill can drop; the eval pass and the redeploy do not. Those two stay in the cost of every update, which is why hourly knowledge still points at RAG after the adapter got smaller.

Whether the answer needs a paper trail

RAG gives you a citation by construction. The model's answer traces back to a retrieved chunk; you can show a user or an auditor which document the claim came from. A fine-tuned model has no equivalent. The knowledge is somewhere in the weight matrix. There is no operation that recovers which training example produced a given output.

For a support bot answering questions about a blog's tone, that gap is irrelevant. For a system in a regulated domain, where a wrong or unverifiable answer is a compliance incident rather than an annoyance, the absence of a provenance trail is disqualifying. Quality on the answers does not rescue it.

The retrieval layer has its own cost

RAG's cost is not just the query-time lookup. It is the vector database sitting underneath, and that choice carries its own build-versus-buy question. A hosted vector database like Pinecone charges for storage and query volume and removes the operational burden of running the index yourself. An extension like pgvector, if the team already runs Postgres, adds retrieval to infrastructure that is already being paid for and operated. You do the index tuning and scaling work in-house instead of paying someone else to have solved it.

Neither is a wrong answer. It is a second decision inside the first one. If you pick RAG without pricing the retrieval layer, you discover the real number after the vector database bill shows up next to the model bill.

Fine-tuning has an equivalent second decision. LoRA and QLoRA cut the trainable parameter count compared to full fine-tuning, which is what makes frequent retraining survivable at all. The eval pass that has to run after every retrain, checking the new adapter against a held-out set before it replaces the one in production, is a cost that does not shrink just because the training run got cheaper. If you automate the training step and skip the eval step, you are one bad update away from a regression nobody notices until a user does.

What a query actually costs

The per-query cost of RAG is a retrieval lookup plus the extra context tokens in the prompt — real, but marginal and roughly constant no matter how often the answer changes. Fine-tuning front-loads its cost into training and pays close to nothing extra per query. Every knowledge update means paying the training cost again.

The breakeven is a comparison: retrain cost times how often you retrain, against marginal retrieval cost times query volume.

def recommend(update_frequency_per_month, retrain_cost, marginal_query_cost, monthly_query_volume, needs_provenance):
    if needs_provenance:
        return "RAG"
    fine_tune_monthly_cost = update_frequency_per_month * retrain_cost
    rag_monthly_cost = monthly_query_volume * marginal_query_cost
    return "fine-tune" if fine_tune_monthly_cost < rag_monthly_cost else "RAG"

This is a toy version, deliberately. The real inputs are specific to a team's retrain pipeline and query volume. Plugging in invented numbers here would be worse than useless. The function shows the shape of the decision, not a number to copy — if you fill in fake costs, you get a fake answer.

Where flat retrieval still loses

A flat vector index answers "what's the most similar chunk to this query" well. It answers "what connects these three separate facts across documents" badly. Similarity search has no notion of the relationships between chunks, only their proximity in embedding space.

That is a different problem than the one this framework solves. Flat RAG does not cover every retrieval case. It is a big enough problem to deserve its own treatment.

When either architecture has failed

The decision framework picks an architecture. It does not tell you when that architecture is quietly failing in production, and the two failure modes look nothing alike.

A RAG system fails when retrieval returns the wrong chunk, or the right chunk but the model ignores it. The fix is in the retrieval or prompting layer, not the model's knowledge. A fine-tuned model fails when the world moved and the weights didn't — the fix requires knowing the model is stale, which nothing in the model's own output signals.

Both failures need the same kind of instrumentation an evaluation gate provides elsewhere in the stack. You log the retrieved context alongside the answer for RAG. You log the confidence and correctness of answers over time for a fine-tuned model, so staleness shows up as a trend instead of a support ticket.

The common answer is both

In practice the axes rarely all point the same direction. The honest recommendation for a lot of production systems is to do both. Fine-tune for the parts that are stable: a consistent output format, a house style, a way of following instructions that the base model does not do well out of the box. Reach for retrieval for the parts that are volatile: prices, policies, anything that changes on a schedule the training pipeline cannot keep up with. Treating the choice as exclusive is what produces teams re-litigating the decision every quarter instead of building the system that matches how their knowledge behaves.

There is a fourth variable underneath the three above that the framework does not capture and should not pretend to. What the team already knows how to operate sits outside the formula. A team with a mature training pipeline and an existing eval harness pays a lower real cost for the fine-tuning path than the formula suggests. The retrain cost that matters is not the compute bill; it is the engineering time to run it safely, and that time is already sunk into tooling that exists. A team with neither built already pays a startup tax on fine-tuning that a spreadsheet will not show.

The pragmatic answer for that team is to start with RAG regardless of what the update-frequency and provenance axes say, and build toward fine-tuning once the operational muscle exists to do it without an incident. The framework says what the right architecture is. It does not say what a specific team can build safely this quarter. Conflating the two is how technically correct recommendations turn into six-month outages.

Write the three axes down for the knowledge you actually have. If you do not already run a training pipeline and an eval harness, start with RAG this quarter.