Fine-Tuning's Real Cost Isn't the Training Run
What fine-tuning actually specializes versus what it can't do, and the data and evaluation work that determines whether it's worth the cost.
Fine-tuning is the second most frequently requested thing in applied AI work, after RAG. And like RAG, it's often applied to the wrong problem. The technique is not the issue. The issue is that teams often have the wrong mental model of what fine-tuning does, which leads them to use it in situations where it can't help and to miss the situations where it could.
What fine-tuning is
Fine-tuning continues training a pretrained model on a new dataset, adjusting the model's weights to perform better on the target task. The pretraining has already given the model a general representation of language; fine-tuning specializes that representation for a narrower purpose. In practical terms, it changes the model's default behavior, the shape of the answers it tends to produce before a prompt has to force it there.
The key word is "specializes." Fine-tuning changes how the model applies what it already knows. It does not reliably teach the model new knowledge, and it does not reliably improve the model's general reasoning capability. These are the two most common misapplications, and both fail for the same underlying reason: fine-tuning is adjusting the output distribution of a model, not upgrading its internal world model.
What fine-tuning is not good for
Knowledge injection is the most common misuse. The reasoning is intuitive: "the model doesn't know our internal product documentation, so we'll fine-tune it on the documentation." This sometimes works, in the narrow sense that the model will produce outputs that mention the right product names and terminology. But the model is not storing facts reliably in the way a database stores records. Fine-tuning on factual information produces a model that sounds more like it knows your domain; it doesn't reliably know your domain in the way that retrieval does.
The failure mode is specific: fine-tuned models hallucinate confidently within the domain they were fine-tuned on. The terminology sounds right, the general framing is correct, and the specific facts are wrong in ways that are harder to detect than the generic hallucinations of the base model. The model has learned to write in your register without learning to be accurate about your content. RAG is the right solution to the knowledge problem. Fine-tuning is not.
General capability improvement is the second misuse. Teams see a frontier model outperforming their current model on reasoning tasks and conclude that fine-tuning on reasoning examples will close the gap. This almost never works. The difference in capability between a 7B and a 70B model is not a function of the training data distribution; it's a function of parameter count, and fine-tuning doesn't change parameter count. You can adjust the surface behavior of a model with fine-tuning; you cannot change its fundamental reasoning ceiling.
What fine-tuning is actually good for
Format and style normalization is the strongest use case. If you have a task that requires a specific output structure, such as a particular JSON schema, a consistent tone for customer communications, or a specific way of formatting code examples, fine-tuning is very good at internalizing that requirement. The base model can be prompted to follow a format, but with fine-tuning, the format compliance rate approaches ceiling and doesn't degrade over long context windows.
Narrow, high-frequency tasks with consistent schema benefit from fine-tuning for a related reason: the task is well-defined, the correct outputs are unambiguous, and you have (or can generate) enough training examples to make the specialization reliable. A model that classifies support tickets into categories, extracts structured fields from a specific document type, or translates between two well-defined formats is a good fine-tuning candidate. A model that answers general questions about a domain is not.
Latency reduction through system prompt compression is underappreciated. Long system prompts that specify personas, output formats, behavioral guardrails, and domain context add tokens to every request and increase inference cost. Fine-tuning a model on examples that reflect those instructions produces a model that behaves as if it has read the system prompt without the prompt being present. For high-volume inference applications, this can cut cost.
There is also a deployment reason this matters: behavior that lives in a prompt is easy to change but also easy to accidentally bypass. Behavior that lives in weights is harder to change, but more stable across product surfaces, prompt variants, and long conversations where the instruction hierarchy starts competing with user-provided context. That stability is valuable when the task is narrow enough that you actually want the model to behave the same way every time.
The data requirements
The data requirement is the most common underestimated constraint. Below roughly 5,000 to 10,000 training examples, fine-tuning on a general-purpose model produces unreliable results: the model improves on examples that look like your training data and degrades in unpredictable ways on everything else. The fine-tuning objective pushes the model toward your examples; it doesn't preserve the behavior on inputs outside your training distribution.
The quality requirement is stricter than quantity. Noisy training data produces a model that has memorized the noise patterns along with the signal. If your training examples contain inconsistencies, such as the same input with different expected outputs, labeling errors, or edge cases handled differently across raters, the fine-tuned model will reflect that inconsistency. Unlike pretraining, which can average over enormous amounts of noisy data, fine-tuning on a small high-noise dataset tends to amplify the noise.
The practical implication: before fine-tuning, you need a data audit. Not just "do we have 10,000 examples" but "are these 10,000 examples consistent, correctly labeled, and representative of the actual production distribution." This takes longer than the fine-tuning itself, which is usually a few hours on a modern training cluster.
The evaluation problem
Fine-tuning evaluation is harder than it appears. The standard approach is to hold out a portion of the fine-tuning dataset and measure performance on the held-out set. This measures how well the model has generalized from the training examples. But the held-out set is drawn from the same distribution as the training set, which means it's measuring generalization within your fine-tuning distribution, not on the full production distribution.
The failure mode: the model improves significantly on your evaluation set, gets deployed, and performs worse on a meaningful fraction of production traffic because those inputs don't look like the fine-tuning distribution. This isn't specific to fine-tuning. It's a general evaluation problem. But it's particularly acute for fine-tuned models because the fine-tuning objective is specifically pushing the model toward the training distribution and away from inputs that don't resemble it.
An honest evaluation requires sampling from production traffic, labeling those samples, and comparing the fine-tuned model to the base model on that sample. This is more expensive and takes longer. It's also the evaluation that tells you whether fine-tuning actually helped.
The comparison needs to include the actual alternative, not an artificially weak baseline. A fine-tuned small model should be compared against the prompted base model, a stronger off-the-shelf model, and a retrieval-backed version if the task touches knowledge. The useful question is not whether fine-tuning improved the model relative to itself; it is whether the improvement is large enough to justify the dataset maintenance, training process, regression risk, and operational complexity that come with owning a specialized model.
The honest state of it
Fine-tuning is useful in a narrow set of circumstances: format normalization, narrow high-frequency tasks, and latency optimization through prompt compression. In those circumstances, with sufficient high-quality data and an honest evaluation setup, it works reliably.
It is not a solution to the knowledge problem, not a path to capability improvement, and not a substitute for investing in retrieval infrastructure. The teams I've seen get the most out of fine-tuning are the ones that spent a long time being honest about what problem they were actually trying to solve before reaching for the technique, and discovered that the answer was sometimes "fine-tuning is not the right tool here" before they'd wasted six weeks on a training run. That discipline is the economics of fine-tuning: the training run is cheap compared with being wrong about why you needed it.