Calibration
Why calibration matters more than accuracy in production ML systems, how miscalibration survives undetected, and what a calibrated system actually looks like.
A model is calibrated if, among all the predictions it makes with 80% confidence, 80% of them are correct. This sounds like a minimal requirement, almost too basic to deserve its own discipline. It is almost never met in production machine learning systems, and the consequences of miscalibration are less visible and more costly than most accuracy metrics would suggest.
The reason calibration matters is not aesthetic. Production systems do not consume predictions in isolation; they route them, price them, escalate them, suppress them, or turn them into user-facing decisions. Once a probability estimate becomes an operational input, dishonesty in the probability becomes a systems problem, not just a modeling flaw.
What calibration actually measures
Accuracy measures whether a model's predictions are correct. Calibration measures whether a model's confidence in its predictions is honest. These are related but not the same property, and a model can have high accuracy and poor calibration simultaneously.
The classic case is overconfidence: a model that assigns 95% probability to outputs that are correct only 70% of the time. The model is often right, but it's telling you it's more certain than it is. The less common case is underconfidence: a model trained with heavy regularization or trained to be conservative about its outputs assigns 60% probability to things that are actually correct 85% of the time, leaving useful signal on the table.
Both failure modes create the same downstream problem: the probability estimate is not useful for decision-making. If you can't trust that an 80% confidence score means something meaningfully different from a 60% confidence score, then confidence scores don't help you decide when to act on a prediction and when to route it to a human reviewer. You end up either applying uniform trust across all outputs or ignoring the confidence scores entirely, both of which waste information.
Why production systems are miscalibrated
The standard deep learning training procedure has a structural tendency toward overconfidence. Softmax output layers produce probability distributions, but those distributions are not calibrated probabilities. They are normalized scores, and the gap between the highest-scoring class and the second-highest tends to grow larger as the model trains longer and converges to lower loss values. The model becomes more confident in its decisions as it overfits, not because it's getting more accurate.
Temperature scaling is the most common post-hoc calibration technique: you learn a single scalar that divides the logits before the softmax, effectively widening or narrowing the distribution. It's computationally cheap and works well in the specific setting where it was developed (image classification on held-out data from the same distribution). The problem is that production ML systems are not operating in that setting. The calibration correction learned on the validation set applies to the validation distribution, and as the deployment distribution shifts, which it does continuously, the calibration degrades.
There's a more fundamental issue: calibration is typically evaluated once, at model development time, on a fixed held-out dataset. It's rarely tracked as a live metric in production. The model drifts, the input distribution shifts, the calibration degrades, and nothing in the monitoring stack flags it because accuracy metrics can remain stable even as calibration deteriorates. You're measuring the right answer rate but not the honest confidence rate.
The feedback loop problem
What makes calibration hard in production isn't just the technical measurement challenge. It's the feedback structure.
For calibration to be evaluated, you need to know the ground truth for past predictions. In many production settings, ground truth is delayed (sales forecasts evaluated against actual sales weeks later), incomplete (only a fraction of predictions get verified), or unavailable (a user didn't click, but you don't know if that's because the recommendation was wrong or because they were busy). Calibration evaluation requires a labeled dataset of predictions and outcomes, and in many settings that dataset doesn't exist in a form that's straightforward to use.
The result is that most production ML teams have a sense of their accuracy metrics but no systematic picture of their calibration. They know, approximately, how often the model is right. They don't know whether a prediction with 90% confidence is meaningfully more trustworthy than one with 70% confidence, or whether the entire confidence range has compressed to an uninformative band between 72% and 78% because the model learned to be conservative.
What superforecasters figured out
Prediction market research has produced a body of practice around calibration that applied ML has largely ignored. Tetlock's work on superforecasters identified calibration as one of the primary distinguishing properties of accurate forecasters. They did not simply know more; their confidence estimates were honest. When a superforecaster says 70%, outcomes happen roughly 70% of the time. When an overconfident analyst says 90%, outcomes might happen 65% of the time.
The practice that produces this calibration is simple and almost nobody in ML does it systematically: you make a prediction with a probability, you record it, you wait for the outcome, and you score yourself using the Brier score or a similar proper scoring rule. You track your calibration history and you notice when you're consistently overconfident in a particular domain. You update.
This feedback loop is the thing that's missing. The model makes predictions, the predictions are acted on, some fraction of outcomes are observed, but the connection from observed outcomes back to calibration assessment of the model is rarely closed. There's no production equivalent of tracking your prediction history against outcomes and updating your prior on your own calibration.
What a calibrated production system would look like
A calibrated production ML system looks less like a model property and more like a feedback system. The confidence scores are post-hoc calibrated on a recent sample of production traffic, not on the validation set from training time, and recalibrated on a schedule that reflects the rate of distribution shift. The calibration correction is treated as a perishable artifact, not a fixed property of the model.
Calibration is tracked as a live metric alongside accuracy. Not just "is the model right?" but "when the model says 80%, is it right 80% of the time?" This requires monitoring infrastructure that matches predictions to outcomes and computes calibration curves continuously. It's more work than accuracy monitoring. It's also more informative, because it reveals failure modes that accuracy can hide.
The same metric also needs to be sliced by subgroup. A model can be globally calibrated and locally miscalibrated, which means the aggregate curve looks honest while the model is overconfident for one product category, one geography, one user segment, or one document type. This is where calibration becomes operationally important: the average probability estimate is less relevant than whether the specific decisions being routed through the system are receiving honest confidence estimates.
Confidence scores are then used to route decisions. High-confidence predictions go one path; low-confidence predictions go another, typically involving human review or a more expensive fallback. This only makes sense if the confidence scores are calibrated. If they're not, routing on confidence is worse than routing randomly. The whole downstream value of having confidence estimates depends on those estimates meaning something.
The honest assessment
Calibration is not an exotic property. It's a basic requirement for probabilistic outputs to be useful in decision-making. The gap between how often calibration is discussed in research and how often it's measured in production is, in my experience, large. Teams usually do care, but calibration is harder to measure than accuracy, requires infrastructure that most teams haven't built, and produces consequences that are diffuse and hard to attribute.
The model is right 87% of the time. The model is also confident 95% of the time when it should be confident 82% of the time. Both things are true, and only one of them shows up in the metrics dashboard. Miscalibration survives because most systems were never instrumented to look at it, not because it's subtle.