Calibration

A model is calibrated if, among all the predictions it makes with 80% confidence, 80% of them are correct. This sounds like a minimal requirement, almost too basic to deserve its own discipline. It is almost never met in production machine learning systems, and the miss is less visible than an accuracy drop and more expensive once the probability is an operational input.

People treat accuracy as the production scoreboard. The scoreboard is whether 80% confidence is right 80% of the time, measured on live traffic.

Production systems do not consume predictions in isolation; they route them, price them, escalate them, suppress them, or turn them into user-facing decisions. A dishonest probability is a routing error, a bad price, or a review that never fires.

What calibration actually measures

Accuracy measures whether a model's predictions are correct. Calibration measures whether the confidence on those predictions is honest. The two are related, and they are not the same property. A model can score high on accuracy and still be dishonest about how sure it is.

The classic case is overconfidence. The model assigns 95% probability to outputs that are correct 70% of the time. It is often right, and it is telling you it is more certain than it is. The less common case is underconfidence. A model trained with heavy regularization, or trained to be conservative about its outputs, assigns 60% to things that are correct 85% of the time, and leaves usable signal on the table.

Both failure modes make the probability useless for a decision. If an 80% score does not mean something different from a 60% score, you cannot use confidence to decide when to act and when to send the row to a reviewer. You apply the same trust to every output, or you ignore the scores. Either way the information is gone.

Why production systems are miscalibrated

The standard deep learning training procedure has a structural tendency toward overconfidence. Softmax layers emit probability distributions, but those distributions are normalized scores, not calibrated probabilities. The gap between the highest-scoring class and the second-highest grows as the model trains longer and converges to lower loss. The model gets more confident as it overfits, not because it got more accurate.

Temperature scaling is the usual post-hoc fix. You learn a single scalar that divides the logits before the softmax, which widens or narrows the distribution. It is computationally cheap, and it works in the setting where it was developed: image classification on held-out data from the same distribution.

It still works if you recalibrate on recent traffic. What dies is the frozen scalar from the validation set. That correction applies to the validation distribution, and as the deployment distribution shifts — which it does continuously — the calibration degrades. A scalar that was honest on the held-out set does not apply to this week's inputs.

Calibration is usually evaluated once, at development time, on a fixed held-out set, and it is rarely a live metric. The model drifts, the input distribution moves, and calibration degrades while accuracy can stay flat through all of that. The stack flags the right-answer rate and misses the honest-confidence rate.

The feedback loop problem

The hard part in production is the feedback structure.

You need ground truth on past predictions. In many settings that label is delayed (sales forecasts scored against actuals weeks later), incomplete (only a fraction of rows get verified), or missing — a user did not click, and you do not know if the recommendation was wrong or they were busy. Calibration needs a join of predictions to outcomes, and that join often does not exist in a form you can score.

Most teams have a sense of accuracy and no systematic picture of calibration. They know, approximately, how often the model is right; they do not know whether 90% is more trustworthy than 70%, or whether the whole range has compressed to a 72–78% band because the model learned to be conservative.

What superforecasters figured out

Prediction-market research produced a practice around calibration that applied ML has largely ignored. Tetlock's work on superforecasters named calibration as one of the properties that distinguished the accurate ones. They did not simply know more. Their confidence estimates were honest. When a superforecaster says 70%, outcomes happen roughly 70% of the time. When an overconfident analyst says 90%, outcomes might happen 65% of the time.

The practice is simple, and almost nobody in ML does it on a schedule. You write a probability, record it, wait for the outcome, and score it with the Brier score or another proper scoring rule. You track the history, notice the domain where you are consistently overconfident, and update.

The model already makes predictions, the predictions are acted on, and some fraction of outcomes get observed, but the join back to a calibration assessment of the model is rarely closed. There is no production equivalent of tracking that history against outcomes and updating the prior on the model's own calibration.

What a calibrated production system would look like

A calibrated production system looks like a feedback loop, not a model property. Confidence scores get post-hoc calibrated on a recent sample of production traffic, not on the training-time validation set, and get recalibrated on a schedule that matches the rate of shift. The correction is a perishable artifact.

Calibration sits next to accuracy as a live metric. When the model says 80%, the dashboard has to show whether it is right 80% of the time. That needs infrastructure that matches predictions to outcomes and computes calibration curves continuously. It is more work than accuracy monitoring, and more informative, because it shows failures accuracy can hide.

The same metric needs subgroup slices. A model can be globally calibrated and locally miscalibrated, which means the aggregate curve looks honest while the model is overconfident for one product category, one geography, one user segment, or one document type. The average probability is less relevant than whether the decisions being routed are getting honest confidence.

Then you route on those scores. High-confidence one path; low-confidence another, usually a person or a more expensive fallback. This only holds if the scores are calibrated. If they are not, routing on confidence is worse than routing randomly. The downstream value of the scores depends on those scores meaning something.

The honest assessment

Calibration is not an exotic property. It is the requirement that makes a probability usable for a decision. The gap between how often it is discussed in research and how often it is measured in production is, in my experience, large. Teams usually do care, but calibration is harder to measure than accuracy, needs infrastructure most teams have not built, and produces costs that are diffuse and hard to attribute.

The model is right 87% of the time. It is also confident 95% of the time when 82% would be honest. Those three figures are a shape, not a named production run. Both can be true at once, and only one of them is on the dashboard.

Miscalibration survives because most systems were never instrumented to look at it. Put the 80% bucket next to accuracy, and score it on live traffic.