What Superforecasters Got Right

The field inherited accuracy as the scoreboard and a verbal "probably" as the forecast. You report how often the predicted class was right, or you say there is a risk of recession, and neither one is a number you can lose.

What works is a proper score on probabilities you stored and later matched to outcomes. Dull bookkeeping, until the day you try to tell a 0.51 from a 0.99 after the fact.

Philip Tetlock published his study in 2005. It ran twenty years and covered 284 experts — economists, political scientists, foreign policy analysts — each scored against what actually happened. The average expert barely beat chance. The few who did beat it weren't more credentialed. They were calibrated.

Tetlock called them superforecasters. The AI field has mostly ignored this research, and I think that is a mistake.

What they were doing

Say 70% and the outcome happened roughly 70% of the time. Say 90% and it happened roughly 90% of the time. Confidence tracked frequency because superforecasters wrote down an explicit probability, recorded it, waited, scored it with a proper scoring rule, and kept the history.

The usual rule was the Brier score, which rewards an honest probability over a correct class. Overconfidence in a domain showed up as a number, and the next forecast in that domain moved.

Most expert prediction never enters that loop. "Interest rates will probably rise." "There's a risk of recession." Those phrases imply a probability and never meet an outcome. The speaker can't say whether "probably" means 55% or 85%, because nobody has ever checked.

The arithmetic accuracy cannot see

Take a prediction that was 0.51 correct and one that was 0.99 correct. Accuracy credits both the same, since both named the right class. The first is close to a coin flip. The second is a commitment you would route on.

The Brier score is the mean squared error of a probability against a binary outcome. A 0.9 forecast for an event that happens scores better than a 0.9 forecast for an event that doesn't, and better than a 0.5 forecast for an event that does. It is a proper scoring rule, so the strategy that minimizes expected Brier over time is to report the honest number. Rounding toward certainty can't game it.

Applied ML still uses accuracy as the primary metric for classification. Accuracy asks whether the predicted class is right and ignores the rest of the distribution, even though downstream systems price, route, and escalate on the probability. Those two outputs are different objects, and accuracy can't see the difference.

A probabilistic classifier in production should carry the Brier score, or any equivalent proper score, on its eval. It almost never does.

Why nobody owns the join

Tetlock scored people on questions that resolve. Yours don't. Labels show up weeks late, or never, or for a fraction of rows. Reliability diagrams already live in the notebooks, and the business bought accuracy anyway.

Those facts explain why the loop stays open. They don't make accuracy a proper score.

A model predicts continuously, and outcomes happen. In recommendation systems, content classifiers, and demand forecasts, the ground truth for a given row is unavailable, delayed by weeks, or available for only a fraction of traffic. Even when the label does arrive, the serving path that emitted the prediction, the operational store where the outcome lands, and the eval layer were never built to join. Nobody owns that join, so nobody owns calibration.

What you get is a feel for accuracy and no systematic picture of calibration. Roughly what fraction of predictions are correct? You know. Whether 80% means 80% of the time? You don't. Accuracy can stay flat while calibration degrades under shift, particularly as the input distribution moves, and Brier is almost never on the dashboard.

What closing it takes

Store the probability, the inputs, and the timestamp. Without the original probability, a later outcome can't tell you whether the model was honest. When ground truth arrives, join it. That usually means wiring together three systems that were never designed to talk: serving, the operational database, and the analytics layer.

Once the records match, compute calibration curves on a rolling window and put them next to accuracy. When the curve moves past a threshold, page the same way you page an accuracy drop, because downstream is still consuming a signal it no longer has.

Picture a three-stage pipeline that emits 70% confidence. It can be wrong in retrieval, in extraction, or in generation, and scoring that 70% end to end won't tell you which stage to fix.

Superforecasters already made this cut on the human side. They broke a complex geopolitical question into sub-claims, estimated each contributing factor on its own, and combined the pieces. That forced the causal structure into the open and created more places where a probability could later meet an outcome. The ML version is the same cut.

None of this is technically hard. It takes someone who prioritizes it, which is why it mostly doesn't happen. Stakeholders watch accuracy; calibration is harder to explain, harder to draw, and absent from the dashboard until someone puts it there.

A number you can lose

Prediction markets help here as culture, not as infrastructure. They make probability a public commitment. A forecast isn't a vibe or a recommendation or a confident paragraph; it is a number you can lose.

The model said 0.73. The outcome happened or it didn't, and the system either scored that record or it didn't.

The practice of measuring is the intervention. Superforecasters didn't acquire a better temperament and then start scoring. They scored, saw the overconfidence in a domain, and the next number in that domain moved. See that you are overconfident and you become less overconfident.

The honest limitation

Calibration monitoring solves the measurement problem. It doesn't make the probabilities useful. A model that is perfectly calibrated at 50% for everything is honest and still useless, because it is telling you it doesn't know.

You still need accuracy alongside calibration, globally and by subgroup, over shift. Then you use the score to route, sending uncertain rows to a fallback or a person.

A reliability diagram on the held-out set is a training-time snapshot, not the production join. The production object is a rolling match against labels that arrive late, if they arrive.

Start with the join. Store the probability, the inputs, and the timestamp on every prediction you already emit, and when the label arrives, score Brier on a rolling window and page it like an accuracy drop.