# What Superforecasters Got Right > Tetlock's superforecaster research and what it means for production ML: calibration as a feedback system, the Brier score as a standard, and why the loop almost never closes. - Published: 2025-02-09 - Reading time: 7 min read - Author: Caio Theodoro (https://caio.theodoro.dev/about.md) - Canonical HTML: https://caio.theodoro.dev/blog/what-superforecasters-got-right --- In 2005, [Philip Tetlock](https://goodjudgment.com/) published the results of a twenty-year study in which he had tracked the predictions of 284 experts, including economists, political scientists, and foreign policy analysts, against what actually happened. The main finding was that the experts were, on average, barely better than chance. The secondary finding, which has had more lasting influence, was that a small subset of forecasters were substantially better, and their superiority was not a function of domain expertise or credentials. It was a function of cognitive practice. Tetlock called them superforecasters. The AI field has mostly not engaged with this research. I think that's a mistake. --- ## What made superforecasters different The core difference was not that superforecasters knew more. It was that they were calibrated. When a superforecaster said 70%, outcomes happened roughly 70% of the time. When they said 90%, outcomes happened roughly 90% of the time. Their confidence estimates tracked reality. This property was not innate. It was developed through a specific practice: making predictions with explicit probabilities, recording them, waiting for outcomes, scoring themselves with proper scoring rules (most commonly the Brier score, which rewards accurate probability estimates rather than just correct predictions), and tracking their calibration history over time. The feedback loop was tight and honest. When they were overconfident in a particular domain, they could see it in their scoring history and adjust. The contrast with expert prediction in general: most expert prediction is verbal and non-quantitative. "Interest rates will probably rise." "There's a risk of recession." These expressions carry implicit probabilities but are never measured against outcomes in a way that would reveal systematic overconfidence or underconfidence. The expert never knows if their "probably" means 55% or 85%, because they've never checked. --- ## The Brier score as a standard The Brier score is defined as the mean squared error between a probability forecast and the binary outcome. A forecast of 0.9 for an event that occurs scores better (lower error) than a forecast of 0.9 for an event that doesn't occur, and it scores better than a forecast of 0.5 for an event that does occur. It is a proper scoring rule, meaning it cannot be gamed: the strategy that minimizes your expected Brier score over time is to report your honest probability estimate. Applied ML uses accuracy as the primary evaluation metric for classification tasks. Accuracy measures whether the predicted class is correct; it ignores the probability distribution. A model that predicts 0.51 probability for the correct class and a model that predicts 0.99 probability for the correct class score identically on accuracy. From a calibration perspective, these are completely different outputs, and the difference matters enormously if downstream decisions are made based on the probability estimate. The Brier score, or equivalent proper scoring rules, should be standard evaluation metrics for any probabilistic classifier in production. They almost never are. --- ## The decomposition practice Superforecasters also developed the practice of decomposing complex questions into sub-claims. Rather than estimating the probability of a complex geopolitical outcome directly, they would decompose it into the probability of each contributing factor, estimate those individually, and combine them. This practice had two effects: it forced explicit reasoning about the causal structure of the outcome, and it created more opportunities for calibration feedback (the sub-claims could be evaluated separately, not just the final prediction). The ML equivalent: rather than evaluating a pipeline end-to-end, decompose it into stages and evaluate calibration at each stage. If a three-stage pipeline produces outputs with 70% confidence, where is the miscalibration? The retrieval stage? The extraction stage? The generation stage? End-to-end calibration evaluation misses this structure and makes it harder to identify where the problem is and how to fix it. --- ## What production ML is missing The core thing that distinguishes superforecaster practice from standard production ML is the feedback loop. The superforecaster makes a prediction, records it explicitly with a probability, waits for the outcome, scores it, and updates their model of their own calibration. This loop closes. In production ML, the loop rarely closes. A model makes predictions continuously. Outcomes occur. In many production settings, including recommendation systems, content classifiers, and demand forecasters, the ground truth label for each prediction is either unavailable, delayed by weeks, or available for only a fraction of predictions. Even when ground truth is available, the infrastructure to match predictions to outcomes, aggregate them into calibration curves, and monitor those curves over time is often absent. The result: teams have a general sense of accuracy (what fraction of predictions are correct?) but no systematic picture of calibration (when the model says 80%, is it right 80% of the time?). Accuracy can remain stable while calibration degrades significantly, particularly as the input distribution shifts over time. --- ## The production implication Closing the feedback loop in production means making prediction history legible in the same way superforecasters made their own judgment legible. Every model prediction needs to be stored with its probability estimate, the relevant input fields, and a timestamp, because without the original probability the later outcome cannot tell you whether the model was honest. Ground truth labels then need to be linked back to those prediction records when they become available, which usually means connecting systems that were not designed to talk to each other: the serving path that emitted the prediction, the operational database where outcomes eventually appear, and the analytics layer where evaluation happens. Once those records are matched, calibration curves can be computed on a rolling window and tracked as live metrics alongside accuracy. When calibration degrades beyond a threshold, it should trigger the same kind of alerting as an accuracy drop, because the operational consequence is similar: the system is no longer producing the signal downstream decisions believe they are consuming. None of this is technically difficult. It requires organizational prioritization, which is why it mostly doesn't happen. The output that stakeholders see and care about is accuracy. Calibration is harder to explain and harder to visualize, and it doesn't appear in the metrics dashboard unless someone decided to put it there. This is where prediction markets are a useful cultural reference, not just a technical one. They make probability a public commitment. A forecast is not a vibe or a recommendation or a confident paragraph; it is a number that can be scored later. Production ML would benefit from the same discipline. The model said 0.73, the outcome happened or did not happen, and the system either learned from that record or it did not. --- ## The honest limitation Calibration monitoring solves the measurement problem. It doesn't solve the calibration problem. A well-calibrated model is one whose probabilities are honest; it is not necessarily one whose probabilities are useful. A model that is perfectly calibrated at 50% for everything is honest. It's telling you it doesn't know. It is still not useful. The goal is a system that is both calibrated and accurate, and that knows which it is for which types of inputs. In practice this means: calibration across different input subgroups (not just globally), calibration tracking over time as the distribution shifts, and using calibration scores to route uncertain predictions to fallback systems or human review. The superforecaster research suggests that the practice of measuring calibration, rather than any particular technique for improving it, is what drives improvement. When you can see that you're overconfident in a particular domain, you become less overconfident in that domain. The feedback loop is the intervention, because it turns confidence from a style of expression into a measurable claim about the world. --- More posts: https://caio.theodoro.dev/blog.md · About the author: https://caio.theodoro.dev/about.md · Machine-readable index: https://caio.theodoro.dev/llms.txt