The Third Number

Every agent system I know is scored twice. Once for how often it is right, and once for how much it costs to ask. The two numbers sit on separate dashboards, and the gap between them is where the money leaks out.

NVIDIA's Switchyard benchmark — the one where 93% of agent calls go to a 30B open model for 10.4% of the spend — reduces routing viability to a single inequality: the judge's cost divided by the price gap between the two models. If the cheap model is cheap enough relative to the expensive one, routing pays for itself. In their numbers: Opus alone cost $11.45 per run of the 145-task suite; routing cut it to $3.00, 74% cheaper, for a loss of six accuracy points (86.0 to 80.0). The judge ate 21.2% of the routed spend, the second-largest line item after the frontier model. That formula has no term for the consequence of an error. A wrong answer costs nothing in it, whether the wrong answer moves a $5 million wire, approves a chargeback, or files a return with the wrong beneficiary. Six accuracy points are treated as a fixed toll for savings. The formula never prices what those six points are worth.

Six accuracy points is not a number; it is six percent of your errors. What those errors cost depends on where they land. On a workload of 800 exceptions a month, six points is about 48 mistakes. Land them on high-severity cases at the going rate and they are worth 48 times the cost of a missed principal. Land them on duplicates and they are worth almost nothing. Benchmarks score correctness. Routers optimize price and quality. Gateways enforce spend and rate limits. None of them price the failure. The decision that matters is what it costs when the system is wrong, and whether you are routing around that cost or straight through it.

The assumption everyone shares

The missing error-cost term is not a Switchyard oversight. The measurement culture inherited a two-number scoreboard, accuracy and price, and never examined it.

A benchmark is built from tasks with ground-truth answers. A model is scored by the fraction it gets right. The metric is the incentive. Model makers optimize it, procurement reads it, deployment inherits it. If the metric is correctness, correctness is what gets optimized. Whether the system is calibrated, whether its failures cluster on expensive cases, and whether it knows when to escalate become someone else's problem. The layer between how often the model is right and how much the automation is worth was never instrumented. That is where operational loss lives.

The same assumption sits in every layer of the stack. Selective prediction research stops at the answer: abstain when uncertain, measured by risk-coverage curves, with no severity structure. Agent benchmarks score task completion, pass^k and reliability across trials, and treat every failure as equally costly. Cost-sensitive classification exists, but only in binary-screening settings like literature review. In 55% of the studies examined, cost-weighting changed which model was best. Gateways can tell you how many tokens a tenant spent and how fast. They cannot tell you that this customer's errors cost ten times more than that customer's, so the escalation threshold should differ.

The layers grew separately, each rational inside its own boundary. The cost of being wrong sits on the boundary. Nobody owns that layer. It is the largest unowned measurement space in the agent stack, and it sits on top of the systems that will move money.

What expected loss actually is

Expected loss is a different objective. A production agent should estimate the cost of being wrong, choose an execution strategy accordingly, stop before irreversible harm, escalate when appropriate, and produce evidence that the policy worked.

Formally, it is small. Let K(σ) be the business cost of a failure at severity σ. Let p̂ be the calibrated probability of that failure. A single auto-decision costs p̂·K(σ) in expectation; a reviewed decision costs the reviewer's time. Escalating is worthwhile exactly when the expected avoided loss exceeds the review cost. Routing among models is the same shape: pick the model that minimizes p̂·K(σ) plus its price. That is Bayes decision theory from 1970. No new math.

The reason this is not already in use is the inputs. A calibrated p̂ needs a calibration loop most production systems do not have. The calibration essay I wrote earlier is the case that confidence scores are usually unmeasured, and that routing on uncalibrated confidence is worse than routing randomly. A defensible K(σ) needs sourced, contested, versioned cost figures, which nobody publishes. A measurement that shows severity changes the answer needs a benchmark whose ground truth can be trusted.

The theorem at the center is simple, and it explains the field: when all severities cost the same, expected loss ranking is identical to accuracy ranking. Every leaderboard is measuring the flat-cost special case without saying so. The divergence between the accuracy-optimal policy and the loss-optimal policy grows monotonically with cost asymmetry — with how wrong it is to treat a missed $5 million wire the same as a missed duplicate. The repo ships that as an executable test (tests/test_flat_cost_theorem.py): when severity is flat, the metric agrees with everyone else's. When it is not, it does not. That disagreement is the product.

Why the divergence is not theoretical

Error costs in operational systems are skewed by orders of magnitude.

The project ships four cost profiles. The spans between severities are the load-bearing assumption, stated where everyone can contest them:

A reconciliation exception makes the asymmetry legible. An amount mismatch on a principal transfer is principal at risk; a duplicate is a rebook at worst. A missed high-severity exception costs fifty times a missed duplicate under any sane cost model. The ratio is anchored in what the failure does. Fedwire transfers average in the millions. A misrouted one is a $1M–$10M incident class. The US payments machine moves $140 trillion a year. An ACH credit averages $3,881. The fraud multiplier says every dollar of fraud loss costs the merchant $3.75 in total costs. Basel capitalizes operational risk for exactly these event classes, which means banks already carry capital for the errors being scored. The asymmetry is the definition of the domain.

When the asymmetry is that large, the scoreboard changes. Take 10,000 reconciliation decisions a month, 800 of them exceptions (500 HIGH, 200 MEDIUM, 100 LOW), at the shipped reconciliation weights (10 / 1.0 / 0.2). Two models, same monthly decision count:

Model B is five and three-quarter points less accurate. Under the reconciliation cost model it still loses 39% less money. Under the principal-risk model (large-value settlement, where HIGH is 1,000 and CRITICAL 100,000) it loses 76% less. Which model is better is not a capability question; it is a question about which scoreboard you believe. The scoreboard you believe is decided by the cost model you can defend.

I measured this divergence in the previous project. A 1.7B model fine-tuned in about a hundred minutes on a laptop beats a frontier model on severity-weighted recall on an 800-task benchmark, 0.913 to 0.872, while losing raw accuracy 0.805 to 0.876. The frontier model gets more tasks exactly right and loses the ones that cost money. The small model catches every high-severity exception (1.000 recall across all four HIGH classes), parses 100% of 800 outputs where the frontier misses 0.4%, and averages 38 tokens per verdict with zero reasoning-token overhead and zero API cost. The WMCC studies found the same flip at scale: cost-weighting changes the best model in a majority of cases.

The implication is uncomfortable for procurement. If the metric is accuracy, you buy the frontier model. If the metric is expected loss, you buy the small fine-tuned one, deploy it on your own hardware, and spend the savings on review capacity. The industry currently has only one scoreboard to look at.

Why nobody owns it

The measurement layer for expected loss is empty because each input belongs to a different discipline, and no discipline had an incentive to complete the picture.

Selective prediction theory owns p̂ (the abstention threshold, the risk-coverage curve) and stopped at the answer level, where severities do not exist. Routing research owns the price term and published a formula that excludes everything else. Benchmarks own the tasks but score completion, not consequence. Gateways own the runtime but enforce spend, not risk. Regulators demand documented, logged, overseen systems. SR 26-2 replaced SR 11-7 this year with guidance written for modern modeling, and the EU AI Act's high-risk obligations are landing. They cannot demand a metric nobody has built.

The empty layer is where a benchmark and a control plane can live without colliding with an incumbent, because no incumbent is answering the question. A benchmark that scores severity-weighted expected loss, with ground truth nobody can dispute, becomes the thing model cards cite and routing papers get measured against. A control plane that consumes the same contracts becomes the thing risk committees can examine. The two halves share one trajectory record (every decision, its calibrated risk, its cost, its policy revision), which is what makes the measurement auditable and replayable. You can re-run last month's workload under a different escalation threshold, deterministically, with zero model calls, and read the counterfactual cost. That sentence did not exist anywhere before this project, as far as I can tell. It was not in the routing papers, the gateways, or the observability platforms. The platforms went to sleep on accuracy drift. Consequence-weighted drift, errors flat and cost rising, is unmonitored everywhere.

What it takes to own it

None of this is hard because the math is hard. It is hard because every input has to be built with the discipline to be trusted, and trust is a property you cannot bolt on afterward.

Ground truth comes first. A benchmark is only as good as its labels, so the generator is paired with an independent verifier that recomputes the answer from the fields alone, never reading the label the generator attached, and every task must pass before it is admitted. One hundred percent agreement, enforced by code. The benchmark is seeded. The same seed produces byte-identical tasks forever. Train and evaluation splits are signature-checked against each other. Contamination is a monitored quantity, not a hope. If a benchmark's numbers cannot be reproduced by a stranger, the benchmark is a press release.

Calibration is the hinge. The control loop assumes p̂ means something, so calibration is a first-class output (ECE, reliability curves, per-policy-version) and the pipeline refits thresholds from labeled outcomes that arrive through the ledger. The first local judge I ran measured a kappa of 0.3672 on a rubric fix that fixed a frontier judge to 0.9037. The failure is published. The fix is a judge-specific fine-tune, not more prompting, and it is a named workstream. Negative results stay because they are the evidence that the positive ones are real. The B2 study is the same discipline: rebalancing the training mix toward the rare classes collapsed severity-weighted recall from 0.913 to 0.723. Cutting a class's training share destroys its recall. Upsampling a subtle-signal class buys nothing. Kept, published. Training distribution must match deployment distribution.

Cost models are the contested part, and they are treated as such. K(σ) is a versioned, sourced input: a registry of empirical anchors with citations and confidence levels, plus pluggable profiles, plus the rule that every conclusion must be shown across a range of K, never at one chosen K. The registry (src/lossbench/costs/data/registry.yaml in the repo) currently holds ten entries across five domains, each with a low/typical/high range and a source. Misposted ACH sits at ~$3,881 typical (Federal Reserve Payments Study 2024), and misrouted wire is $1M–$10M (Fedwire Funds Services). Fraud is $3.75 per dollar of loss (LexisNexis True Cost of Fraud). A missed SAR is $1M–$100M (FinCEN, enforcement actions), and a false-positive AML review is $3–10 (LexisNexis True Cost of AML Compliance). Insurance fraud is $308.6B a year (Coalition Against Insurance Fraud). A manual prior-authorization touch is $12.43 against $2.56 electronic (CAQH Index). Medicare improper payments run around 6% — roughly $40B a year (CMS). Settlement failure is $100k–$1B (BIS). If your severity costs differ, you substitute yours and rerun; the analysis has to survive that. The claim is never that these are the true costs. The claim is what the ranking does as costs vary. That is the only claim that can be defended.

Around that core sits the control plane: record every decision into an append-only, hash-chained ledger, calibrate from labeled outcomes, decide under a versioned policy, escalate through durable review workflows with SLAs, monitor drift on the loss distribution itself, and replay any alternative policy against the recorded history. Every decision is an auditable record with its policy revision, model revision, prompt hash, and expected loss attached. This is what SR 26-2-style examination looks for. It is also what makes the counterfactual demo possible. You cannot re-run last month under a different policy unless last month was recorded.

What changes

The benchmark changes which decisions you can make defensibly.

It changes which model wins. Severity-weighting flips the best model in a majority of studied cases. The measured divergence — a 1.7B laptop fine-tune beating a frontier model on the money metric while losing on accuracy — is millions of dollars of inference spend, and the difference between a model that runs in-bank behind a data-residency wall and a frontier API call that leaves the building.

It changes what an agent is allowed to do unattended. The escalation threshold stops being a tuned-by-vibes constant and becomes a policy derived from measured costs: a documented, versioned, counterfactually-evaluated decision about how much risk the system can take autonomously.

It makes the failure mode legible to the people who audit it. A regulator or risk committee does not need to trust a demo. They need to see that every decision was recorded with its policy version, its calibrated risk, its evidence, and its human resolution, and that the policy can be replayed and contested. That is the gap between "the model is right 87% of the time" and a record of what the system costs when it is wrong, plus the policy derived from it.

Severity-weighting is the axis where small models have a structural advantage over frontier economics. A frontier model distributes its errors evenly. A narrow fine-tuned model can be shaped to never miss the expensive class. A 1.7B model trained in two hours on a laptop, catching every high-severity exception on a benchmark at zero API cost, is the shape of the axis. The axis favors the small, the local, the domain-specialized, and the auditable, because it prices what the big model prices away.

The honest assessment

The objections are real.

The data is synthetic. There is no live financial data, no real counterparties, no production loss history. The methodology is the subject, and the numbers are self-measured on a self-built benchmark. That is the point, because the measurement is the product, and it is also the limit. The severity costs are contested inputs, anchored in real figures with citations but assembled by one person. Transfer to a live book is unmeasured. The local judge calibration gap is open. The benchmark's conclusions will be attacked, and some of the attacks will be right. Pluggable costs, published negative results, and reproducible runs exist to survive the attacks that are.

There is a subtler risk. The moment a metric becomes the scoreboard, it becomes the target. If severity-weighted expected loss becomes standard, someone will game it: allocate costs to make their model look best, or shape a benchmark to flatter their weights. The defense is the defense every honest benchmark has: ground truth nobody controls, contamination that is monitored, costs that are contestable, and negative results that are kept. The DavidAU lesson of this year is the community enforcing a rule — downloads without verifiable evals now actively hurt you.

The thesis is falsifiable. If severity weighting rarely changes real-world rankings, if the flat-cost special case is the general case in practice, the project collapses into a well-engineered special case. That is why the flat-cost theorem is an executable test in the repository, not an essay claim. The property tests go further: raising the cost of high-severity failures must never lower the optimal escalation rate. Those assertions run in CI.

Sources and links

Everything here is measured or contested in public. The rule is that a headline number has to be reproducible by a stranger.

• Repository: github.com/caiotheodoro/lossbench. The benchmark, the control plane, and all 351 tests. make validate runs them. make determinism proves two full runs from the same seed are byte-identical.
• The theorem, executable: tests/test_flat_cost_theorem.py. Flat K ⇒ loss ranking equals accuracy ranking, and the property test that raising the cost of high-severity failures never lowers the optimal escalation rate.
• The cost registry: src/lossbench/costs/data/registry.yaml. Ten sourced anchors, five domains, ranges and citations.
• The design: design spec, architecture, implementation plan.
• The paper draft: lossbench-draft.md. The metric science behind the essay.
• The predecessor: a 1.7B model fine-tuned on a laptop beat DeepSeek on the metric that matters, where the 0.913 vs 0.872 divergence was first measured.

tests/test_flat_cost_theorem.py is the claim in a form a stranger can fail. Flat K, and loss ranking equals accuracy ranking. Raise the cost of high-severity failures, and the optimal escalation rate must not fall. If those assertions fail, expected loss was a special case. Until they do, it is the scoreboard that decides whether an agent should act.