Evaluation Gates Are the Reliability Moat

Most agent deployments that get shut down were not failing the average. They failed once, in a way nobody caught, and the person who found it was a customer or a regulator instead of the team. The base rate of correct decisions was probably fine. What killed the project was that nobody could tell, before the decision shipped, that this particular one was the bad one.

People treat the base model as the reliability moat. The scoreboard is the online gate that can execute, escalate, or reject before an action ships.

What a gate actually is

Offline evaluation happens before a model ships. You run it against a benchmark, look at the aggregate score, and decide if it clears the bar. A strong offline eval can look like a gate if you never ship the agent. That resemblance is real, and it stops at the release meeting. The offline number is a ship-or-hold decision; it does not sit on the live path.

A gate is online. It sits between the moment an agent proposes an action and the moment that action executes, and it decides, per decision, whether to let it through.

Strip the tooling and a gate is a function with three outputs, not two. Given a proposed action, it returns execute, escalate, or reject.

Execute means the confidence signal cleared the bar and the action runs unsupervised. Escalate means the signal was ambiguous enough that a human should look before anything happens. That is human-in-the-loop done as a band: a human reviews the cases the system itself flagged as uncertain. Reject means the signal was bad enough that the action should not happen at all. No human required.

Most teams build agents with two states: it runs, or it doesn't compile. The three-state version is the difference between a demo and a system you can trust with real actions, because "I don't know" is a legitimate output instead of a bug.

def gate(proposed_action, confidence_score, execute_threshold, reject_threshold):
    if confidence_score >= execute_threshold:
        return EXECUTE
    if confidence_score < reject_threshold:
        return REJECT
    return ESCALATE

Two thresholds, not one. The gap between them is the escalation band, and its width is a decision someone has to make on purpose — not a default left over from a tutorial.

Gates and guardrails

The two get bundled together in most reliability write-ups. They solve different problems.

A guardrail is a class filter. A rule or classifier on input or output blocks a category of content regardless of the specific decision underneath it: profanity, PII, an off-topic request. It does not reason about whether this particular action is likely to be correct. It only asks whether the action belongs to a class that is disallowed on sight.

A gate reasons about the individual decision. Two proposed actions can both pass every guardrail, contain no disallowed content, stay on topic, and still deserve different outcomes, because one of them is the kind of claim the system has historically gotten right and the other is the kind it hasn't. Guardrails narrow the space of actions that are even eligible to run. The gate then decides, within that eligible space, whether this specific one should run unsupervised. If you only have guardrails, you still let confidently wrong answers through, because wrong is not the same category as disallowed.

In an MCP-based agent, the natural place to put a gate is the same boundary that already exists in the protocol — the point where the model's tool call is about to execute. That boundary is already instrumented for logging and permissioning in most MCP servers, which makes it the cheapest place to add a gate rather than building a parallel review layer around the whole agent.

Confidence scoring

The gate is only as good as confidence_score. The tempting shortcut is the model's own token probabilities, or a self-consistency check across resampled generations. Both measure how sure the model is of itself, not whether it is right.

A model that is confidently and consistently wrong passes that kind of check every time. Agreement with your own prior sample is not evidence from outside your own weights. If you use that signal as the gate, the execute path ships the cases the model is most sure about — exactly where a confidently wrong model does the most damage.

The signal that holds up comes from somewhere the model does not control: a tool call that returns a verifiable result, a retrieval that either supports or contradicts the claim, a schema or type check the output either satisfies or doesn't, or a human label on a past decision the system can learn to predict. None of these are exotic. They are the difference between asking the model how sure it is and checking the answer against something the model did not generate.

The threshold is a P&L line

Move the execute threshold up and escalation volume rises: more human review hours, slower response times, higher operating cost, fewer bad actions reaching a customer. Move it down and the reverse happens. Cheaper, faster, and the tail risk of an unreviewed bad decision grows.

This is not a hyperparameter search for the number that maximizes an F1 score. It is a cost curve someone in the room needs to own, the same way a fraud team owns a false-positive rate instead of asking a data scientist to pick one that feels right.

The execute threshold is a staffing decision dressed as a float. Raise it and you hire reviewers; lower it and you accept more tail risk. Teams that skip that conversation end up with a threshold nobody chose on purpose, usually whatever the default was in the framework they copied from a tutorial. Then they are surprised when the escalation rate is either so high the human reviewers become the bottleneck or so low the gate wasn't doing anything.

The escalation queue itself is a product surface, not a debug log. A reviewer looking at an escalated decision needs the proposed action, the confidence score, and enough context to decide quickly. A queue that takes five minutes per item to review will accumulate backlog at any real request volume, and the gate's benefit disappears into a growing pile of unreviewed cases. The gate's benefit is the reviewed decision, not the unread flag. The interface a human sees at the point of escalation matters as much as the threshold that put them there.

The log

Every gated decision produces a record: what was proposed, what the confidence score was, whether it executed, escalated, or got rejected, and eventually, once a human or a downstream outcome confirms it, whether it was actually right. That record is a labeled dataset that only exists because the system has been running in production against real cases, and it grows every day the gate operates.

A competitor can license the same base model. They can copy the architecture from a conference talk. What they cannot copy is six months of gated decisions paired with real outcomes. Six months is a horizon, not a measured dataset. It is a byproduct of having shipped and kept the gate running, not a file you can download.

It is also the input to recalibrating the gate itself. Retrain the confidence scorer on the outcome log; the escalation band tightens as the system gets a better sense of which of its own decisions to trust.

def log_and_recalibrate(decision_log, scorer, retrain_every_n):
    if len(decision_log) % retrain_every_n == 0:
        labeled = [(d.features, d.confirmed_outcome) for d in decision_log if d.confirmed_outcome is not None]
        scorer.fit(labeled)
    return scorer

The loop closes only if outcomes get confirmed and fed back in. A gate that logs decisions but never reconciles them against what actually happened is a dashboard, not a moat.

The model isn't the differentiator

Two companies can run the same frontier model behind the same agent framework and ship different products, because the model was never where the differentiation lived. One of them built a gate, chose its thresholds like a cost decision instead of a tuning parameter, and spent a year accumulating outcome data it can use to tighten that gate further. The other shipped the agent raw and found out where the gate should have been the same week a customer did.

A log without confirmed outcomes is a dashboard. Put the gate on the tool-call boundary, pick the two thresholds as a cost decision, and start confirming the cases that ship. The six-month set only exists if you keep doing that.