Evaluation Gates Are the Reliability Moat
Agent deployments do not die from average model error. They die from the first undetected one. Why evaluation gates, not the base model, are the actual product differentiator.
Most agent deployments that get shut down are not shut down because the agent was wrong on average. They are shut down because it was wrong once, in a way nobody caught, and the person who found it was a customer or a regulator instead of the team. The base rate of correct decisions was probably fine. What killed the project was that the team had no way to know, before the decision shipped, that this particular one was the bad one.
An evaluation gate is the fix, and it is a narrower thing than "evaluation" usually means. Offline evaluation happens before a model ships: run it against a benchmark, look at the aggregate score, decide if it clears the bar. A gate is online. It sits between the moment an agent proposes an action and the moment that action executes, and it decides, per decision, whether to let it through.
What a gate actually is
Strip away the tooling and a gate is a function with three outputs, not two. Given a proposed action, it returns execute, escalate, or reject. Execute means the confidence signal cleared the bar and the action runs unsupervised. Escalate means the signal was ambiguous enough that a human should look before anything happens, the core of a human-in-the-loop system done right: not "a human reviews everything" and not "a human reviews nothing," but a human reviews the cases the system itself flagged as uncertain. Reject means the signal was bad enough that the action should not happen at all, no human required.
Most teams build agents with two states: it runs, or it doesn't compile. The three-state version is the entire difference between a demo and a system someone can trust with real actions, because it's the only version where "I don't know" is a legitimate output instead of a bug.
def gate(proposed_action, confidence_score, execute_threshold, reject_threshold):
if confidence_score >= execute_threshold:
return EXECUTE
if confidence_score < reject_threshold:
return REJECT
return ESCALATE
Two thresholds, not one. The gap between them is the escalation band, and its width is a decision someone has to make on purpose, not a default left over from a tutorial.
Gates and guardrails are not the same primitive
The two get bundled together in most reliability write-ups, and they solve different problems. A guardrail is a filter: a rule or classifier applied to input or output that blocks a category of content regardless of the specific decision underneath it, profanity, PII, an off-topic request. It doesn't reason about whether this particular action is likely to be correct, only whether it belongs to a class that's disallowed on sight.
A gate reasons about the individual decision. Two proposed actions can both pass every guardrail, contain no disallowed content, stay on topic, and still deserve different outcomes, because one of them is the kind of claim the system has historically gotten right and the other is the kind it hasn't. Guardrails narrow the space of actions that are even eligible to run. The gate then decides, within that eligible space, whether this specific one should run unsupervised. Systems that only have guardrails still let confidently wrong answers through, because wrong isn't the same category as disallowed.
In an MCP-based agent, the natural place to put a gate is the same boundary that already exists in the protocol: the point where the model's tool call is about to execute. That boundary is already instrumented for logging and permissioning in most MCP servers, which makes it the cheapest place to add a gate rather than building a parallel review layer around the whole agent.
Confidence scoring is the part that's usually wrong
The gate is only as good as confidence_score, and the tempting shortcut is to use the model's own token probabilities or a self-consistency check across resampled generations. Both measure how sure the model is of itself, not whether it's right. A model that is confidently and consistently wrong passes that kind of check every time, because agreement with your own prior sample is not evidence from outside your own weights.
The signal that actually holds up comes from somewhere the model doesn't control: a tool call that returns a verifiable result, a retrieval that either supports or contradicts the claim, a schema or type check the output either satisfies or doesn't, or, at the base of the whole thing, a human label on a past decision the system can learn to predict. None of these are exotic. They're the difference between asking the model "how sure are you" and checking the answer against something the model didn't generate.
The threshold is a P&L line, not a modeling choice
Move the execute threshold up and escalation volume rises: more human review hours, slower response times, higher operating cost, fewer bad actions reaching a customer. Move it down and the reverse happens: cheaper, faster, and the tail risk of an unreviewed bad decision grows. This is not a hyperparameter search for the number that maximizes an F1 score. It's a cost curve someone in the room needs to own, the same way a fraud team owns a false-positive rate instead of asking a data scientist to pick one that "feels right."
Teams that skip this conversation end up with a threshold nobody chose on purpose, usually whatever the default was in the framework they copied from a tutorial, and then they're surprised when the escalation rate is either so high the human reviewers become the bottleneck or so low the gate wasn't doing anything.
The escalation queue itself is a product surface, not a debug log. A reviewer looking at an escalated decision needs the proposed action, the confidence score, and enough context to decide quickly, because a queue that takes five minutes per item to review will accumulate backlog at any real request volume and the gate's benefit disappears into a growing pile of unreviewed cases. The interface a human sees at the point of escalation matters as much as the threshold that put them there.
The log compounds into something a competitor can't buy
Every gated decision produces a record: what was proposed, what the confidence score was, whether it executed, escalated, or got rejected, and eventually, once a human or a downstream outcome confirms it, whether it was actually right. That record is a labeled dataset that only exists because the system has been running in production against real cases, and it grows every day the gate operates.
A competitor can license the same base model. They can copy the architecture from a conference talk. What they can't copy is six months of gated decisions paired with real outcomes, because that dataset is a byproduct of having shipped and kept the gate running, not something you can download. It's also the input to recalibrating the gate itself: retrain the confidence scorer on the outcome log, and the escalation band tightens over time as the system gets a better sense of which of its own decisions to trust.
def log_and_recalibrate(decision_log, scorer, retrain_every_n):
if len(decision_log) % retrain_every_n == 0:
labeled = [(d.features, d.confirmed_outcome) for d in decision_log if d.confirmed_outcome is not None]
scorer.fit(labeled)
return scorer
The loop closes only if outcomes get confirmed and fed back in. A gate that logs decisions but never reconciles them against what actually happened is a dashboard, not a moat.
The model isn't the differentiator
Two companies can run the same frontier model behind the same agent framework and ship completely different products, because the model was never where the differentiation lived. One of them built a gate, chose its thresholds like a cost decision instead of a tuning parameter, and spent a year accumulating outcome data it can use to tighten that gate further. The other shipped the agent raw and found out where the gate should have been the same week a customer did.