Copying ARC-AGI's Benchmark Method, Then Stress-Testing It

An adversarial audit of my pipeline found the efficiency metric measuring nothing and the 2-of-10 human bar computed and then never enforced. Two of my findings flipped. The tables below are the corrected ones.

The pipeline copies ARC-AGI-3's benchmark method, and I then measured the method. Every task gets a programmatic verifier, ARC's 2-of-10 human bar, a random-policy floor of 1 in 10,000, and an RHAE score against a human baseline. Code: github.com/caiotheodoro/substrate. A green suite and a formula that matches the paper say nothing about whether any of those parameters hold.

Why copy ARC at all

ARC-AGI-1 survived five years of scaling. ARC-AGI-2 lasted months. ARC-AGI-3's technical report documents why: once models can sample the task space densely enough, a generate → verify → train loop eats the benchmark. Their own Gemini-3 verification model, never told ARC's integer-to-color mapping, used the correct mapping in its reasoning chain. The data was in the model. Benchmarks die of absorption, not saturation.

ARC's answer was a production methodology. Every environment attempted by 10 humans, solved by at least 2. Random-policy floors at 1 in 10,000. Efficiency scoring against human baselines. Private splits structurally out-of-distribution from public ones. Humans score 100%. Frontier models score under 1%.

I built that recipe as a pipeline, then measured the recipe's own parameters. Benchmark builders almost never do that part.

What the audit found

Before this pass I asked for an adversarial code and data audit of the whole pipeline. Not whether it runs. Whether the code does what the docs and the numbers claim. It found real defects.

RHAE, the efficiency metric, was measuring nothing. The simulated human baseline was generated independently of how many actions a task actually needed. Every solved task hit the metric's cap identically. RHAE and raw solve rate were the same number, dressed as two confirmations.

The 2-of-10 human-calibration bar was computed and then never enforced. Tasks that failed it were scored anyway.

Then the smaller list. A gauntlet novelty check compared every task against a permanently empty corpus, so duplicates were never caught. A contamination monitor's leaked knowledge base was built from the same population it tested, so any leaked task trivially matched itself. A Spearman implementation didn't average-rank ties, which biased the one number every study reports through. A few silent-failure paths let a dead network endpoint and a genuine negative result produce identical-looking output.

All of it is fixed, verified with tests written against the specific failure. Two of the fixes changed the findings. The difficulty-compression sign flip no longer replicates. The claim that structural features can't predict difficulty was largely an artifact of the same bugs. Both are corrected below, old and new numbers side by side where it matters.

The pipeline

The Calibrated Evaluation Foundry generates verifiable agent tasks (programmatic verifiers only, no judge scores an answer). It runs them through a gauntlet (random-policy floor, fuzz, reproducibility, novelty), calibrates difficulty against a human oracle with ARC's 2-of-10 bar, and fits a difficulty model. It then stratifies public/private splits matched on difficulty, scores solvers with ARC's RHAE efficiency metric, and monitors contamination with reasoning-chain leak probes. Code: apps/trust/py/src/trust/forge/.

The whole thing is deterministic and offline. Every parameter was inherited from somewhere: ARC's 1/10,000, Airbnb's kappa≥0.85, the 5x action budget. Inherited parameters are opinions until measured.

Airbnb's two 2026 posts are the direct ancestors of this project. "Eval-driven development" gives the layered stack (programmatic checks, then LLM-as-judge, then human) and the judge-calibration loop (golden dataset, kappa/alpha agreement, refine, re-run). "From weeks to a day" gives deterministic caching and the epistemic/aleatoric noise split. The layered stack, the judge-calibration loop, and the eval cache are load-bearing here.

What both articles treat as sound practice (target kappa in the high-80s to 90s, 3-5 well-calibrated evaluators) is calibrated for product evaluation. There the loop closes on human judgment quickly. Benchmark construction is a narrower case: fewer free variables, a ground-truth oracle I can query repeatedly, and a payoff (S1, S6) for running the sensitivity analysis all the way out. S6 asks the budget question directly. If a calibrated LLM judge pre-screens task difficulty before spending human-attempt budget, how much of that budget can I cut before the difficulty model starts lying? The real judge itself picks up a compression bias. I found that by running the pipeline, not by assuming the judge is neutral.

What the studies found

S1: difficulty compression weakens the signal, it does not flip it

The generator emits a difficulty prior per task (trajectory length times arg richness). I compressed and stretched it by a scale factor. The previous version reported a sign flip at low compression. That result does not replicate once the Spearman tie-averaging bug and the generator's duplicate-signature bug are both fixed.

scale	calibration monotonicity (corrected)	calibration monotonicity (original, before the audit)	split predictability
0.1	-0.128	+0.26 (claimed sign flip)	0.8
0.3	-0.058	+0.22 (claimed sign flip)	0.9
0.6	-0.297	-0.13	0.95
1.0	-0.688	-0.59	1.0
1.5	-0.726	-0.56	1.0
2.0	-0.430	-0.12	1.0

The corrected result is more modest. Compression weakens the calibration signal. Monotonicity drops from -0.69 at natural scale to -0.06 at 10x compression. It does not invert. A generator with heavily compressed difficulty is still a worse generator; it is not the "the model actively lies" failure the buggy measurement reported. A tie-blind Spearman combined with a generator that had structural duplicate collisions produced that sign flip, both fixed in this pass. Split predictability's robustness (0.8-1.0 across the whole sweep) is unaffected by either bug.

S1: the human calibration bar has a Goldilocks zone

I varied the human population's skill, the oracle's d0. This table's shape was already correct before the audit.

d0	pass rate (2-of-10)	calibration monotonicity
0.2	0.37	-0.567
0.35	0.66	-0.675
0.5	0.92	-0.688
0.65	0.99	-0.575
0.8	1.00	-0.498

If everyone passes (d0=0.8) or most fail (d0=0.2), the calibration signal degrades. There is a real operating window. ARC's "most candidates rejected" intuition is a measurable curve.

S2: sample size until the benchmark stops wobbling

Draw random subsets, score, repeat. Greedy solver's score variance versus subset size:

tasks	greedy RHAE (mean ± stdev)
30	0.316 ± 0.064
60	0.360 ± 0.050
120	0.328 ± 0.053
240	0.304 ± 0.028
480	0.316 ± 0.014

Variance drops roughly 4.6x from 30 to 480 tasks. Previously reported as ~6x, a real but modest change from the generator and RHAE fixes, same conclusion. Below ~100 tasks, a benchmark's score is noise with a mean. This is METR's evals-as-scaling applied to benchmark construction. A benchmark isn't trustworthy until I've measured how much benchmark I need.

S3: structural features predict human solvability better than the buggy pipeline reported

K-fold cross-validation of the difficulty model: fit on train folds, measure rank correlation between fitted difficulty and measured solve rate on held-out folds. This is the other finding that reversed. The original claim, that structural features can't predict human solvability, that the calibration oracle is required because the difficulty model overfits, was substantially a measurement artifact.

feature set	out-of-fold (corrected)	out-of-fold (original, before the audit)
all features (calls + tools + arg complexity)	-0.662	-0.15
n_calls + n_tools	-0.634	n/a (collinear with n_calls alone before the fix)
n_calls only	-0.418	-0.18
arg complexity only	-0.345	-0.19
n_tools only	-0.323	n/a (identical to n_calls before the fix)

In-sample on the same 230-task pool: -0.685. Out-of-fold: -0.662. That's a 0.02 gap, not the 0.44 gap ("-0.59 in-sample vs -0.15 out-of-fold") originally reported. The original comparison was apples-to-oranges besides. It compared S1's in-sample number, fit on a different 120-task pool with the full pipeline's gauntlet and calibration bar applied, against S3's out-of-fold number on a differently-constructed 230-task pool with neither. Once the same pool is used for both sides, the difficulty model generalizes almost perfectly.

Two bugs drove the original overstated gap. A tie-blind Spearman correlation, biasing every number in this table. And a generator defect where n_calls and n_tools were mathematically identical for every task: the tool-sequence generator never repeated a tool within one task. The single-feature ablation was silently testing the same feature twice under two names. Fixing the second bug is also why n_tools_only and n_calls_only are different numbers now.

The corrected finding: on this task domain, simple structural features (how many tool calls, how many distinct tools, how much argument complexity) predict human solve rate well. They generalize out of fold. That is the opposite of what a benchmark builder wants to hear if the goal is that structural proxies are worthless and only real humans count. The proxies aren't worthless here.

Whether that generalizes beyond this deliberately simple tool-use domain to something ARC-AGI-3-scale is open; simple domains may just have easy-to-learn difficulty surfaces. ARC still runs 400 humans through 2,893 attempts. That's still the right call for a domain where the difficulty surface isn't this legible. "The calibration oracle is the only thing that grounds the difficulty axis" was too strong a claim from this data.

S4: a contamination monitor I can validate

The leak probe fires when a task's value-level signature appears in an independently-built knowledge base. A prior version built that knowledge base from the same task population being tested, which made "fire on leaked" guaranteed by construction. Fixed: the knowledge base now comes from a disjoint, separately-generated reference pool. Leaked tasks are deliberately mutated to match entries drawn from it.

leaked fraction	fire on leaked	false-fire on clean
5%	1.000	0.0000
10%	1.000	0.0000
20%	1.000	0.0000
30%	1.000	0.0000
50%	1.000	0.0000

Detection is perfect at every leak rate with zero false-fires. Cleaner than the previous tautological version. That one reported a slowly-growing false-fire rate (0.0054 at 20%, 0.0087 at 50%), which turned out to be an artifact of the generator's duplicate-signature bug. Duplicates are gone and the knowledge base is external. The monitor's operating curve is real.

S4b: the real-surrogate contamination probe

S4 validates the leak-probe mechanism. This asks whether a real model actually leaks. run_llm_leak_probes sends a real model (DeepSeek deepseek-chat) a task's structural hint (tool names and argument keys only, every value withheld). It checks whether the completion reproduces the withheld values anyway, with a word-boundary match, not a bare substring check. A prior version would count a value of 3 as reproduced by any text containing the digits 13. That's the Gemini-3 scenario from the ARC report: a model that knows content it was never shown.

Across 20 never-leaked tasks, 0 of 20 probes fired, with 0 network failures. The model has never seen this benchmark. It can't have: these tasks are generated fresh, per run, from a mock tool registry that doesn't exist anywhere on the public internet. The first run of this pass hit 6/20 timeouts against DeepSeek. The same audit pass logged completions and failure counts explicitly, instead of letting a timeout produce the same fired=False as an honest answer, which is what caught it. Rerun with a longer timeout, the result is clean: false-fire rate 0.0000, matching S4's synthetic clean-baseline, and this time a real measurement rather than a partially-masked outage. Code: s4b_real_surrogate.py, run_llm_leak_probes.

S6: human-attempt budget a judge saves

This extends Airbnb's judge-calibration loop into a budget question. If a calibrated LLM judge pre-screens which tasks are unambiguously easy or hard from the prompt alone (no trajectory, no oracle), how many of the 10 human attempts per task can I skip before the difficulty model starts lying? The judge here is a real model (DeepSeek), not simulated.

screening threshold	tasks auto-resolved	budget used	calibration monotonicity
0.0 (no screening)	0/60	100%	-0.789
0.1	9/60	85.0%	-0.789
0.2	14/60	76.7%	-0.789
0.3	46/60	23.3%	-0.634
0.4	46/60	23.3%	-0.634
0.5 (max screening)	60/60	0%	-0.522

A real judge saves real budget. Thresholds 0.1-0.2 cut 15-23% of the human-attempt budget at zero monotonicity cost (-0.789, identical to the unscreened baseline). The unexpected result, unchanged by the audit: the real judge's own score distribution is compressed. Across all 60 tasks it never rated anything above 0.6 on a 0-1 difficulty scale. It has no confident hard tail, only a trivial one and an undifferentiated middle. That's why the budget drops so sharply between thresholds 0.2 and 0.3 (14 → 46 auto-resolved). It is also why monotonicity only degrades moderately even at maximum screening (-0.522, not collapsing to zero).

The judge is applying a compression pattern to itself, similar in shape to what S1 originally mismeasured in the synthetic generator. A coincidence worth naming, given S1's own version of that pattern didn't survive the audit. A judge's difficulty ratings still deserve the same sensitivity check S1 applies to the generator.

Rerunning this study against the live DeepSeek API twice produced slightly different auto-resolved counts each time (9 vs an earlier run's 8, 14 vs 12) even at temperature 0. The rest of this pipeline is byte-reproducible by construction. A real judge calling a real API is the one place that guarantee doesn't hold. The numbers above are from the run that landed in the committed artifact, not an average. Code: s6_judge_ladder.py.

The real-LLM run: the first honest human-vs-AI point on this benchmark

Every number above compares synthetic solvers (random, greedy, perfect) against each other or against the simulated oracle. None of it says anything about a real system. I wired LlmSolver to DeepSeek's deepseek-chat endpoint and ran it against the live benchmark. Two fixes changed what this run could measure: bearer-token auth (it previously had none, so real endpoints silently failed every call), and recovery from a single bad tool call (a prior version treated one hallucinated tool name the same as a dead endpoint).

solver	solve rate	RHAE
random	0.0%	0.000
greedy (prompt-following heuristic)	25.8%	0.258
DeepSeek (real model)	95.2%	0.952
perfect	100%	1.000

DeepSeek solved 95.2% of the 75-task run (62 survived the gauntlet + 2-of-10 calibration bar) at RHAE 0.952. An earlier, bug-affected run of this same pass reported 90.7%. Both the gauntlet and the RHAE computation changed under it since that number; they aren't directly comparable. This is the first number produced by the corrected pipeline.

Split predictability on this 62-task calibrated population came in at 0.7, below the 0.8 acceptance bar. A real instance of S2's lesson: a calibrated population this size is too small for a stable predictability read. A local Ollama endpoint was not available in this environment to run the same comparison against a smaller open-weight model. Code: LlmSolver, run artifact: benchmark-agentic-tooluse-deepseek.json.

The absorption thesis, revisited

The core claim is that benchmarks die of absorption, not saturation. The previous version of this section leaned on S1's sign-flip and S3's overfitting gap as the mechanism-level evidence. Both of those specific results didn't survive the audit, so that argument doesn't stand as written. I'm not going to rebuild a version that happens to fit whatever data is left.

What holds: the pipeline's own difficulty axis is real and predictable (S3, corrected). When a difficulty model's calibration does break, on a domain this legible, that's a signal rather than noise from an unmeasurable axis. Split predictability holds across every parameter stress test (0.8-1.0 throughout S1's corrected sweeps). Public and private splits stop predicting each other when something upstream breaks, independent of the absorption-versus-saturation framing.

What I no longer have clean evidence for: a mechanism-level signature that distinguishes absorbed from saturated within this pipeline's own instrumentation. The ARC report's Gemini-3 anecdote — a verification model using a color mapping it was never told — is still the clearest real-world example of absorption as a distinct phenomenon from raw capability saturation. This project's own studies, once measured correctly, don't add a new quantitative signature on top of that anecdote. That's a narrower claim than the original version made.

SOTA evals literature: what's changed since ARC and Airbnb

Beyond ARC-AGI-1/2/3, JudgeBench, and METR, four threads from 2025-2026 evals research sit on what the Foundry does and doesn't cover.

Agentic trajectory evaluation. Recent work (a 2026 survey on LLM-based agent evaluation, and specialized tools like TrajAD for runtime trajectory-anomaly detection) has moved past final-answer correctness toward step-level trajectory scoring. It looks at tool choice, argument correctness, decision order, and when an agent should have stopped but didn't. The Foundry already does the strong version of this for free. exact_trajectory_verifier and reference_run_verifier check the full call sequence, not just the outcome. RHAE's action-count penalty punishes an agent that looks busy without being efficient, a claim now backed by a non-degenerate metric. What it doesn't do yet is anomaly localization mid-run, TrajAD's contribution. The verifiers are binary pass/fail on the completed trajectory; they are not per-step diagnostics.

Reward hacking and verifier gaming under RL. 2026 work on RLVR ("LLMs Gaming Verifiers," reward-hackability audits of code-RL training environments, adversarial hacker-fixer loops for hardening agent benchmarks) documents models that learn to exploit what the verifier fails to enforce. They monkey-patch test harnesses, delete assertions, and exploit credit leakage from spurious reasoning traces. This is the generate-verify-train loop from the ARC report, formalized. The Foundry's verifiers are programmatic and check exact trajectories and values, which closes the most common hacking surface. A model can't monkey-patch a Python assertion it never has access to. Nothing in the pipeline currently tests whether a solver under a training loop, not just a one-shot eval, starts exploiting the verifier's matching logic instead of solving the task. That's untested, and it's the single most important gap this literature surfaces.

Contamination detection beyond n-gram matching. 2026 work (DyePack's backdoor-based flagging, distributional and black-box sample-level detection methods) responds to a real weakness. N-gram and exact-match detection misses rephrased leaked content entirely, and it gets more expensive as pretraining corpora grow. The Foundry's contamination monitor (S4/S4b) is value-level signature matching, closer to DyePack's spirit than plain n-gram. It has no defense against a model that reproduces the right values in a rephrased structure, and no backdoor-style provable-flagging mechanism. Both are natural S7/S8 candidates.

LLM-as-judge reliability. "Reliability without Validity" is part of a larger 2025-2026 body confirming judges are unreliable in specific, structural ways. Position bias is tied to the quality gap between compared outputs, no judge is uniformly reliable across benchmarks. Frontier judges fail over half of dedicated bias tests. S6's finding, that the real judge's own difficulty ratings are compressed and never use the top of its scale, is a small, concrete instance of that warning, caught empirically. That's why the essay leads with the audit rather than treating any single study's result as self-certifying.

What the audit taught

An adversarial pass on this pipeline found two bugs that had inverted or overstated the two most interesting findings in the original version of this piece. Sensitivity analysis on inherited parameters and adversarial review of my own measurement code are the same discipline aimed in two directions. Skipping the second one meant publishing a sign flip and an overfitting claim that were substantially bugs.

A tie-blind rank correlation is a silent, structural bias, not a rounding error. Every headline number in this pipeline goes through Spearman, and the fitted difficulty values are heavily tied in practice. An estimator that breaks ties by array index instead of averaging them doesn't fail loudly. It reports a different number, in a direction that depends on the tie structure.

A metric can look right on paper — formula matches ARC's exactly — and still measure nothing. RHAE's math was correct; the human-baseline generation that fed it was decoupled from the thing being measured. The metric always saturated its own cap. Verifying a formula against its citation isn't the same as verifying the pipeline that feeds it.

The stratification property is still the most reliable canary in this pipeline. Split predictability survived every parameter stress test, before and after the audit (0.8-1.0 throughout). When a benchmark's public and private splits stop predicting each other, something upstream broke. The difficulty model is still the first suspect.

Reproducibility is free if I design for it, verified twice now. The entire pipeline, from generation to scoring, is byte-reproducible across seeds by construction. Every fix in this pass was verified against a fresh, from-scratch regeneration of every artifact.

I can validate a contamination monitor, including catching my own tautologies. The first version of S4's monitor validated correctly by construction: the leaked knowledge base was built from the same tasks it tested. Fixed to check against a genuinely external reference corpus, it's now a real detector with a real operating curve.

A judge saves real budget, cleanly, once the measurement is trustworthy. S6's real-DeepSeek judge cut human-calibration spend by 13-20% at zero monotonicity cost. That is cleaner than the "essentially no cost" the pre-audit version reported, because the underlying correlation estimator is no longer biased. Airbnb's measure-judge-agreement principle, applied against this pipeline's own attempt-budget cost function.

The honest limits

The human oracle is still simulated for the main benchmark run. Real-human calibration now has infra behind it (CalibrationQueue / RealHumanOracle, protocol-conformant with SimulatedOracle and now honoring its n_attempts argument correctly, proven by a mock-backed integration test), but zero live human data. That is the biggest remaining gap, and it is deliberately not faked: RealHumanOracle.calibrate() raises instead of fabricating an outcome for a task whose attempt quota isn't met.

The obvious objection is that I built the benchmark I am scored on. That is true, and it is why the generator, the oracle, the leak probe, the scorer, and every study artifact are published. A simulated oracle is also not ARC's 400 humans / 2,893 attempts. I am not going to pretend otherwise.

The real-LLM run used DeepSeek only. A local Ollama endpoint, for a comparison against a smaller open-weight model, was not available in this environment. The run that would produce that comparison is queued, not run.

S4b's real-surrogate contamination probe reports a clean baseline (0% false-fire, 0 network failures) on a model that has genuinely never seen this benchmark. It says nothing about detection on a model that has leaked, which requires either an actual leak event or a deliberate fine-tune, neither of which happened here.

S3's reversed finding is measured on a deliberately simple, single-domain task space. Whether it generalizes to harder or more heterogeneous domains, where a difficulty surface might genuinely not be structurally legible, is open.

Harder task domains (compositional multi-step, retrieval-grounded, classification), and the generality question of whether this methodology transfers beyond tool-use, were scoped out of this pass. The task domain remains narrow by design: tool-use over a mock registry, so the methodology stays the subject, not the tasks. The benchmark itself is still not competitive with ARC's in difficulty. It's a proof that the methodology runs end-to-end with every property measured, now including an adversarial audit of the measurement itself.

All studies are reproducible: uv run python -m trust.forge.studies.s1_sweeps (and s2/s3/s4/s5/s6), uv run python -m trust.forge.studies.s4b_real_surrogate --tasks 20 (needs MODEL_PROVIDER_BASE_URL/MODEL_PROVIDER_MODEL_ID/MODEL_PROVIDER_API_KEY), artifacts in docs/validation/studies/. The real-LLM benchmark run: uv run python -m trust.forge.cli bench --llm --llm-base <endpoint>/v1 --llm-model <model> --llm-name llm-<label>.