Ornith-1.5 Wrote Its Own Training Data. Distribution Match Won

The self-proposed curriculum loses alone. It wins only as a supplement.

The reward that generates it has no term for the distribution it will be deployed on.

What was being scored

Ornith-1.5 is the current claim that models can teach themselves. The release table shows what that claim does to the models. The 397B flagship scores Terminal-Bench 2.1 at 86.1 and DeepSWE at 56.0. Even the 9B beats a dense model more than three times its size on agentic coding. The models are strong. That was never the number I wanted.

Ornith-1.5 also proposes the tasks it trains on. It writes the curriculum, the scaffolds, and the rollouts, then optimizes all three with GRPO. The claim I wanted scored is that a self-improvement loop that invents its own training data produces data worth training on. Nobody in the release measures that — people treat the leaderboard as a score on self-proposed data, when the scoreboard is whether that curriculum beats a distribution-matched set on a held-out oracle.

So I built the instrument that asks that question, pointed it at a verifiable niche, and let a just-released self-improving model propose the set. Then I measured the proposals against the boring alternative.

The instrument

It is the one I've been building all year. Ground truth is recomputed by a deterministic oracle that never reads the labels it is being scored against. Severity is weighted, because catching a $5 million misapplied payment must outweigh catching forty cents. Contamination is a SHA-256 contract, because a benchmark is only as good as the guarantee that the model never saw the answer. Determinism is complete, so the only variance left is the sampling marginal. That's the ReconForge methodology, and the LossBench thread underneath it.

The objective is expected loss, not accuracy. Whenever all severities cost the same, expected-loss ranking collapses into accuracy ranking, the flat-cost special case that every leaderboard measures without saying so.

For readers new to the series, the scoring works like this. Exception severities weigh HIGH 1.0, MEDIUM 0.6, LOW 0.3. Severity-weighted recall is the weight of caught exceptions over the weight of planted ones, counted only where the exception type matches. Precision counts unmatched flags as false positives. Exact-match adds the corrected-figure check, so an exception only scores if the model's correction ties to the oracle's. Unparseable output misses everything. I evaluated at seed 777, n=1000, severity-weighted, with zero contamination.

The niche is AIA G702/G703 construction pay applications: the monthly request for payment, with its schedule of values, retainage, prior payments, change orders, and waivers. It is a closed world of recomputable arithmetic. Every line must tie, and the oracle recomputes every tie. The oracle is a wall rather than a judge, and that is the load-bearing rule of everything below. No task enters training without passing the oracle gate. A proposed task the oracle rejects is discarded rather than repaired.

The obvious objection is that the held-out benchmark is also mine. That is true. It is why the generator, the oracle, the signatures, the scorer, and every published row sit in the repo and on the Hub. If the instrument is wrong, it is wrong in public.

The loop

Two tiers. On top sits the proposer, Ornith-1.5, asked to design training tasks for the niche. It emits a parameter vector (the exception set to hide, the difficulty, the schedule shape, the retainage state) and my seeded generator renders that vector into a concrete, always-valid pay application. Ornith's own reward for proposing is the product of three terms, and each maps one-to-one onto the instrument:

• V, validity: does the proposed task pass the oracle gate? Hard gate: V = 0 makes the whole reward zero.
• D, frontier difficulty: sample rollouts of the current policy on the proposed task; reward tasks whose success rate lands near a frontier target (p*=0.2). Neither trivial nor impossible.
• N, novelty: how dissimilar is the proposed task from everything already in the buffer, measured at the parameter level and at the content level with MinHash.

Accepted proposals feed the task buffer, and that buffer is the training set for the lower tier: the policy that reviews pay applications. For this study the policy is a text model small enough to fine-tune on a laptop. The production version is the multimodal 27B that reads the rendered scan. The study isolates one variable, the curriculum. Same policy, same oracle, same held-out benchmark. Only the training distribution varies.

What the proposer proposed

I ran the real model. Ornith-1.5-9B in bf16 on a cloud GPU, and the same weights as a Q4_K_M GGUF on a 16GB laptop, sampled at temperature 0.9 throughout. The bf16 MLX variant needs ~18GB and doesn't fit. The GGUF does. I gave the proposer the oracle, a specification of the domain, and the reward above, and asked it to fill a curriculum.

It produced valid, verifiable tasks. In these two P0 runs every proposal parsed and passed the oracle gate, with no malformed or unverifiable task in either. That result belongs to the bf16 and MLX-Q4 stacks. The Ollama-served grows later in this post broke the parser — the serving stack changed, not the model.

Ornith also proposed tasks harder than anything a uniform baseline or a stand-in proposer produced. Two-injection combinations no baseline attempted, such as an overpayment layered on a change-order overrun, or a double-counted line combined with a missing waiver. The scaffolds had checks that matched the injected types, and the rationales read like a grader explaining the trap. On difficulty it converged on the middle of the range, 0.55–0.68, the band where a policy is learning rather than guessing.

The reward's D term measures difficulty against the current policy's rollouts, and the current policy was a placeholder. Under that placeholder the frontier sat at difficulty 0.8, which a generic stand-in proposer happened to hit. Ornith's harder combinatorial tasks landed just short of it. "Harder than the baseline" and "on its own target frontier" are different claims, and the second one depends on who is estimating the frontier. Without that caveat, P0 reads as an Ornith win it hasn't earned.

The head-to-head

I exported the accepted curriculum into a contamination-clean training set: 58 Ornith-proposed tasks (48 grown locally through the Q4 model, 10 from the bf16 run), with zero exact signature leaks and zero MinHash near-duplicates against the held-out seed-777 benchmark. Two controls sat next to it. The hand-seeded set is 223 tasks, and the blend is 281.

The Ornith set is distribution-shifted against the benchmark it will be measured on.

• difficulty 0.66 on average vs 0.49 for the benchmark
• no easy tasks at all; 35 of 58 are two-injection combinations (60%), vs 24% in the benchmark
• no PASS tasks; the proposer never asked the model to say "this is fine"
• heavily concentrated in two retainage states

The hand-seeded set is the distribution-matched control by construction. Mean difficulty 0.48 against the benchmark's 0.49. One asymmetry to keep in mind: its mix runs 32% PASS and 4% double-injection against the benchmark's 23% and 24%. So the head-to-head has real tension in it, with the harder, more informative self-proposed curriculum on one side, the boring matched one on the other, and the blend of both.

I trained a Qwen3-1.7B-4bit policy, LoRA, 8 epochs on each curriculum, ~3.3GB peak on the M5. I evaluated on the same 1000-task held-out benchmark, same oracle, same severity-weighted scoring.

95% bootstrap CIs (10,000 resamples, seed 11, over the same 1000 tasks). Ornith sw-recall 0.241 [0.214, 0.268] sits below hand-seeded 0.318 [0.290, 0.347] with no overlap. Blend 0.334 [0.306, 0.363] overlaps hand-seeded on recall. Precision is the metric that separates the blend from the matched base: 0.374 [0.342, 0.406] vs 0.308 [0.279, 0.337]. Point estimates that "blend wins every row" are true; the interval that survives is precision, and Ornith-only losing.

Before trusting that table I ran the confound check. The rows are not at equal N: 58 Ornith tasks against 223 hand-seeded and 281 blended. "Blended wins" might be "blended has more data." Fifty-eight tasks cannot beat 223 on volume alone. I cut all three curricula to 58 tasks with an identical 49-train/9-valid split and an identical 8-epoch budget, then ran a stratified variant, because a random 58-task draw is a noisy thing.

The honest reading of the small-N run: 58 tasks cannot train this task well. Every row collapses to 0.05–0.17 severity-weighted recall against 0.24–0.33 at full size. These rows carry no confidence intervals. Each arm is a single curriculum draw, and most orderings inside that band are noise.

A random 58-task blend scores below a random 58-task hand-seeded set. Ornith edges that same hand-seeded draw. A stratified hand-seeded set, forced to keep the train prior's shares (20 PASS of 58) and all nine exception types so it sees fewer of each, scores below a random one. That is what an underpowered sample looks like.

One direction is worth carrying forward — the stratified blend. A coverage-preserving mix of matched tasks and Ornith's combinatorial ones is the best small-N row (0.169 severity-weighted recall, 0.095 exact-match), pointing the same way as the full-size table. Read it as a hint, not a result. The confound check does not overturn the full-size result.

What the numbers support

Ornith-only underperforms. The self-proposed, harder, distribution-shifted curriculum generalizes worse on the held-out benchmark than the distribution-matched hand-seeded set: worse severity-weighted recall, worse precision, worse exact-match. The equal-N check does not rescue it. At 58 tasks every curriculum is underpowered, and the best small-N row is the stratified blend, not Ornith alone.

I've now watched this happen in a second domain. ReconForge's B2 was the first. Rebalance the training mix toward the classes you care about, and recall on the classes you cut collapses for almost nothing on the class you boosted. The training distribution has to match the deployment distribution. Ornith's loop, optimizing for frontier difficulty, manufactures a shifted distribution by design (0% easy, 60% two-injection, no PASS tasks). That shift is what costs it on the benchmark it never saw.

Blended wins every point estimate. The blend, matched base plus Ornith frontier tasks as a supplement, beats both parents on every row of the point-estimate table. The bootstrap is stricter. Blend vs hand-seeded overlaps on severity-weighted recall (0.334 [0.306, 0.363] vs 0.318 [0.290, 0.347]) and barely clears on precision (0.374 [0.342, 0.406] vs 0.308 [0.279, 0.337]). The lower bound clears the other's upper by 0.005. One comparison picked out of many — treat it as the lead claim, not settled fact.

That precision swing still has a mechanism. Ornith-only never saw a PASS task, so it learned to flag everything, and its two-injection combinations taught it a style of over-flagging. Its precision collapsed (0.111 [0.098, 0.124], no overlap with either matched arm). The hand-seeded base taught the model when to say "this is fine." Adding the Ornith tasks on top of that restored the discrimination the harder tasks were supposed to teach, without destroying the restraint the matched base had already installed.

A second mechanism plausibly works alongside. The blend restores double-injection coverage the hand-seeded set was thin on (16% against its 4% and the benchmark's 24%), so some of the recall gain may be plain distribution completion. Both readings agree on the upshot. The self-generated tasks work as a supplement, not a replacement.

The D term is distribution-blind. Ornith's reward for proposing is a product of validity, frontier difficulty, and novelty, with no term for the deployment distribution. "Harder" is not the same as "worth training on," and not even the same as "on the frontier the deployment occupies." The loop pushed difficulty up and pushed the composition away from the benchmark, and both moves cost it. The fix is to give the loop a deployment anchor rather than abandon self-improvement. Blend the proposed tasks with matched data, or retarget D at the deployment distribution so "frontier" means where the production errors live. That is the part I would want any self-improvement reward to steal.

What broke when I tried to improve the proposer

The natural next move is to train the proposer itself. Run the same GRPO loop the Ornith blog describes, so the proposer learns to propose better tasks. I built that machinery in the repo's curriculum package: the reward wrapper, group-relative advantages with the DAPO decoupled clip, a per-completion LoRA trainer on PyTorch and transformers, unit-tested end to end. Then the training stack got in the way.

Training the proposer crashes in the with-grad forward. Inference is clean. Generation, parsing, reward, everything works. The moment you ask for gradients, Qwen3_5GatedDeltaNet, the hybrid linear-attention block Ornith-1.5-9B shares with the whole current Qwen line, reshapes a zero-element tensor — and dies.

I checked the usual suspects. It is not a quantization problem (bf16 crashes too), not a cache problem (on and off), not a masking problem (explicit position ids and attention masks), and a real small qwen3_5 model trains fine. So the batch-0 crash I hit on the 9B looks like a harness- or scale-level artifact rather than a clean architecture bug, and I am not going to claim otherwise.

The related bugs that are real (the stale rope-deltas batch mismatch, the flash-attention-with-3D-position-ids crash, the packing leak, the bf16 A_log edge) are already filed and fixed upstream. A just-released model family can be a strong proposer and still not be a trainable loop in today's harness. You get the capability before you get the training stack.

What didn't work

The full-size table left an obvious next question: whether the loop can be steered so the self-proposed set matches the train prior, and still beat naive Ornith at equal N. I ran that as a sequence of grows. Three of those runs are not the method. The one that is a method, the clean protocol, lost.

Start with more naive Ornith and no novelty floor. I grew the accepted buffer from 58 to 184, and severity-weighted recall went 0.241 → 0.163 [0.139, 0.186]. Late accepts had novelty 0.01–0.07, so the extra rows were near-duplicates. Of those 184, only 24 still clear a novelty floor of 0.3. Growing a self-proposed buffer without a floor added copies, not information.

Next I fitted a fourth term M against the held-out seed-777 mix, multiplied it into an accept-any-positive product, and told the proposer to match it. That is eval leakage, and those rows stay exploratory. Even on its own terms M did not steer. Accept was r > 0, so a positive D×N let any mix through, and count_match scored the common class (single-injection) lower for being far from uniform. Composition stayed 0% PASS / 66% two-injection. The naive profile drifted further; the original 58-task buffer sits at 60%. The leak alone is enough to discard the run. The control that filters is a hard mix constraint against the train prior, not a fourth multiply.

The proposer then would not emit parseable JSON. Ollama's 9B-q4 wrote "seed": 84402e075f20 (unquoted hex) and oversized ints. json.loads failed, parse_proposal returned None, and a 180-try mix grow accepted one task. I repaired unquoted and coerced seeds in the parser. The grow that started before the repair is not a result.

Mix deadlocked after one single. Train prior is 32% PASS / 64% single / 4% double: the measured composition of the hand-seeded train set, which the mix constraint fits directly. After one accepted single, running mix is 100% single, so every further single was over-quota. The 9B kept proposing mid-difficulty singles. I stopped enforcing quotas until the buffer had six tasks, since you cannot estimate a 3-class mix from n=1, and on a mix reject I sampled a proposal from an under-represented cell (deficit-fill). Feedback in the prompt was not enough. The controller had to emit PASS itself. Of 22 mix accepts, 16 were fills, not LLM proposals.

Then the clean protocol, equal N=18. Train prior (never seed-777), mix constraint, novelty floor 0.3, D from a 1.7B residual-error profile on a train sample of 80. The floor of 0.3 capped the grow at 18–24 diverse tasks. I picked 0.3 off the 184-grow, where late accepts sat at 0.01–0.07 and 0.3 is the lowest round cutoff well clear of them. I did not sweep it, so treat it as a default, not a tuned constant. Equal N locked to 18 (15 train / 3 valid). Contamination audit vs seed-777: exact 0, near 0. 95% bootstrap CIs, 10k, seed 11:

Mix 0.075 [0.059, 0.093] sits below naive 0.135 [0.114, 0.156] with no overlap. Mock-anchored 0.125 [0.104, 0.147] overlaps naive. Error-targeted 0.031 [0.022, 0.042] sits below hand-seeded 0.060 [0.044, 0.076], and below blend's point estimate with intervals touching. Every arm here lands in the 0.03–0.14 band. None of them touch the full-size bars (hand-seeded 0.318, blend 0.334).

One signal in that table I cannot explain — naive Ornith holds a 0.98 parse rate while the matched arms sag to 0.84–0.88, as if scaffold variety teaches output format faster than matched content does at tiny N. Worth a follow-up, not a conclusion. And N=18 is what a 0.3 novelty floor yields on this generator. It is not a smaller version of the 58-task run. Relax the floor and the buffer fills back up with near-duplicates.

The leaked M-term grows, the 184-without-floor run, and the N=18 adapters are not weights I am releasing. The method that survived contact with the numbers is still the full-size blend.

Honest limits

The object-level policy is a 1.7B text model, not the 27B multimodal production model. It reads the pay application as structured text, not as a rendered scan, so OCR is out of scope. The laptop 1.7B is not the 27B. The Ornith curriculum is 58 tasks; 60% two-injection is 35 tasks.

The full-size comparison is not at equal N, and equal-N at 58 is underpowered. Growing naive Ornith to 184 without a novelty floor hurt (0.241 → 0.163). The clean protocol at N=18 did not beat naive Ornith. The pow-* grows, meaning the M-term fit and its anchored siblings, fitted their prior on seed-777. They are not a methods result, and those adapters are not on the Hub.

All data is synthetic. Reported metrics carry 95% bootstrap CIs (10,000 resamples, seed 11). The 100% oracle pass rate is a low bar in a closed-world arithmetic domain, and it may say more about the task than about the proposer.

A leaderboard tells you how strong a model is. It does not tell you whether the data a model proposes for itself is worth training on. This instrument separates those two questions, and on this niche the harder self-proposed tasks lose on their own and win only as a supplement to a distribution-matched base.

Code is at github.com/caiotheodoro/plumb. The instrument and the three full-size adapters are on the Hub: caiotheodoro/plumb, plumb-handseeded, plumb-ornith, plumb-blended. Collection: caiotheodoro/plumb. The dataset's evals config holds every published row and its CI, so each table above is reproducible without rerunning anything. N=18 mix/error weights are not there.