# Ornith-1.5 Wrote Its Own Training Data. Distribution Match Won
> Ornith-1.5 proposes its own training curriculum. I audited it with contamination-controlled data on a verifiable niche, construction pay-app review, and found the harder tasks it invents only help as a supplement, not a replacement, for distribution-matched data.
- Published: 2026-08-21
- Reading time: 19 min read
- Tags: ML, Agent Systems
- Author: Caio Theodoro (https://caio.theodoro.dev/about.md)
- Canonical HTML: https://caio.theodoro.dev/blog/ornith-curriculum-audit-distribution-wins
- Source repository: https://github.com/caiotheodoro/plumb
- Hugging Face: https://huggingface.co/collections/caiotheodoro/plumb-ornith-wrote-the-curriculum-match-won-6a89f2ea18c97d11b0238529
---
[Ornith-1.5](https://ornith.ai/ornith_1_5.html) is the current claim that models can teach themselves. Its release table shows what that claim does to the models: the 397B flagship scores [Terminal-Bench 2.1](https://www.tbench.ai/benchmarks/terminal-bench-2-1) at 86.1 and [DeepSWE](https://arxiv.org/abs/2607.07946) at 56.0, and even the 9B beats a dense model more than three times its size on agentic coding. The models are strong. That was never the question I wanted answered. Ornith-1.5 also proposes the tasks it trains on. It generates its own curriculum, its own scaffolds, its own rollouts, and optimizes all three with [GRPO](https://arxiv.org/abs/2402.03300). The claim I care about is that a self-improvement loop, left to generate its own training data, produces data worth training on. Nobody in the release measures that. The leaderboard scores the model; it never scores the curriculum. So I built the instrument that does, pointed it at a verifiable niche, and let a just-released self-improving model propose the training set. Then I measured whether the proposals beat the boring alternative.
| | |
| --------------- | -------------------------------------------------------------------- |
| **Measured** | the curriculum, not the model |
| **Policy** | Qwen3-1.7B-4bit, LoRA, 8 epochs, identical for every arm |
| **Eval** | 1000 held-out tasks, seed 777, severity-weighted, zero contamination |
| **Ornith-only** | 0.241 severity-weighted recall, below hand-seeded with no CI overlap |
| **Hand-seeded** | 0.318, the distribution-matched control |
| **Blend** | 0.334 recall (overlaps hand-seeded), 0.374 precision (does not) |
The self-proposed curriculum loses on its own and wins only as a supplement. The reward that generates it has no term for the distribution it will be deployed on.
## The instrument
The instrument is the one I've been [building all year](https://caio.theodoro.dev/blog/building-benchmarks-like-arc-measuring-whether-it-works). A benchmark whose numbers can be trusted, because its ground truth is recomputed by a deterministic oracle that never reads the labels it is being scored against; a severity-weighted scoring function, because catching a $5 million misapplied payment must outweigh catching forty cents; a zero-contamination contract enforced by SHA-256 signatures, because a benchmark is only as good as the guarantee that the model never saw the answer; and full determinism, so the only variance in the results is the sampling marginal. That's the [ReconForge methodology](https://caio.theodoro.dev/blog/reconforge-1-7b-beats-deepseek-on-the-money-metric), and the [LossBench thread](https://caio.theodoro.dev/blog/why-lossbench-significant) underneath it: the objective is expected loss, not accuracy, and whenever all severities cost the same, expected-loss ranking collapses into accuracy ranking, the flat-cost special case that every leaderboard measures without saying so.
The scoring, for readers new to the series: exception severities weigh HIGH 1.0, MEDIUM 0.6, LOW 0.3. Severity-weighted recall is the weight of caught exceptions over the weight of planted ones, counted only where the exception type matches. Precision counts unmatched flags as false positives. Exact-match adds the corrected-figure check: an exception only scores if the model's correction ties to the oracle's. Unparseable output misses everything.
The niche is [AIA G702/G703](https://aiacontracts.com/documents/g702-1992) construction pay applications: the monthly request for payment, with its schedule of values, retainage, prior payments, change orders, and waivers. It is a closed world of recomputable arithmetic. Every line must tie, and the oracle recomputes every tie. The oracle is a wall rather than a judge, and that is the load-bearing rule of everything below. No task enters training without passing the oracle gate. A proposed task the oracle rejects is discarded rather than repaired.
## The loop
The loop has two tiers. On top sits the proposer: Ornith-1.5, asked to design training tasks for the niche. It emits a parameter vector (the exception set to hide, the difficulty, the schedule shape, the retainage state) and my seeded generator renders that vector into a concrete, always-valid pay application. Ornith's own reward for proposing is the product of three terms, and each maps one-to-one onto the instrument:
- **V, validity**: does the proposed task pass the oracle gate? Hard gate: V = 0 makes the whole reward zero.
- **D, frontier difficulty**: sample rollouts of the current policy on the proposed task; reward tasks whose success rate lands near a frontier target (p*=0.2). Neither trivial nor impossible.
- **N, novelty**: how dissimilar is the proposed task from everything already in the buffer, measured at the parameter level and at the content level with [MinHash](https://en.wikipedia.org/wiki/MinHash).
Accepted proposals feed the task buffer, which becomes the training set for the second tier: the policy that reviews pay applications. For this study that policy is a text model small enough to fine-tune on a laptop; the production version is the multimodal 27B that reads the rendered scan. The study isolates one variable: the curriculum. Same policy, same oracle, same held-out benchmark. Only the training distribution varies.
```mermaid
flowchart TD
P["Ornith-1.5 proposer
emits a parameter vector"]
R["Seeded generator
renders it, oracle checks it"]
X["Discard"]
S["Score the proposal
D = frontier difficulty
N = novelty"]
B[("Task buffer")]
T["Train the policy
Qwen3-1.7B LoRA"]
E["Held-out benchmark
severity-weighted score"]
P --> R
R -->|"V = 0"| X
R -->|"V = 1"| S
S -->|"V x D x N"| B
B --> T
T -->|"rollouts"| S
T --> E
```
## What the proposer proposed
I ran the real model: [Ornith-1.5-9B](https://huggingface.co/ornith-ai/Ornith-1.5-9B) in bf16 on a cloud GPU, and the same model as a [Q4_K_M GGUF](https://huggingface.co/ornith-ai/Ornith-1.5-9B-GGUF) on a 16GB laptop, sampled at temperature 0.9 throughout. The bf16 MLX variant needs ~18GB and doesn't fit; the GGUF does. The proposer was given the oracle, a specification of the domain, and the reward above, and asked to fill a curriculum.
It produced valid, verifiable tasks. In these two P0 runs every proposal parsed and passed the oracle gate, with no malformed or unverifiable task in either. That result belongs to the bf16 and MLX-Q4 stacks; the Ollama-served grows later in this post broke the parser, and the serving stack is what changed, not the model. Ornith proposed tasks harder than anything a uniform baseline or a stand-in proposer produced: two-injection combinations no baseline attempted, an overpayment layered on a change-order overrun, a double-counted line combined with a missing waiver, with scaffolds whose checks matched the injected types and rationales that read like a grader explaining the trap. On difficulty it converged on the middle of the range, 0.55–0.68, the band where a policy is learning rather than guessing.
The reward's D term measures difficulty against the _current policy's_ rollouts, and the current policy was a placeholder. Under that placeholder the frontier sat at difficulty 0.8, which a generic stand-in proposer happened to hit; Ornith's harder combinatorial tasks landed just short of it. "Harder than the baseline" and "on its own target frontier" are different claims, and the second one depends on who is estimating the frontier. Without that caveat, P0 reads as an Ornith win it hasn't earned.
## The head-to-head
The accepted curriculum was exported into a contamination-clean training set: [58 Ornith-proposed tasks](https://huggingface.co/datasets/caiotheodoro/plumb/viewer/train_ornith) (48 grown locally through the Q4 model, 10 from the bf16 run), with zero exact signature leaks and zero MinHash near-duplicates against the held-out seed-777 benchmark. Alongside it, the two control curricula: the [hand-seeded set](https://huggingface.co/datasets/caiotheodoro/plumb/viewer/train_handseeded) (223 tasks) and the [blend](https://huggingface.co/datasets/caiotheodoro/plumb/viewer/train_blended) (281).
The Ornith set is distribution-shifted against the benchmark it will be measured on:
- difficulty 0.66 on average vs 0.49 for the benchmark
- no easy tasks at all; 35 of 58 are two-injection combinations (60%), vs 24% in the benchmark
- no PASS tasks; the proposer never asked the model to say "this is fine"
- heavily concentrated in two retainage states
```json
// cv-chart
{"type":"bars",
"title":"The Ornith curriculum is shifted away from the benchmark it is scored on",
"series":[{"key":"ornith","label":"Ornith curriculum (58 tasks)","color":0},
{"key":"benchmark","label":"Held-out benchmark (1000 tasks)","color":1}],
"domain":[0,0.7],
"data":[{"label":"mean difficulty","ornith":0.66,"benchmark":0.493},
{"label":"two-injection share","ornith":0.603,"benchmark":0.24},
{"label":"PASS share","ornith":0,"benchmark":0.23}],
"caption":"Benchmark figures computed from the published seed-777 set. The proposer never emitted a PASS task, so the policy trained on it never saw an application that was simply correct."}
```
The hand-seeded set is the distribution-matched control by construction: mean difficulty 0.48 against the benchmark's 0.49. One asymmetry to keep in mind: its mix runs 32% PASS and 4% double-injection against the benchmark's 23% and 24%. So the head-to-head has real tension in it: the harder, more informative self-proposed curriculum against the boring matched one, and the blend of both.
The training: a [Qwen3-1.7B-4bit](https://huggingface.co/mlx-community/Qwen3-1.7B-4bit) policy, [LoRA](https://github.com/ml-explore/mlx-lm) on 8 epochs for each curriculum, ~3.3GB peak on the M5. The eval: the same [1000-task held-out benchmark](https://huggingface.co/datasets/caiotheodoro/plumb/viewer/benchmark), same oracle, same severity-weighted scoring.
| curriculum | parse | sw-recall | HIGH-recall | precision | exact |
| ---------------- | --------- | --------- | ----------- | --------- | --------- |
| base (untrained) | 0.127 | 0.020 | 0.034 | 0.118 | 0.000 |
| handseeded | 1.000 | 0.318 | 0.336 | 0.308 | 0.178 |
| ornith | 0.997 | 0.241 | 0.287 | 0.111 | 0.084 |
| **blended** | **1.000** | **0.334** | **0.365** | **0.374** | **0.228** |
95% bootstrap CIs (10,000 resamples, seed 11, over the same 1000 tasks): Ornith sw-recall 0.241 [0.214, 0.268] sits below hand-seeded 0.318 [0.290, 0.347] with no overlap. Blend 0.334 [0.306, 0.363] _overlaps_ hand-seeded on recall. Precision is the metric that separates the blend from the matched base: 0.374 [0.342, 0.406] vs 0.308 [0.279, 0.337]. Point estimates that "blend wins every row" are true; the interval that survives is precision (and Ornith-only losing).
```json
// cv-chart
{"type":"forest",
"title":"Full-size results with 95% bootstrap intervals",
"panels":[
{"metric":"severity-weighted recall","rows":[
{"label":"base","point":0.02,"color":"neutral","note":"untrained"},
{"label":"ornith","point":0.241,"ci":[0.214,0.268],"color":0},
{"label":"hand-seeded","point":0.318,"ci":[0.29,0.347],"color":1},
{"label":"blend","point":0.334,"ci":[0.306,0.363],"color":2}]},
{"metric":"precision","rows":[
{"label":"base","point":0.118,"color":"neutral","note":"untrained"},
{"label":"ornith","point":0.111,"ci":[0.098,0.124],"color":0},
{"label":"hand-seeded","point":0.308,"ci":[0.279,0.337],"color":1},
{"label":"blend","point":0.374,"ci":[0.342,0.406],"color":2}]}],
"caption":"10,000 resamples, seed 11, over the same 1000 tasks. On recall the blend and hand-seeded whiskers overlap; on precision they do not. Ornith-only sits clear of both on every panel."}
```
Before trusting that table I ran the confound check. The obvious objection is that the rows are not at equal N: 58 Ornith tasks against 223 hand-seeded and 281 blended, so "blended wins" might be "blended has more data." I ran the equal-N comparison, all three curricula cut to 58 tasks with an identical 49-train/9-valid split and an identical 8-epoch budget, and then a stratified variant, because a random 58-task draw is a noisy thing.
| equal-N (58 tasks, 49 train) | parse | sw-recall | HIGH-recall | precision | exact |
| ---------------------------- | ----- | --------- | ----------- | --------- | ----- |
| hand-seeded (random 58) | 0.998 | 0.106 | 0.036 | 0.126 | 0.023 |
| ornith (58) | 0.999 | 0.124 | 0.083 | 0.111 | 0.038 |
| blended (random 58) | 0.997 | 0.062 | 0.013 | 0.100 | 0.020 |
| hand-seeded (stratified 58) | 0.999 | 0.053 | 0.081 | 0.132 | 0.011 |
| **blended (stratified 58)** | 0.968 | **0.169** | 0.094 | 0.142 | 0.095 |
The honest reading of the small-N run is that 58 tasks cannot train this task well. Every row collapses to 0.05–0.17 severity-weighted recall against 0.24–0.33 at full size. These rows carry no confidence intervals, and each arm is a single curriculum draw, so most orderings inside that band are noise: a random 58-task blend scores below a random 58-task hand-seeded set, Ornith edges that same hand-seeded draw, and a stratified hand-seeded set, forced to keep the train prior's shares (20 PASS of 58) and all nine exception types so it sees fewer of each, scores below a random one. That is what an underpowered sample looks like. The one direction worth carrying forward is the stratified blend: a coverage-preserving mix of matched tasks and Ornith's combinatorial ones is the best small-N row (0.169 severity-weighted recall, 0.095 exact-match), pointing the same way as the full-size table. Read it as a hint, not a result. The confound check does not overturn the full-size result.
## Three findings I'm keeping
**Ornith-only underperforms.** The self-proposed, harder, distribution-shifted curriculum generalizes worse on the held-out benchmark than the distribution-matched hand-seeded set: worse severity-weighted recall, worse precision, worse exact-match. The equal-N check does not rescue it. At 58 tasks every curriculum is underpowered, and the best small-N row is the stratified blend, not Ornith alone. This is the second domain where I've watched this happen. ReconForge's B2 was the first: rebalance the training mix toward the classes you care about, and recall on the classes you cut collapses for almost nothing on the class you boosted. The training distribution has to match the deployment distribution. Ornith's loop, optimizing for frontier difficulty, manufactures a shifted distribution by design (0% easy, 60% two-injection, no PASS tasks), and that shift is what costs it on the benchmark it never saw.
**Blended wins every point estimate; the interval that survives is precision.** The blend, the distribution-matched base plus the Ornith frontier tasks as a supplement, beats both parents on every row of the point-estimate table. The bootstrap is stricter: blend vs hand-seeded overlaps on severity-weighted recall (0.334 [0.306, 0.363] vs 0.318 [0.290, 0.347]) and barely clears on precision (0.374 [0.342, 0.406] vs 0.308 [0.279, 0.337]; the lower bound clears the other's upper by 0.005, and it is one comparison picked out of many, so treat it as the lead claim rather than settled fact). That precision swing still has a mechanism. Ornith-only never saw a PASS task, so it learned to flag everything, and its two-injection combinations taught it a style of over-flagging; its precision collapsed (0.111 [0.098, 0.124], no overlap with either matched arm). The hand-seeded base taught the model when to say "this is fine." Adding the Ornith tasks on top of that restored the discrimination the harder tasks were supposed to teach, without destroying the restraint the matched base had already installed. A second mechanism plausibly works alongside: the blend restores double-injection coverage the hand-seeded set was thin on (16% against its 4% and the benchmark's 24%), so some of the recall gain may be plain distribution completion. Both readings agree on the practical upshot. The self-generated tasks work as a supplement, not a replacement.
**The D term is distribution-blind.** Ornith's reward for proposing is a product of validity, frontier difficulty, and novelty. There is no term for the deployment distribution. "Harder" is not the same as "worth training on," and not even the same as "on the frontier the deployment occupies." The loop pushed difficulty up and pushed the composition away from the benchmark, and both moves cost it. The fix is to give the loop a deployment anchor rather than abandon self-improvement: blend the proposed tasks with matched data, or retarget D at the deployment distribution so "frontier" means where the production errors live. That is the part I would want any self-improvement reward to steal.
## What broke when I tried to improve the proposer
The natural next move is to train the proposer itself, running the same GRPO loop the [Ornith blog](https://ornith.ai/ornith_1_5.html) describes on the proposer so it learns to propose better tasks. I built that machinery in the repo's `curriculum` package: the reward wrapper, group-relative advantages with the [DAPO](https://arxiv.org/abs/2503.14476) decoupled clip, a per-completion LoRA trainer on PyTorch and transformers, unit-tested end to end. Then the training stack got in the way.
Training the proposer crashes in the with-grad forward. Inference is clean. Generation, parsing, reward, everything works. The moment you ask for gradients, [`Qwen3_5GatedDeltaNet`](https://huggingface.co/docs/transformers/model_doc/qwen3_5) (the hybrid linear-attention block Ornith-1.5-9B shares with the whole current Qwen line) reshapes a zero-element tensor and dies. I checked the usual suspects: not a quantization problem (bf16 crashes too), not a cache problem (on and off), not a masking problem (explicit position ids and attention masks), and a real small qwen3_5 model trains fine. So the batch-0 crash I hit on the 9B looks like a harness- or scale-level artifact rather than a clean architecture bug, and I am not going to claim otherwise. The related bugs that are real (the [stale rope-deltas batch mismatch](https://github.com/unslothai/unsloth/issues/5678), the flash-attention-with-3D-position-ids crash, the packing leak, the bf16 A_log edge) are already filed and fixed upstream. A just-released model family can be a strong proposer and still not be a trainable loop in today's harness. You get the capability before you get the training stack.
## What didn't work
The full-size table left an obvious next question: can the loop be steered so the self-proposed set matches the train prior, and still beat naive Ornith at equal N? I ran that as a sequence of grows. Three of those runs are not the method. The one that is a method, the clean protocol, lost.
**More naive Ornith, no novelty floor.** I grew the accepted buffer from 58 to 184. Severity-weighted recall went 0.241 → 0.163 [0.139, 0.186]. Late accepts had novelty 0.01–0.07. The extra rows were near-duplicates. Of those 184, only 24 still clear a novelty floor of 0.3. Growing a self-proposed buffer without a floor added copies, not information.
```json
// cv-chart
{"type":"steps",
"title":"Growing the self-proposed buffer without a novelty floor",
"series":[{"key":"sw","label":"severity-weighted recall","color":0}],
"domain":[0,0.4],
"band":{"from":0.29,"to":0.363,"label":"full-size hand-seeded and blend, 95% CI"},
"data":[{"label":"58 tasks","sw":0.241},{"label":"184 tasks","sw":0.163}],
"caption":"Late accepts scored 0.01-0.07 on novelty. Only 24 of the 184 clear a floor of 0.3. More self-proposed data made the policy worse, and neither point comes near the full-size band."}
```
**A multiplicative match term on an accept-any-positive product.** I fitted a fourth term M against the held-out seed-777 mix and told the proposer to match it. That is eval leakage; those rows stay exploratory. Even on its own terms M did not steer: accept was `r > 0`, so a positive D×N let any mix through, and `count_match` scored the common class (single-injection) _lower_ for being far from uniform. Composition stayed 0% PASS / 66% two-injection, the naive profile drifted further (the original 58-task buffer sits at 60%). The leak is enough to discard the run. The control that filters is a hard mix constraint against the _train_ prior, not a fourth multiply.
**The proposer would not emit parseable JSON.** [Ollama](https://ollama.com/)'s 9B-q4 wrote `"seed": 84402e075f20` (unquoted hex) and oversized ints. `json.loads` failed, `parse_proposal` returned None, and a 180-try mix grow accepted one task. I repaired unquoted and coerced seeds in the parser. The grow that started before the repair is not a result.
**Mix deadlocked after one single.** Train prior is 32% PASS / 64% single / 4% double: the measured composition of the hand-seeded train set, which the mix constraint fits directly. After one accepted single, running mix is 100% single, so every further single was over-quota. The 9B kept proposing mid-difficulty singles. I stopped enforcing quotas until the buffer had six tasks (you cannot estimate a 3-class mix from n=1), and on a mix reject I sampled a proposal from an under-represented cell (deficit-fill). Feedback in the prompt was not enough; the controller had to emit PASS itself. Of 22 mix accepts, 16 were fills, not LLM proposals.
**The clean protocol, equal N=18.** Train prior (never seed-777), mix constraint, novelty floor 0.3, D from a 1.7B residual-error profile on a train sample of 80. Novelty floor 0.3 capped grow at 18–24 diverse tasks. I picked 0.3 off the 184-grow: late accepts there sat at 0.01–0.07, and 0.3 is the lowest round cutoff well clear of them. I did not sweep it, so treat it as a default, not a tuned constant. Equal N locked to 18 (15 train / 3 valid). Contamination audit vs seed-777: exact 0, near 0. 95% bootstrap CIs, 10k, seed 11:
| equal-N clean (18 tasks, 15 train) | parse | sw-recall | precision | exact |
| ---------------------------------- | ----- | ------------------------ | -------------------- | ----- |
| hand-seeded | 0.880 | 0.060 [0.044, 0.076] | 0.110 [0.082, 0.139] | 0.000 |
| naive Ornith | 0.980 | **0.135 [0.114, 0.156]** | 0.079 [0.067, 0.091] | 0.025 |
| mix-constrained | 0.843 | 0.075 [0.059, 0.093] | 0.082 [0.065, 0.100] | 0.011 |
| error-targeted | 0.990 | 0.031 [0.022, 0.042] | 0.116 [0.083, 0.152] | 0.002 |
| mock-anchored | 0.987 | 0.125 [0.104, 0.147] | 0.108 [0.089, 0.128] | 0.033 |
| blend | 0.980 | 0.055 [0.040, 0.071] | 0.101 [0.074, 0.129] | 0.001 |
Mix 0.075 [0.059, 0.093] sits below naive 0.135 [0.114, 0.156] with no overlap. Mock-anchored 0.125 [0.104, 0.147] overlaps naive. Error-targeted 0.031 [0.022, 0.042] sits below hand-seeded 0.060 [0.044, 0.076], and below blend's point estimate with intervals touching. Every arm here lands in the 0.03–0.14 band, and none of them touch the full-size bars (hand-seeded 0.318, blend 0.334). One unexplained signal in the same table: naive Ornith holds a 0.98 parse rate while the matched arms sag to 0.84–0.88, as if scaffold variety teaches output format faster than matched content does at tiny N. Worth a follow-up, not a conclusion. N=18 is what a 0.3 novelty floor yields on this generator. It is not a smaller version of the 58-task run; relax the floor and the buffer fills back up with near-duplicates.
The leaked M-term grows, the 184-without-floor run, and the N=18 adapters are not weights I am releasing. The method that survived contact with the numbers is still the full-size blend.
## Honest limits
The object-level policy is a 1.7B text model, not the 27B multimodal production model. It reads the pay application as structured text, not as a rendered scan, so OCR is out of scope. The Ornith curriculum is 58 tasks; 60% two-injection is 35 tasks. The full-size comparison is not at equal N. Equal-N at 58 is underpowered. Growing naive Ornith to 184 without a novelty floor hurt (0.241 → 0.163). The clean protocol at N=18 did not beat naive Ornith. the pow-* grows (the M-term fit and its anchored siblings) fitted their prior on seed-777; they are not a methods result, and those adapters are not on the Hub. [All data is synthetic](https://caio.theodoro.dev/blog/on-synthetic-data). Reported metrics carry 95% bootstrap CIs (10,000 resamples, seed 11). The 100% oracle pass rate is a low bar in a closed-world arithmetic domain; it may say more about the task than about the proposer.
## What this is for
A leaderboard tells you how strong a model is. It does not tell you whether the data a model proposes for itself is worth training on. This instrument separates those two questions, and on this niche the answer was that harder self-proposed tasks lose on their own and win only as a supplement to a distribution-matched base. The reward driving the loop has no term for the distribution it will be deployed on. The D term is the thing to fix.
Code is at [github.com/caiotheodoro/plumb](https://github.com/caiotheodoro/plumb). The instrument and the three full-size adapters are on the Hub: [caiotheodoro/plumb](https://huggingface.co/datasets/caiotheodoro/plumb), [plumb-handseeded](https://huggingface.co/caiotheodoro/plumb-handseeded), [plumb-ornith](https://huggingface.co/caiotheodoro/plumb-ornith), [plumb-blended](https://huggingface.co/caiotheodoro/plumb-blended). Collection: [caiotheodoro/plumb](https://huggingface.co/collections/caiotheodoro/plumb-ornith-wrote-the-curriculum-match-won-6a89f2ea18c97d11b0238529). The dataset's [`evals config`](https://huggingface.co/datasets/caiotheodoro/plumb/viewer/evals) holds every published row and its CI, so each table above is reproducible without rerunning anything. N=18 mix/error weights are not there.
---
More posts: https://caio.theodoro.dev/blog.md · About the author: https://caio.theodoro.dev/about.md · Machine-readable index: https://caio.theodoro.dev/llms.txt