Three gates lied to me before one told the truth

I trained an 8B vision-language adapter to diff an underwriting binder against the issued policy and list what changed. Two page images in, one JSON verdict out: PASS or FLAG, plus discrepancies typed against a fixed 13-class taxonomy. On the contracted held-out benchmark, seed 777, n=1000, zero signature overlap with the training set, it scores 0.959 severity-weighted recall at 0.956 precision and parses on 1000 rows out of 1000. GPT-5.6 Luna, zero-shot with vision, on the same images through the same scorer, gets 0.373. The base model without the adapter gets 0.098.

That number took seven checkpoints over six days. Most of what I learned came out of the first three signals I used to decide whether training was working. All three were wrong, each in a different way, and each cost a run to find out. What follows is the benchmark that produced those numbers, the three gates that misled me, three RLVR runs that lost to plain supervised fine-tuning on a task built for RLVR, the diagnostic that moved recall from 0.62 to 0.96, and two checkpoints I trained, measured, and refused to publish.

The problem

When a commercial insurance policy is bound, an underwriter agrees to a specific set of terms: named insured, policy period, which coverage parts apply, the limit and deductible on each, which ISO or AAIS forms are attached, who the additional insureds are, what the premium is. Then a carrier system issues a declarations page and a forms schedule. Somebody has to check that the second document says what the first one said. Carriers do this with people, and with rule engines inside Guidewire and Duck Creek. Public error-rate figures for issuance sit around 1 to 5%. The cost of a miss is premium leakage, an E&O claim, or a reissue, none of which show up on a leaderboard.

This task is a closed diff. Nothing in it requires a coverage opinion or a judgment call about whether two clauses are 40% or 60% similar. Either the issued policy matches the binder on a finite list of terms or it does not. That property is what makes the whole project possible, because it means the ground truth can be computed by a program instead of being argued about by a judge model.

The failure modes are not equal. Coverage that was bound and never issued is a HIGH severity miss, weight 1.0. A limit that came out lower than agreed is MEDIUM, weight 0.6. A form the carrier added that nobody asked for is LOW, weight 0.3. So the metric is severity-weighted recall: the weighted fraction of real discrepancies the model listed, where a HIGH class counts as caught only if the model got both the type and the severity right. Every number below is computed against those weights.

The benchmark

Verifier-as-oracle. The generator does not write a binder and a policy independently and then hope they diverge in a known way. It draws a binder, copies it into a policy, and then injects zero to two discrepancies drawn from the taxonomy. Consistency is the default state; divergence is deliberate and enumerated. A separate function, suture_forge.verify, then diffs the structured binder against the structured policy with no knowledge of what was injected, and the generator refuses to emit the task unless the verifier recovers exactly the injected set. It retries up to twelve times and raises if it cannot. Oracle agreement with gold is therefore a structural property of the data, not a number I measured on a sample and reported.

The substrate had to be checked too. The first scaffold cited four ISO form numbers that do not exist. GL 04 02 and GL 21 05 are not real, because ISO general liability forms use the CG prefix. Wrong form numbers would have silently poisoned every FORM_MISSING and FORM_ADDED label in the corpus, and the model would have learned them perfectly. They were replaced with public, verifiable numbers from the ISO CP, CG, CA, CM, CR, CU and EB programs before anything was trained.

Zero contamination. Every task gets a SHA-256 signature over its sorted ground-truth fields. Train and validation are seed 7; the contracted benchmark is seed 777 and was generated with a --train flag that exits non-zero if the signature overlap with training is anything other than zero. The contamination module that does this is seventeen lines. The CLI around it is what matters: leakprobe runs the clean probe and deliberately leaks ten training rows into a copy of the eval set, then fails unless the clean probe reads 0 and the poisoned probe fires on all ten. A contamination checker that has never been shown a real leak is a checker you have no evidence works. Every training mix in the chain ran it twice, against validation and against 777, and every run is in the decision log.

Determinism. The dataset stores structured Task objects, not images. Pages are re-rendered from those fields at training and evaluation time. This matters because macOS and Linux rasterize the same page to different bytes, so shipping PNGs would mean shipping a platform. The labels do not depend on the raster. Difficulty is a real knob rather than a label: at difficulty d, the generator renames round(d*3) coverage labels to decoys seeded off a hash of the task id, rotates the page by up to (d-0.25)*10 degrees, jitters a fraction of pixels, and lays a glare gradient across one corner. Near-miss values scale the same way, so a limit discrepancy at difficulty 0.6 or above is a 5% delta instead of a 20% one.

The obvious objection is that I built the benchmark I am scored on. That is true, and it is why the generator, the oracle, the leak probe, the scorer, the gold set and every prediction file are published. The measurement is the product. If the benchmark is wrong, it is wrong in public and in a way anyone can rerun without a GPU.

The gates that lied

Token accuracy plateaued at 0.51 and meant nothing. The first supervised run stopped with mean token accuracy at 0.515 and training loss at 3.75. I read that as a schema that had not locked, and went looking for the cause. First hypothesis: the target JSON was being truncated, because max_length was 1024 and two 384px thumbnails eat most of that. So I re-ran at 2048. The token counts came back identical, 3.006e5 at epoch two, exactly matching the 1024 run, which ruled truncation out completely rather than probably. Second hypothesis: the vision-language collator was dropping images. So I switched to typed two-image messages with remove_unused_columns=False. Token count rose about 1%, the loss curve copied the previous run almost exactly, and final accuracy came back 0.510.

Three runs to reach the actual conclusion, which is in the decision log in one sentence: mean token accuracy is computed over prompt and vision tokens as well as the target, so on a task where the prompt is two images and a long system message it is dominated by tokens the model was never being asked to predict. It was never going to move, and I spent three runs asking it to.

Parse rate hit 0.94 and licensed exactly one thing. The replacement gate was better. It asked whether the model emits parseable, schema-valid JSON. On the 80-row validation set, 75 rows parsed, 0.9375. That is a real property and it went up honestly. It also told me nothing about whether the model could do the job, because severity-weighted recall on those same rows was 0.216. Parse rate was sufficient to justify starting reinforcement learning against an oracle, and insufficient to call anything a win. The decision log says so explicitly, and I still nearly talked myself into treating it as progress, because it was the first number in the project that had gone up for a legible reason.

Validation recall lied in both directions, and once by omission. The seed-7 holdout is 80 rows drawn from the same generator as training. Early on it read high: sft-vl scored 0.216 on validation and 0.163 on seed 777. Late on it read high again, and worse: the published sft-dedhi checkpoint scores a perfect 1.000 recall, 1.000 HIGH recall on validation, against 0.959 and 0.969 on the benchmark. A saturated validation set cannot rank two candidates.

The omission is the part worth keeping. Validation contains zero DEDUCTIBLE_HIGHER gold examples. None. The class that turned out to be the single largest hole in the model, the one that took three separate training loops to close, is a class the validation set is structurally incapable of scoring. Had I gated on validation, I would have shipped a model with 0.507 recall on that class and a validation scorecard reading 1.000, and I would have believed it.

The only gate that never lied was a held-out seed, contracted before the first run, with the metric and the publish bar written down in advance. CONTRACTS.md was committed on day one and specifies the taxonomy, the severity weights, the scoring rule, and the bar: recall above 0.95, precision above 0.95, HIGH recall 1.00. Not one of those was renegotiated when a run came in under it.

What RLVR did on a task built for RLVR

Reinforcement learning from verifier rewards should have owned this problem. The output is machine-checkable, the reward is exact rather than modelled, there is no process reward model to mis-specify, and the oracle is the same program that generated the labels. Every precondition in the recipe was satisfied. It lost three times.

Run one collapsed to constant PASS on the first completion. Fifty GRPO steps at group size 2 from the supervised checkpoint. Parse rate rose from 0.9375 to 0.9625. Recall fell from 0.216 to 0.135, HIGH recall from 0.241 to 0.103. The first logged completion was {"verdict":"PASS","discrepancies":[]} and it never really left. The mechanism was in the reward scalar and it is embarrassing in retrospect: a correct clean PASS scored +1.0, a FLAG task where the model listed nothing and got the verdict right scored +0.2, and a FLAG task with wrong types scored -0.2. Emitting PASS with an empty list on every input was the global optimum of the reward I had written. Training diagnostics looked healthy throughout: frac_reward_zero_std averaged 0.445 and hit 1.0 on only 4 of 50 steps, clipped_ratio was zero at every step, mean completion length 43.

Run two fixed the reward and collapsed anyway, shorter. Zero-recall FLAG now scored -1.0 minus the false-positive penalty, which makes silence strictly worse than a wrong guess. Mean training reward duly fell to -0.628, which is what you would expect if the policy were being punished out of the attractor. It shortened instead. Mean completion length went 43, then 22, and by the last step every sampled completion was 13 tokens long. frac_reward_zero_std hit 1.0 on 23 of 50 steps, meaning that on nearly half the steps every sample in the group scored identically and there was no gradient signal at all. Validation parse rate reached 0.9875 while recall fell from 0.216 to 0.069. At group size 2, the short parseable mode wins regardless of what you pay for it, because with two samples the policy rarely sees a long completion and a short one in the same group and almost never sees two different long ones.

Run three ran clean and still lost. Group size 8 from the distilled checkpoint, TRL 1.10 with DAPO decoupled clipping and dynamic sampling, the Dr. GRPO fix on length normalization. No out-of-memory on the L4. frac_reward_zero_std averaged 0.26 and never touched 1.0, so the group-size-2 collapse genuinely did not recur. Completions stayed long, mean length 61 at the last step. Fifty steps in 3007 seconds, training loss 0.002. Validation recall fell from 0.799 to 0.770. HIGH recall was flat at 0.862. Precision rose from 0.658 to 0.687.

That third run is the honest one, because there is no bug to blame. The optimization worked. The reward was correct. The recipe I had written down said this was the phase where the number goes up, and it went down. GRPO appears in the validation table three times and in the published adapter zero times. On this stack, at this scale, the RLVR half of my own methodology did not pay off, and I am not going to claim a hyperparameter would have rescued it, because I did not run the experiment that would show that. One caveat the section owes you: every GRPO verdict here rests on validation. The decision log forbade running seed 777 on a GRPO adapter, so all three runs were judged on the 80-row holdout, the same gate this post spends a section calling unreliable. For the two collapses that hardly matters, because the completions themselves are the evidence and they are 13 tokens long. For the third, group size 8, validation is all I have.

The lever that worked

After ReST-EM the model sat at 0.622 on the benchmark. It had learned to list: FORM_MISSING was at 0.61, AI_MISSING at 0.46, up from near zero. It had learned nothing about reading numbers. DEDUCTIBLE_HIGHER was 0.07, and had gone down from 0.10 at the distillation checkpoint, which is to say the self-training round that was supposed to help actively hurt the class that most needed it. ReST-EM was capped by construction. Its trace selector keeps the best of the sampled completions and gold, and gold is always a perfect type label, so sampling could improve formatting and could never lift type recall past what gold had already taught.

The fix started with a diagnostic, not a training run. class-report broke recall and precision out per class, which turned "recall is 0.62" into "six classes are fine and the numeric ones are broken." Then numeric-slice broke the numeric classes out by difficulty decile and, for each false positive, by the gold label of the row that produced it. 40 of the 51 DEDUCTIBLE_HIGHER false positives were rows whose true label was DEDUCTIBLE_LOWER. The model was seeing the deductible change. It was getting the direction backwards.

That distinction decided everything downstream. The generator always perturbs deductibles by exactly a factor of two, up for HIGHER and down for LOWER, with no near-miss tail at any difficulty. A model that could not read the digits would have produced false positives spread across every gold class, and would have argued for native-640 rendering and a bigger image budget. A model confusing two labels that are numerically far apart is making a sign error, and more pixels will not fix a sign error. It also ruled out the obvious remedy. Oversampling DEDUCTIBLE_HIGHER on its own would have taught the model that deductible changes are usually HIGHER, which is exactly the shortcut that produced the swap.

So the mixes were contrastive and cumulative. Seed 13 boosted both limit classes and DEDUCTIBLE_HIGHER at 0.5 frequency, 1520 rows. Seed 17 boosted LIMIT_HIGHER at 0.85 and replayed the previous mix, 2720 rows. Seed 19 boosted DEDUCTIBLE_LOWER and DEDUCTIBLE_HIGHER equally at 0.85, so the two directions appear in the same batch against the same layout, 3920 rows. Seed 23 boosted DEDUCTIBLE_HIGHER alone, once the swap was already gone, 5120 rows. Each mix replays every mix before it, so nothing is forgotten. Each ran the leak probe twice before any GPU time was spent.

The seed-19 contrastive mix did exactly what the diagnosis predicted and nothing more. It killed the swap: DEDUCTIBLE_HIGHER false positives whose gold label was DEDUCTIBLE_LOWER went from 40 to 6, and precision on the class went from 0.407 to 0.700. It taught DEDUCTIBLE_LOWER outright, 0.494 to 0.897. And it did not catch a single one of the 34 true DEDUCTIBLE_HIGHER misses, which stayed at exactly 34. Those were a separate failure from the swap and needed a separate mix. Seed 23 took them from 34 to 3.

The lever was reading the errors, working out which of the two failures I actually had, and oversampling against the one I found. Four shell scripts and four seeds took recall from 0.622 to 0.959.

Two checkpoints I refused to publish

sft-numeric scored 0.788 on the benchmark against a published model at 0.622. It was better on almost every class. The written gate for that loop was LIMIT_HIGHER recall above 0.35, because that was the class the loop existed to fix. It came in at 0.309, on a gold count of 81, which is a shortfall of roughly three examples. Not published.

sft-ded scored 0.886 recall at 0.941 precision against a published model at 0.839 and 0.870. It was the best checkpoint in the project at that moment. The written gate for that loop was DEDUCTIBLE_HIGHER recall at 0.70. It came in at 0.507, unchanged, the same 34 false negatives as the checkpoint before it. Not published.

Both would have moved the headline number. Publishing either would have meant overwriting the default eval adapter and the Hub weights with a model that failed the specific bar the loop was named after, which converts a written gate into a preference. The reason to write gates down before the run is that afterwards you will always have a good argument, and the argument will always be the aggregate number, and the aggregate number is precisely what a class-specific hole hides inside.

Honest limits

HIGH recall is 0.969, not the contracted 1.00. Recall and precision clear 0.95; that one does not, and it is the gate I care most about, because a HIGH class is coverage that was bound and never issued.

The pages are rendered by my generator, not scanned from a carrier system. Transfer to real PDFs is unmeasured, and I have no basis for guessing at it. This is the largest open question in the project and it is not one more training loop away.

The remaining HIGH misses on the benchmark are AI_MISSING at 0.912 and FORM_MISSING at 0.944. I have not started a mix for them, because the standing rule is that the error slice gets written before the training job, and I have not written it.

Citation exact match reads near 1.0 on every supervised checkpoint and 0.000 on Luna. That is not a quality signal. Gold citation strings are frequently the literal word "Declarations" or the empty string, and the model has learned to clone them. It is disclaimed in five separate documents in the repo and it should not be quoted as anything.

The adapter is 4-bit QLoRA at rank 32 on an 8B backbone with 384px page thumbnails. Native 640px was started once and cancelled at 290 of 760 steps on a credit call, and never revisited, because the diagnosis said the problem was a sign and not a pixel. A 27B run was planned at the start and never launched. Neither was needed to clear the bar, and neither has been ruled in or out on evidence.

Everything is one benchmark seed. It is a large one, 1000 tasks with 739 carrying discrepancies, but it is one.

What this is for

The recipe transfers, and it is the reason there is anything to publish. Pick a task where the answer is a closed diff over a finite ontology. Write a generator that constructs the consistent case first and injects known violations second. Make a program the oracle and gate the generator on it, so agreement is structural. Sign every task, and prove your leak detector by leaking on purpose. Contract the benchmark seed and the publish bar before the first run, and honor them when a good checkpoint fails one. The same pattern, at 1.7B on a laptop, produced a reconciliation model that beat a frontier model on severity-weighted recall.

The transferable lesson from this repo is narrower and it is about instrumentation. Three of the four things I measured were measuring something other than the task: an accuracy averaged over tokens nobody was predicting, a parse rate that says a model can format, a validation split with a structural blind spot in the exact place the model was weakest. What finally moved the number was a per-class report and an error slice that could tell a sign error from a reading error. Build those before you build the training loop, because until they exist you cannot tell a run that worked from a run that looked like it worked.

Next is issue #12: slice the AI_MISSING and FORM_MISSING failures on the published predictions and find out which failure they are before mixing anything.

Model: caiotheodoro/suture-8b

Dataset: caiotheodoro/suture

Predictions: caiotheodoro/suture-evals

Code: github.com/caiotheodoro/suture