# I Pinned a Template Into DiffusionGemma to Read Invoices. One Layout Said Yes. Four Said No.

> I pinned a fixed answer template into DiffusionGemma and pre-registered that it would make invoice extraction more consistent. On the first layout it did, +0.130. Across four municipal layouts it did not, −0.018, and it cost 2.5 to 9 times the latency. The autoregressive sibling beat it by 11 points, and a cheap hosted model beat both.

- Published: 2026-09-25
- Reading time: 12 min read
- Tags: ML, Engineering
- Author: Caio Theodoro (https://caio.theodoro.dev/about.md)
- Canonical HTML: https://caio.theodoro.dev/blog/diffusiongemma-pinned-one-layout-said-yes
- Source repository: https://github.com/caiotheodoro/nfse-canvas
- Hugging Face: https://huggingface.co/collections/caiotheodoro/nfse-canvas-one-layout-said-yes-four-said-no-6ab750b90e32dd23e0fa5cd0

---

I pre-registered a bet: pin a template into a diffusion model and it reads invoices more consistently. One layout agreed with me. Four disagreed.


The model is [DiffusionGemma](https://huggingface.co/google/diffusiongemma-26B-A4B-it), a 26B mixture-of-experts. It writes by denoising 256-token blocks, not one token at a time, which makes extraction look easy: put the field names into the canvas, freeze them, and let the model fill in only the values. The structure should come free. The test documents are 1,006 NFS-e, the municipal service invoices every Brazilian provider issues, pulled from an accounting practice's clients.


Most people assume a pinned template is free structure when you extract with a diffusion model. I scored it on rule consistency across layouts the model was never tuned on, plus latency. By that count pinning bought nothing and cost 2.5 to 9 times the time.

- **Measured**: 604 unique invoice units across four municipal layouts, pre-registered with 13 dated amendments
- **H1, pinning helps**: −0.018, 95% CI [−0.053, +0.014], n=428. Bar was +0.05 with the CI above zero. Not supported
- **The pilot**: +0.130, CI [+0.093, +0.167], n=161, on one layout. The same test, pointing the other way
- **H2, diffusion vs autoregressive**: Gemma 4 26B-A4B under the same template beats pinned DiffusionGemma by 11.1 points of field exact match, CI [−0.139, −0.088], n=393
- **H3, uncertainty flags errors**: AUROC 0.661 over 15,462 fields against a bar of 0.80. Not supported
- **Cost of pinning**: 8.0 s per note pinned against 0.9 unpinned on an RTX Pro 6000; 9.6 against 3.8 on an A100
- **Published**: 10,000 synthetic invoices in four layout families. The real invoices stay private

## Invoices that check themselves


These notes belong to clients. Most are clinic invoices naming a patient, a CPF and a procedure, which puts them under LGPD article 11, the health-data clause. Nothing leaves the laptop until it has been pseudonymized, and the real set will never be published.


Two properties of these files settled the design. First, 790 have a text layer and 216 are image-only prints or scans, and the text layer is useless as it stands: labels and values come out of the PDF in scrambled reading order, `Alíquota` on one line, `2,0000%` three lines later, the patient's name floating free of its label.


Second, and this is the one the whole study stands on, every note checks itself. Federal withholdings are the base times a printed rate. ISS is the base times the municipal rate. The net amount is the gross minus the withholdings. CPF and CNPJ carry check digits. On the first file I opened, 230.00 at those rates gives 1.50, 6.90, 3.45, 2.30 and 4.60, and 230.00 minus the withholdings is the printed 215.85.


Roughly ten deterministic constraints per note. A program computes them, and no judge model has to argue.


## The benchmark


**Oracle-accepted gold.** I wrote a geometric parser per layout, binding values to labels by position instead of reading order, and accepted a note into gold only when the parser output passed every core rule for its layout. 847 of the 1,006 PDFs made it. Twelve were incomplete sub-layouts, 18 broke a rule and went to a review bucket, 124 sat in layouts I wrote no parser for, and 5 were byte-identical duplicates. The oracle was checked before it checked anything: on 300 synthetic notes with one or two injected violations, it recovered exactly the injected set on all 300.


The oracle also found something I did not put in. São Paulo truncates ISS instead of rounding it. On the 24 São Paulo notes where half-up and truncation give different cents, 22 match truncation and none match half-up. That became a per-layout rounding mode, logged as an amendment before any model ran.


**Units, not PDFs.** One prestador — the service provider — issued 598 of the 1,006 notes, and in gold two prestadores in one city account for 595 of 847. Many of those are recurring monthly invoices with identical verified fields. Counting every PDF would inflate n and shrink every interval. The evaluation unit is a unique gold content per layout, which collapses 847 PDFs to 604 units: 491 in Balneário Camboriú, 47 in São Paulo, 20 in Barueri, 46 in Osasco. The headline estimate is an equal-weight mean over the four layouts, and the bootstrap resamples prestadores inside each layout before it resamples units. Three of the four layouts have one or two prestadores, so their intervals are within-prestador only.


**Pre-registered.** The protocol was committed before any code under `src/` existed. H1: pinning raises oracle-rule satisfaction by at least five points. H2: pinned diffusion against the autoregressive sibling, two-sided, no direction claimed. H3: per-slot entropy predicts field errors well enough that escalating the least confident 10% of fields halves the notes with any violation. H4, a service-code classifier, was downgraded to exploratory once the profile showed 598 of 629 labelled notes in one class.


## The gates that lied


The privacy gate passed while the images still leaked. The first leak test re-OCR'd every exported image and searched it for the strings the pipeline had flagged: names, CPFs, CNPJs, codes. It reported 0 leaks on 847 images. A reviewer in a fresh context, given only the rules, then looked at 24 random exports and found identifying data on 19 of them: company names in free-text boxes, 0800 numbers, an untouched QR code on every Osasco note, and verification codes that let anyone pull the original invoice from the city hall site. The test searched for what the detector flagged. It had no way to see what the detector missed.


I was proud of that 0/847. The redesign threw away span-level replacement and wiped whole blocks per layout, anchored on section headers so they follow the page when the layout drifts, and it fails closed when an anchor is missing. Round two found 16 of 32 still dirty. Round three, on a new random seed, found 0 of 32. The gate that shipped is four checks: the flagged-string search, a positional check that no original token survives inside a wiped band, a scan confirming that all 876 valid CPF/CNPJ numbers in the exports are synthetic, and the fresh-context audit.


The one-layout pilot lied by being the only data I had. My GPU runs processed units in layout order, and Modal's spend limit stopped them one layout in. The diffusion and autoregressive arms had covered Balneário Camboriú and nothing else, and the report printed "supported" next to +0.130. That label was mechanical. The hypothesis was about a mean over four layouts, and I had one layout with two prestadores. A continuation on a second provider covered all 113 units in the other three, and in every one of them pinning lost on rule consistency:


| layout             | units     | prestadores | pinned EM | free EM | pinned rules | free rules |
| ------------------ | --------- | ----------- | --------- | ------- | ------------ | ---------- |
| Balneário Camboriú | 320 / 315 | 2           | 0.856     | 0.769   | 0.766        | 0.641      |
| São Paulo          | 47        | 16          | 0.930     | 0.974   | 0.876        | 0.954      |
| Barueri            | 20        | 1           | 0.589     | 0.350   | 0.787        | 0.863      |
| Osasco             | 46        | 1           | 0.926     | 0.938   | 0.924        | 0.967      |


```json
// cv-chart
{"type":"bars","title":"Oracle-rule satisfaction per layout, pinned against free DiffusionGemma","series":[{"key":"pinned","label":"pinned template","color":0},{"key":"free","label":"no template","color":1}],"domain":[0,1],"data":[{"label":"Balneário Camboriú","pinned":0.766,"free":0.641},{"label":"São Paulo","pinned":0.876,"free":0.954},{"label":"Barueri","pinned":0.787,"free":0.863},{"label":"Osasco","pinned":0.924,"free":0.967}],"caption":"Share of each layout's core oracle rules that the extracted fields satisfy, averaged over seeds 0 and 1 and over units. The pilot saw only the first pair of bars. Balneário Camboriú has 2 prestadores, São Paulo 16, Barueri and Osasco 1 each. The headline H1 estimate is the equal-weight mean of the four differences, −0.018."}
```


Balneário Camboriú is the one layout where pinning helps on both measures. Barueri shows the split cleanly: pinning more than doubles field exact match there and still breaks more rules. Balneário Camboriú is also where the pilot came from.


A perfect field score sat next to a failing rule score. The first full report had qwen3.8-flash at 1.000 field exact match in São Paulo and 0.667 rule satisfaction. A model cannot get every field right and break a rule — unless the rule needs a field the page never prints. It did. The base-of-calculation rule subtracts the unconditional discount, which São Paulo and Osasco do not print; gold carried the parser's zero, the model correctly wrote `null`, and my own "unscorable counts as failed" amendment turned a correct answer into a failure. The fix fills unprinted inputs with the same zero gold uses. It came after the analysis freeze, so it is disclosed in the red-team document with its reason; the Balneário Camboriú numbers do not change.


## Why pinning costs


DiffusionGemma's `generate()` returns the argmax canvas, not the accepted canvas, so seeding the first canvas with the template does not keep it there. The pinning here is a logits processor that makes every template position a one-hot at every denoising step. Sampling, argmax and the entropy-bound acceptance rule then all agree on the template token, and the pinned positions add zero entropy to the acceptance budget. On a random-weight copy of the model, run through the real generation loop across two canvases, every pinned token survives.


The cost is the slots. Each fixed-width slot is wider than most values. After `230,00` the model still has to decide what fills the rest of the slot, and it invents filler — `\u00a` fragments, stray newlines. Those positions never become confident, so the adaptive stopping rule keeps denoising. Across roughly 830 generations per arm on the real notes, the median tokens per forward pass was 20.0 unpinned and 7.2 pinned.


**Pinning buys repeatability, not correctness.** Seed-to-seed agreement was 0.899 pinned against 0.870 free, on a model whose rule consistency did not improve.


## What didn't work


H3 failed on both criteria. Final-step slot entropy separated wrong fields from right ones at AUROC 0.661 over 15,462 field instances, against a bar of 0.80. Sending the least confident 10% of fields to a human cut the units with any core violation by 13.7%, against a bar of 50%. The confidence signal a diffusion model gives away for free was not worth routing on here.


The autoregressive sibling took H2 without a fight. Under the same template, decoding greedily, Gemma 4 26B-A4B scored 1.000 field exact match in São Paulo, 0.975 in Osasco and 0.898 in Balneário Camboriú.


The oracle worked. It labelled 847 notes with nobody typing a field. It caught a municipal rounding convention. It also exposed a scoring bug, by producing a number that could not be true. Paired with a cheap hosted model it is most of a product. qwen3.8-flash reads three of the four layouts at 0.993 field exact match or better, using a median of 5,438 input and 860 output tokens per note, about $0.0012 at Zen list price, and the oracle can check every one of its outputs without a label. I have not measured how many of its errors the oracle catches.


## Honest limits


**One practice's clients.** Barueri and Osasco have one prestador each, Balneário Camboriú has two, and those two are 70% of gold. Cross-prestador generalization rests on São Paulo alone, with 16 prestadores in the evaluated units.


**The labels are mine.** Gold is my parser, accepted by my oracle. A parse error that stays consistent across every dependent field survives into gold. That is hard to do in Balneário Camboriú, which has ten core rules. In São Paulo and Osasco there are three, so gold there is weaker.


**Pseudonymized images.** Redrawn CPF and CNPJ text is cleaner than the scan around it, which likely flatters document fields on the image-only layouts. The results describe pseudonymized invoices.


**The design moved.** Thirteen amendments, each dated with its reason. The first budget estimate came in over the $25 gate, so 847 PDFs and three seeds became 604 units and two seeds. Balneário Camboriú is a hash-ordered subset, not the full 491. Some of its pinned seeds ran on an A100 and some on an RTX Pro 6000 while the unpinned arm ran on the A100 only; I did not measure what that does. I looked at partial results once, mid-run, after the analysis code was committed, and nothing in scoring changed after that commit except the disclosed bug fix.


**One slot design.** Two seeds, one slot width, one synthetic exemplar in the prompt. Padding-aware widths or a model trained to fill slots might rescue pinning. I did not run either.


## What this is for


The real invoices stay private. What anyone can run is NFS-e Synth: 10,000 synthetic invoices in four layout families modelled on the structure of these four municipal forms, with no logos and every page watermarked `SINTÉTICA — SEM VALOR FISCAL`. The São Paulo family truncates ISS. 1,998 notes carry injected violations, and the published gold reproduces the published violation labels on all 10,000 rows. Same arguments, same bytes.


The null points at the next run. Train a small vision model on the synthetic set only, then score it on the 604 private units with the same oracle and the same layouts. That measures transfer from synthetic invoices to real ones without publishing a single real one.

- Dataset: [huggingface.co/datasets/caiotheodoro/nfse-synth](https://huggingface.co/datasets/caiotheodoro/nfse-synth)
- Collection: [huggingface.co/collections/caiotheodoro/nfse-canvas-one-layout-said-yes-four-said-no](https://huggingface.co/collections/caiotheodoro/nfse-canvas-one-layout-said-yes-four-said-no-6ab750b90e32dd23e0fa5cd0)
- Code: [github.com/caiotheodoro/nfse-canvas](https://github.com/caiotheodoro/nfse-canvas)

---

More posts: https://caio.theodoro.dev/blog.md · About the author: https://caio.theodoro.dev/about.md · Machine-readable index: https://caio.theodoro.dev/llms.txt