# Can A 1.7B Model Beat a Frontier on Reconciliation Exceptions?

> A Qwen3-1.7B model fine-tuned on a laptop catches more high-severity reconciliation exceptions than DeepSeek v4-flash, including every one in the test set. Covers the benchmark, the training run, and the approaches that failed along the way.

- Published: 2026-06-08
- Reading time: 13 min read
- Tags: ML, Engineering
- Author: Caio Theodoro (https://caio.theodoro.dev/about.md)
- Canonical HTML: https://caio.theodoro.dev/blog/reconforge-1-7b-beats-deepseek-on-the-money-metric
- Source repository: https://github.com/caiotheodoro/reconforge
- Hugging Face: https://huggingface.co/collections/caiotheodoro/reconforge-6a89e9d6539e5b51403dd9ca

---

I built a benchmark-grade evaluation system for financial reconciliation agents and fine-tuned a small open model to run it. The model, [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B) with a LoRA adapter trained in under two hours on an Apple M5, beats DeepSeek v4-flash on the metric that matters for the job: severity-weighted recall. 0.913 vs 0.872. It catches 100% of high-severity exceptions on the 800-task held-out benchmark. DeepSeek gets more tasks exactly right overall. It loses the ones that cost money.


This post is the study case: how the benchmark is built so its numbers can be trusted, how the model was trained, three negative or open results I'm keeping in the writeup instead of hiding, and the parts that are still honestly unresolved.


## The problem


Back-office reconciliation is the plumbing of finance. Every trade, transfer, and FX deal produces two records: your ledger entry and the counterparty's statement. An ops team matches them, straight-through when they agree, and investigates when they don't. A reconciliation exception can be an amount off by a fraction, a wrong FX rate applied at booking, a beneficiary name that doesn't match, a value date that lands on a non-business day, or a message that never arrived.


The failure modes have different financial severity. A missed amount mismatch is principal at risk. A duplicate is a rebook at worst. So the scoring function has to be severity-weighted: catching a $1M misbooked transfer must outweigh catching forty duplicates. That's the premise. Everything below is built on it.


## The benchmark


The evaluation set is 800 tasks generated by a seeded, difficulty-parameterized generator: realistic SWIFT and ISO 20022 message pairs with injected exceptions, adversarial near-misses (amounts that differ only within rounding tolerance so they should match, wrong-but-plausible FX rates), and a fixed severity taxonomy of nine exception types.


A benchmark is only as good as its ground truth, so three properties were verified before any model was scored, and each one is enforced by code, not by hoping the generator got it right.


**Verifier-as-oracle.** An independent program recomputes the verdict from the ledger and statement fields alone. It never reads the label the generator attached. It classifies a pair by walking a fixed priority order: missing message beats field corruption beats amount or FX mismatch beats duplicate (trimmed reference equality) beats beneficiary or counterparty mismatch beats value-date mismatch beats match. That order exists because a single pair can deviate on more than one field, and without a fixed priority, "verifier agrees with ground truth" would be an accident of which field you check first, not a guaranteed property. The generator draws exactly one exception per task and shares the same priority order, so the two sides agree by construction. Measured: 100% agreement on both a 300-task and a 400-task pilot, byte-identical across reruns.


That agreement isn't free. It's enforced by a self-check loop. Every candidate task runs through the verifier before it's accepted; if the verifier disagrees with what the generator intended to inject, the pair is thrown out and regenerated, up to a bounded number of tries, deterministically (same seed still produces the same sequence of draws). Early on this loop was expensive: drawing the beneficiary, counterparty, and value date independently for each side of a pair made about 90% of "clean" pairs look like accidental mismatches, forcing 4,257 verifier evaluations to produce 500 valid tasks. The fix was structural, not statistical: draw one beneficiary, one counterparty, one value date, and one amount family per task and copy them to both sides, so only the deliberately injected exception diverges the two sides. After that fix, the same 500 tasks needed exactly 500 verifier evaluations — zero rebuilds.


**Zero contamination.** The benchmark comes from seed 777, training data from seed 101. Every task gets a signature: SHA-256 of its sorted (field, value) pairs across both the ledger and statement sides, metadata excluded. The two sets share zero signatures. That's a stricter bar than checking whether the same _kind_ of pair shows up in both sets. It's checking whether the literal field values do, which is the actual definition of a model having seen a specific example during training. The monitor built on this signature scheme was stress-tested with a synthetic leak: it fires 1.0 (perfect detection) at every leak fraction from 5% to 50%, and 0.0 false-fires on a genuinely clean set.


**Determinism.** Same seed, byte-identical output, every time. The honest source of variance in these results is the sampling marginal (self-consistency at inference time), not any nondeterminism in how the benchmark itself is built.


Scoring: HIGH-severity exceptions (amount mismatch, FX conversion error, beneficiary mismatch, counterparty mismatch, weight 1.0 or 0.9) count as caught if the model flags the pair as anything other than MATCH. MEDIUM and LOW exceptions (value-date mismatch, missing message, partial match, duplicate, field corruption, weight 0.6 down to 0.2) count as caught only if the model names the exact exception type. Severity-weighted recall is the weighted sum of caught exceptions divided by the weighted sum of all exceptions. That asymmetry is deliberate: for a HIGH-severity case, "something is wrong here" is worth almost as much as the exact diagnosis, because it triggers human review either way. For a LOW-severity case, a vague flag isn't useful; you need to know it's a duplicate specifically to route it correctly.


## The model


Base: `Qwen3-1.7B`, 4-bit MLX quantization, Apache-2.0 licensed. LoRA rank 16, alpha 32, dropout 0.05, batch size 2 with gradient checkpointing. 3,198 synthetic training pairs, 802 held out for validation, both drawn from seed 101 with zero signature overlap against the seed-777 benchmark.


The split isn't a random shuffle. Tasks are binned by (difficulty decile, exception type) and each bin is allocated proportionally between train and validation under a seeded RNG, so every exception class — including rare ones like partial-match — shows up in both splits. A plain random split or a difficulty-only split can lose an entire rare class from validation by chance, which would make its recall number meaningless. The stratified split guarantees every class has a measurable recall on both sides.


Training ran 740 iterations, about 100 minutes on the M5, peak memory 3.35GB. That step count isn't a fixed budget. It's a stopping rule. Training loss plateaued at 0.088 around iteration 330 and stayed flat; the plan called for 1,500 steps, but running the remaining 760 would have cost another 2.5 hours for no measurable gain once the loss curve stopped moving. An earlier, separate run stopped at 700 iterations scored substantially lower (severity-weighted recall 0.729 against 0.913), so the 740-iteration number is not a lucky single checkpoint — a materially different run on the same setup would have shown it. Inference runs in non-thinking mode — the chat template renders an empty `<think></think>` block and the model answers directly — averaging 38 tokens per verdict. The whole pipeline, dataset to trained adapter, runs on a laptop.


Results on the 800-task benchmark:


| Model                        | Accuracy | Severity-w. recall | HIGH recall | Parse rate |
| ---------------------------- | -------- | ------------------ | ----------- | ---------- |
| ReconForge Recon (1.7B LoRA) | 0.805    | **0.913**          | **1.000**   | 1.000      |
| DeepSeek v4-flash            | 0.876    | 0.872              | —           | 0.996      |
| Base Qwen3-1.7B (zero-shot)  | —        | 0.600              | —           | 0.999      |


The fine-tune bought 31 points of severity-weighted recall over the base model. The LoRA is doing real work, not the prompt.


Why the small model wins the money metric: it never misses a high-severity exception, and those four classes carry about two-thirds of the benchmark's total severity weight: AMOUNT_MISMATCH (73/73), FX_CONVERSION_ERROR (32/32), BENEFICIARY_MISMATCH (42/42), COUNTERPARTY_MISMATCH (31/37) are all caught or near-perfect. DeepSeek distributes its errors more evenly across severities, which produces a better raw-accuracy number and a worse severity-weighted one. It also emits unparseable output on 0.4% of tasks and spends reasoning tokens before every answer, even in a domain where the answer is a fixed six-key JSON object. The small model, fine-tuned on exact JSON targets, is parse-disciplined by construction: 100% of 800 tasks parsed.


## The system around it


The model is the decision layer of a cadence-driven pipeline, not a standalone script. Kafka streams raw pairs and verdicts between services. A Postgres ledger is the single writer of the audit trail. Every decision service reads and writes through it over HTTP, so if Kafka goes down, entries still land with a `source: system` fallback instead of silently disappearing. Temporal Cloud hosts a durable human-in-the-loop workflow: a review opens, persists to the ledger, and the workflow blocks on a named signal (`review-resolution`, not the default handler name — the Python SDK will silently buffer signals sent to the wrong name if you don't set this explicitly) with a 24-hour timeout. A human resolution or a timeout both produce a terminal, audited state. This loop was run end-to-end against the real Temporal Cloud, not a local test double: workflow start, ledger entry, human signal, final verdict with `source: human`, verified via a GET request against the ledger API.


Four scheduled workflows run the parts of the system that never stop: a contamination probe every night at 03:00 checking the latest production dataset against the published benchmark's signatures, a judge recalibration every Monday at 04:00 measuring Cohen's kappa against a fresh golden set, a benchmark matrix on demand across three seeds for release gating, and a drift check every hour comparing the live exception-type distribution against a baseline with the population stability index. A PSI above 0.10 fires a retrain trigger. None of this is ad-hoc cron; every schedule has an explicit trigger condition and writes its own event to an audit topic.


## Three findings I'm keeping


**Rebalancing the training mix made things worse.** The obvious fix for a low-recall class is more training data for it. I rebalanced the exception mix: duplicates from 8% to 22% of training examples, field corruption from 8% to 15%, cutting amount, partial-match, and value-date classes to make room, and retrained for 590 iterations. Severity-weighted recall fell from 0.913 to 0.723. HIGH-severity recall fell from 1.0 to 0.89. The classes I cut collapsed: partial-match correct predictions went from 26 out of the benchmark's total down to 1, value-date-mismatch from 37 to 8. The classes I boosted barely moved: duplicate correct went from 0 to 1. The training distribution has to match the deployment distribution the benchmark measures. Cutting a class's training share doesn't free up capacity for another class to learn faster. It destroys recall on the class you cut, in exchange for almost nothing on the class you boosted.


**More data doesn't fix the duplicate blind spot.** The model misses every duplicate in the benchmark (0 out of 31). The signal is a trimmed statement reference matching an already-booked ledger reference, and the model learned a simpler, wrong rule: if every visible field agrees, it's a match. More duplicate examples in training (the rebalancing study above tripled them) didn't move that number. The fix isn't more data, because this isn't a data-count problem. The model never learned to check reference equality as a distinct signal from field agreement. The actual fix is architectural: a rule-based verifier pre-check in front of the model that catches duplicates before they're ever scored, which is a five-line string comparison, not a training problem.


**A judge rubric fix that helped one judge and broke another.** The system uses an LLM as a judge for weekly recalibration: score a fresh golden set against the verifier's oracle labels and measure Cohen's kappa, with 0.85 as the target from Airbnb's eval-driven-development playbook. The first measurement, on a 100-task golden set, put both candidate judges at the same disappointing number: DeepSeek at kappa 0.7407, the fine-tuned local model acting as its own judge at 0.7361. Diffing both judges' mistakes against the oracle labels showed the same three classes failing for both: value-date mismatch defaulted to match four times each, partial-match scattered across four different wrong labels, field corruption confused with match or an unrelated exception type. The prompt those judges were given never stated the actual rule for any of those three classes. It named them as valid exception types and left the definition implicit.


So I wrote a second prompt, judge-only, that states the missing rules explicitly: value-date mismatch requires the date to fall on a non-business day or to land more than two calendar days after the booking date; partial-match requires a token-subset or one-token-drift relationship between names, or a prefix truncation for compact identifiers like BICs, with a fully different name being a hard mismatch instead; field corruption is an anomaly not explained by any other exception type; duplicate is a trimmed statement reference equal to the ledger reference. None of this is new information. It's already how the verifier itself works, never surfaced to the judge in text before.


Rerunning the same golden set with the new prompt: DeepSeek's kappa went to 0.9037, agreement from 82% to 93%, clearing the 0.85 bar. The local fine-tuned model's kappa, on the identical prompt, dropped to 0.3672. Its confusion table explains why: it now flags value-date mismatch on 27 pairs that actually match, and partial-match on another 15, up from missing those classes entirely before. The rubric fix hurt the local judge, and hurt it in the opposite direction from what it fixed in DeepSeek. The mechanism is straightforward once you see it: the local model _is_ the fine-tuned worker, trained on one fixed prompt baked into every training example. DeepSeek has never seen that prompt before either, so extra instructions are more context it can follow. The local model has only ever seen its original prompt; extra rules it wasn't trained on push it off the distribution it learned, and it starts guessing. DeepSeek is now the production judge for the weekly schedule. Getting the local model to judge well would need a judge-specific fine-tune trained on the new prompt, not more text pasted into the one it already has.


## Honest limits


All data is synthetic. There is no live financial data, no real counterparties, no production loss history. The methodology is the subject: a benchmark whose validity is measured, a model whose behavior is scored on the axis that matters, and a system whose every decision is auditable. The numbers are self-measured on a self-built benchmark, which is exactly the claim: the measurement is the product.


The duplicate blind spot is real and unfixed at the model level. It's caught by the verifier pre-check design, not yet wired into the live gate service. Judge calibration is half-closed: DeepSeek clears the bar, the local judge doesn't, and that gap stays open until a judge-specific fine-tune happens. A 16GB laptop also caps training at 1.7B-class models; a 7B+ comparison needs a cloud GPU and hasn't been run.


## What this is for


The point of the exercise is not that 1.7B beats DeepSeek. It is that a narrow, well-defined operational task can be solved by a small model you can fine-tune and run entirely on a laptop, measured by a benchmark whose validity you can defend, and operated inside a system whose every decision is audited. The frontier model's number on the leaderboard is not what your ops team cares about. What they care about is whether a missed exception costs money. That is a metric you can own.

---

More posts: https://caio.theodoro.dev/blog.md · About the author: https://caio.theodoro.dev/about.md · Machine-readable index: https://caio.theodoro.dev/llms.txt