Caio Theodoro Caio Theodoro

Blog

Notes on engineering, ML, and building things on the web.

Ornith-1.5 Wrote Its Own Training Data. Distribution Match Won

Ornith-1.5 proposes its own training curriculum. I audited it with contamination-controlled data on a verifiable niche, construction pay-app review, and found the harder tasks it invents only help as a supplement, not a replacement, for distribution-matched data.

ML Agent Systems
Aug 2026

Suture: Catching Underwriting Errors GPT-5.6 Missed

An 8B vision-language adapter trained to diff underwriting binders against issued policies catches far more errors than GPT-5.6 Luna does zero-shot. The real story is the three measurement gates that gave false confidence before the model actually worked.

ML Engineering
Aug 2026

The Third Number

Agents that touch money get evaluated on accuracy or price. Neither one catches the failure that actually costs the most: a single bad decision with an outsized loss. This is about the metric that does, and why nothing measured it before LossBench.

ML Agent Systems
Aug 2026

Can A 1.7B Model Beat a Frontier on Reconciliation Exceptions?

A Qwen3-1.7B model fine-tuned on a laptop catches more high-severity reconciliation exceptions than DeepSeek v4-flash, including every one in the test set. Covers the benchmark, the training run, and the approaches that failed along the way.

ML Engineering
Jun 2026

Copying ARC-AGI's Benchmark Method, Then Stress-Testing It

Benchmarks decay once models start training on them. I rebuilt ARC-AGI-3's benchmark methodology as a pipeline and tested which parts of it actually hold up: difficulty scaling, the human calibration bar, sample size, and a contamination monitor you can validate yourself.

ML Engineering
May 2026

Evaluation Gates Are the Reliability Moat

Agent deployments do not die from average model error. They die from the first undetected one. Why evaluation gates, not the base model, are the actual product differentiator.

ML Engineering
Apr 2026

The Economics of Model Serving

Model and serving choices get treated as a quality decision and discovered to be a pricing decision once the bill arrives. Batching, quantization, tiering, and the self-host bet, as cost decisions.

ML Engineering
Apr 2026

When AI Agents Stop Taking Turns

An exploration of Perfectman's social presence architecture — replacing turn-based agent scheduling with urge-driven behavior through attention, emotion, pressure, and inhibition to produce believable online social dynamics.

Behavioral AI
Feb 2026

The Wire Format Problem in Generative UI

The wire format for model-generated UI is not a detail — it decides parsing reliability, streaming latency, and token cost, and most teams pick one by accident.

UX ML
Feb 2026

Thinking with Visual Primitives

DeepSeek's paper on the Reference Gap in multimodal reasoning: why spatial coordinates inside the chain of thought change what multi-step visual reasoning can do, and what the benchmark numbers actually tell us.

Jan 2026

RAG vs. Fine-Tune: The Build Decision

The RAG-versus-fine-tuning debate gets argued as a capability question. In production it is a cost and maintenance-surface question, decided by three axes and what a team can actually operate.

ML Engineering
Jan 2026

Structured Outputs Changed How I Think About LLM Pipelines

Constrained decoding was the boring change that mattered more than any prompting technique — it eliminated an entire class of production failures by moving the trust boundary to the inference layer.

ML Engineering
Jan 2026

When a Knowledge Graph Earns Its Keep

GraphRAG has accumulated enthusiasm and skepticism from the same people at different points in their experience. Both are justified. The question reduces to a specific structural property of your retrieval task.

ML Engineering
Nov 2025

A Year of RAG in Production

What retrieval-augmented generation actually does in production, where flat-index vector search fails, and what GraphRAG fixes — and what it doesn't.

Sep 2025

The Gap Between RLHF and Alignment

There is a gap between what RLHF achieves and what alignment implies. The gap produces sycophancy and overconfidence as structural properties, not implementation bugs — and shapes which failures to expect.

ML
Aug 2025

Synthetic Data Isn't New. The Scale Is.

The mechanism underlying synthetic data for fine-tuning is not new. What has changed is scale and accessibility. Where it works, where it breaks, and what the defensible production pattern actually looks like.

ML
Jun 2025

Simulating Markets Instead of Fitting Curves to Them

MiroFish replaces curve-fitting with agent-based social simulation for demand forecasting, seeded with FRED data, macro signals, and decades of history. What the architecture gets right, what it still cannot do, and where the evidence actually holds up.

ML Engineering
May 2025

LLMs and Time-Series Forecasting

An honest look at what LLMs can and can't do with time-series data — what the benchmarks actually test, where the genuine use case is, and when to reach for statistical methods instead.

Apr 2025

When the Problem Is Simulation, Not Prediction

A category of problem gets approached with predictive models because that's what the field knows how to build. The failure mode is specific: right on average, wrong at the moments that matter most.

ML Engineering
Mar 2025

What Superforecasters Got Right

Tetlock's superforecaster research and what it means for production ML: calibration as a feedback system, the Brier score as a standard, and why the loop almost never closes.

Feb 2025

Calibration

Why calibration matters more than accuracy in production ML systems, how miscalibration survives undetected, and what a calibrated system actually looks like.

Jan 2025

Fine-Tuning's Real Cost Isn't the Training Run

What fine-tuning actually specializes versus what it can't do, and the data and evaluation work that determines whether it's worth the cost.

Oct 2024