# Blog — Caio Theodoro > Notes on engineering, ML, and building things on the web. Every post below has a markdown twin at the linked URL. ## Posts - [Ornith-1.5 Wrote Its Own Training Data. Distribution Match Won](https://caio.theodoro.dev/blog/ornith-curriculum-audit-distribution-wins.md) — 2026-08-21 _(ML, Agent Systems)_: Ornith-1.5 proposes its own training curriculum. I audited it with contamination-controlled data on a verifiable niche, construction pay-app review, and found the harder tasks it invents only help as a supplement, not a replacement, for distribution-matched data. - [Suture: Catching Underwriting Errors GPT-5.6 Missed](https://caio.theodoro.dev/blog/suture-8b-three-gates-that-lied.md) — 2026-08-18 _(ML, Engineering)_: An 8B vision-language adapter trained to diff underwriting binders against issued policies catches far more errors than GPT-5.6 Luna does zero-shot. The real story is the three measurement gates that gave false confidence before the model actually worked. - [The Third Number](https://caio.theodoro.dev/blog/why-lossbench-significant.md) — 2026-08-12 _(ML, Agent Systems)_: Agents that touch money get evaluated on accuracy or price. Neither one catches the failure that actually costs the most: a single bad decision with an outsized loss. This is about the metric that does, and why nothing measured it before LossBench. - [Can A 1.7B Model Beat a Frontier on Reconciliation Exceptions?](https://caio.theodoro.dev/blog/reconforge-1-7b-beats-deepseek-on-the-money-metric.md) — 2026-06-08 _(ML, Engineering)_: A Qwen3-1.7B model fine-tuned on a laptop catches more high-severity reconciliation exceptions than DeepSeek v4-flash, including every one in the test set. Covers the benchmark, the training run, and the approaches that failed along the way. - [Copying ARC-AGI's Benchmark Method, Then Stress-Testing It](https://caio.theodoro.dev/blog/building-benchmarks-like-arc-measuring-whether-it-works.md) — 2026-05-11 _(ML, Engineering)_: Benchmarks decay once models start training on them. I rebuilt ARC-AGI-3's benchmark methodology as a pipeline and tested which parts of it actually hold up: difficulty scaling, the human calibration bar, sample size, and a contamination monitor you can validate yourself. - [Evaluation Gates Are the Reliability Moat](https://caio.theodoro.dev/blog/evaluation-gates-reliability-moat.md) — 2026-04-14 _(ML, Engineering)_: Agent deployments do not die from average model error. They die from the first undetected one. Why evaluation gates, not the base model, are the actual product differentiator. - [The Economics of Model Serving](https://caio.theodoro.dev/blog/economics-of-model-serving.md) — 2026-04-02 _(ML, Engineering)_: Model and serving choices get treated as a quality decision and discovered to be a pricing decision once the bill arrives. Batching, quantization, tiering, and the self-host bet, as cost decisions. - [When AI Agents Stop Taking Turns](https://caio.theodoro.dev/blog/when-ai-agents-stop-taking-turns.md) — 2026-02-28 _(Behavioral AI)_: An exploration of Perfectman's social presence architecture — replacing turn-based agent scheduling with urge-driven behavior through attention, emotion, pressure, and inhibition to produce believable online social dynamics. - [The Wire Format Problem in Generative UI](https://caio.theodoro.dev/blog/on-generative-ui.md) — 2026-02-07 _(UX, ML)_: The wire format for model-generated UI is not a detail — it decides parsing reliability, streaming latency, and token cost, and most teams pick one by accident. - [Thinking with Visual Primitives](https://caio.theodoro.dev/blog/thinking-with-visual-primitives.md) — 2026-01-24: DeepSeek's paper on the Reference Gap in multimodal reasoning: why spatial coordinates inside the chain of thought change what multi-step visual reasoning can do, and what the benchmark numbers actually tell us. - [RAG vs. Fine-Tune: The Build Decision](https://caio.theodoro.dev/blog/rag-vs-fine-tune-build-decision.md) — 2026-01-20 _(ML, Engineering)_: The RAG-versus-fine-tuning debate gets argued as a capability question. In production it is a cost and maintenance-surface question, decided by three axes and what a team can actually operate. - [Structured Outputs Changed How I Think About LLM Pipelines](https://caio.theodoro.dev/blog/structured-outputs-llm-pipelines.md) — 2026-01-11 _(ML, Engineering)_: Constrained decoding was the boring change that mattered more than any prompting technique — it eliminated an entire class of production failures by moving the trust boundary to the inference layer. - [When a Knowledge Graph Earns Its Keep](https://caio.theodoro.dev/blog/what-the-graph-actually-adds.md) — 2025-11-03 _(ML, Engineering)_: GraphRAG has accumulated enthusiasm and skepticism from the same people at different points in their experience. Both are justified. The question reduces to a specific structural property of your retrieval task. - [A Year of RAG in Production](https://caio.theodoro.dev/blog/rag-retrieval-one-year-in.md) — 2025-09-19: What retrieval-augmented generation actually does in production, where flat-index vector search fails, and what GraphRAG fixes — and what it doesn't. - [The Gap Between RLHF and Alignment](https://caio.theodoro.dev/blog/what-rlhf-actually-optimizes.md) — 2025-08-08 _(ML)_: There is a gap between what RLHF achieves and what alignment implies. The gap produces sycophancy and overconfidence as structural properties, not implementation bugs — and shapes which failures to expect. - [Synthetic Data Isn't New. The Scale Is.](https://caio.theodoro.dev/blog/on-synthetic-data.md) — 2025-06-27 _(ML)_: The mechanism underlying synthetic data for fine-tuning is not new. What has changed is scale and accessibility. Where it works, where it breaks, and what the defensible production pattern actually looks like. - [Simulating Markets Instead of Fitting Curves to Them](https://caio.theodoro.dev/blog/mirofish-demand-forecasting.md) — 2025-05-09 _(ML, Engineering)_: MiroFish replaces curve-fitting with agent-based social simulation for demand forecasting, seeded with FRED data, macro signals, and decades of history. What the architecture gets right, what it still cannot do, and where the evidence actually holds up. - [LLMs and Time-Series Forecasting](https://caio.theodoro.dev/blog/llms-and-time-series-forecasting.md) — 2025-04-18: An honest look at what LLMs can and can't do with time-series data — what the benchmarks actually test, where the genuine use case is, and when to reach for statistical methods instead. - [When the Problem Is Simulation, Not Prediction](https://caio.theodoro.dev/blog/simulation-not-prediction.md) — 2025-03-14 _(ML, Engineering)_: A category of problem gets approached with predictive models because that's what the field knows how to build. The failure mode is specific: right on average, wrong at the moments that matter most. - [What Superforecasters Got Right](https://caio.theodoro.dev/blog/what-superforecasters-got-right.md) — 2025-02-09: Tetlock's superforecaster research and what it means for production ML: calibration as a feedback system, the Brier score as a standard, and why the loop almost never closes. - [Calibration](https://caio.theodoro.dev/blog/calibration.md) — 2025-01-16: Why calibration matters more than accuracy in production ML systems, how miscalibration survives undetected, and what a calibrated system actually looks like. - [Fine-Tuning's Real Cost Isn't the Training Run](https://caio.theodoro.dev/blog/fine-tuning-when-it-makes-sense.md) — 2024-10-29: What fine-tuning actually specializes versus what it can't do, and the data and evaluation work that determines whether it's worth the cost. ## Elsewhere on this site - [Home](https://caio.theodoro.dev/index.md): Who Caio is, what he works on now, and the current role. - [About](https://caio.theodoro.dev/about.md): Background, how he works, and what he is currently focused on. - [Contact](https://caio.theodoro.dev/contact.md): Email, profiles, what to reach out about and expected response time. - [Privacy](https://caio.theodoro.dev/privacy.md): What this site collects (almost nothing) and which third parties see a request. - [Work](https://caio.theodoro.dev/projects.md): Employment history, certifications and open-source / research projects. - [Journey](https://caio.theodoro.dev/journey.md): Year-by-year timeline of the path into ML engineering. HTML version of this page: https://caio.theodoro.dev/blog. Machine-readable index: https://caio.theodoro.dev/llms.txt. Full URL list: https://caio.theodoro.dev/sitemap-index.xml.