Structured Outputs Changed How I Think About LLM Pipelines

Most LLM pipeline reliability work is scored on the prompt or the next model. You rewrite the system message, you swap a checkpoint, you add a retrieval hop. Those moves change what the model is asked, or which model is asked. They do not change whether an ill-typed token can leave the sampler. The scoreboard is whether the inference layer can refuse that token.

The change that held up over the last two years was constrained decoding. You specify, at inference, that the output must conform to a typed schema. People treated prompting and new models as the reliability upgrade. The sampler's refusal is the upgrade. Constrained decoding is a boring capability, and it cut a class of production incidents that formatting instructions never closed.

The old pipeline

The standard approach, before the constraint, was a prompt. You asked for a JSON object with name (string), confidence (float between 0 and 1), and category (one of: A, B, C). Then you parsed. Then you handled the valid JSON wrapped in a markdown code block, the explanation first and the JSON second, the renamed key (confidence_score instead of confidence), the string where a float belonged, and the category that was not in the set.

The failure rate per call was small, maybe 2% to 5% depending on the model and the schema. In a multi-step pipeline those rates compound. A three-step pipeline at 3% per step has a roughly 9% end-to-end failure rate. A five-step pipeline at the same per-step rate has nearly 14%. Those percentages are arithmetic on named rates, not a production incident log.

The failures required retry logic, fallback prompts, validation layers, and monitoring, because a single malformed object could stall every step downstream of the call. The code around the calls grew larger than the calls. You spent more time unwrapping markdown and renaming keys than you spent on the task the model was supposed to do.

Constrained decoding removes this at the output layer. Token sampling is constrained by the schema at generation time: at each token position, only tokens consistent with the current parse state are valid candidates. The output matches the schema because the sampler cannot emit a token that violates it, not because the model understood the formatting instructions.

What remains

The failure mode that goes away is parse failure. What remains is output that parses and is semantically wrong. The JSON is valid and the schema is respected; the values are wrong.

This is a better problem to have. A poorly calibrated confidence, or a category that is legal and incorrectly assigned, is a model quality issue. You evaluate it, measure it, and iterate with prompts, few-shot examples, retrieval, or fine-tuning. Semantic errors sit on a scoreboard you can move. Parse failures did not, because they were unpredictable, they cascaded through multi-step pipelines, they were hard to reproduce from small variations in model output, and they required try/except, retries, and output normalization that cluttered the codebase without moving model quality. You stop writing retry wrappers, and the pipeline stops dying on a markdown fence.

How the mask works

The mechanism matters because it sets what you can enforce. At inference the model samples the next token from a probability distribution over the vocabulary. In unconstrained generation, every token in the vocabulary is a candidate, weighted by its probability under the model. In constrained decoding, a mask is applied at each token position that zeros out the probabilities of all tokens inconsistent with the current parse state of the schema.

If the schema says the next token is a number between 0 and 1, non-numeric tokens are masked. If the current field is an enum with values ["A", "B", "C"], then after the opening quote only tokens that begin a valid enum value are candidates.

The constraint is at sampling, not post-processing. The model cannot emit a valid JSON object with an illegal field value, even if it would have been inclined to. Some providers already ship JSON mode; when that mode masks at sample time, it is the same constraint.

The mask enforces types, allowed values, and required fields — not whether the values are logically consistent. Whether confidence reflects actual uncertainty stays a prompt and evaluation problem.

The trust boundary

Constrained decoding moves the trust boundary. In an unconstrained pipeline, every downstream component that consumes model output has to handle the possibility of a malformed object. Validation, error handling, and retry sit everywhere a model is called, because the failure can occur at any of those calls.

In a constrained pipeline, the structural guarantee lives at inference, and downstream components can trust the schema. Validation moves to the semantic layer, checking that values are plausible once they are already well-formed. The pipeline is simpler, the error handling is more targeted, and the failure modes are easier to name.

A five-step pipeline where each step is constrained to a typed output can be built and maintained without the defensive overhead that made earlier architectures painful. You can chain the calls because each step refuses an ill-typed token before the next step has to see it.

Once code consumes the output, the schema is an interface, not a formatting preference. Adding a required field, renaming an enum value, or narrowing a type breaks consumers the way changing an API response breaks a client. Version the schema, test it against representative inputs, and keep examples tied to the schema version they exercise.

What this hasn't changed

Constrained decoding does not address semantic quality. A poorly prompted, poorly fine-tuned, or under-contextualized model produces structured nonsense: valid JSON with wrong values. The mask does not catch wrong values. The move from parse errors to semantic errors is progress. Structured nonsense is still a model problem.

It also does not address latency. The mask at each token position adds compute. The overhead is typically small relative to the model's total inference cost; for high-throughput work where latency is the primary concern, the cost may be meaningful.

The mask does not design the schema for you. Fields need the right granularity: coarse enough that the model can fill them reliably, fine enough that they contain the information the downstream system needs. Enum values need to cover the distribution of cases, not the cases you thought of first. That is a design problem, and constrained decoding provides no guidance on it.

The best schemas I have seen in production are not the most detailed ones. They are the ones whose fields map to downstream decisions. If no system behaves differently when a field changes, that field is commentary masquerading as structure. If two fields always change together, the schema is encoding a distinction the pipeline does not use. Constrained decoding makes structure reliable, which makes unused structure more expensive to keep.

The alternative, prompt-based output formatting with defensive parsing, is survivable in a small pipeline and a maintenance burden in a large one. For any output consumed by code, put the constraint at the inference layer. Define the schema in a library you already version — Pydantic in Python, Zod in TypeScript — wire the mask at sampling, and treat a required-field add as a contract change. If the output is not consumed by code, skip the constraint.