Structured Outputs Changed How I Think About LLM Pipelines

The change that made the most practical difference in the reliability of LLM-based systems over the past two years was not a new model, not a new prompting technique, and not a new retrieval architecture. It was constrained decoding: the ability to specify, at the inference level, that the model's output must conform to a typed schema. This is not a flashy capability. It is a boring one, and it addressed a failure mode that was quietly responsible for a large share of production incidents in LLM pipelines.

The problem it solved

The standard approach to getting structured data out of an LLM, before constrained decoding, was prompt engineering. You'd write a system prompt that said something like "return your answer as a JSON object with the following fields: name (string), confidence (float between 0 and 1), category (one of: A, B, C)." Then you'd parse the output. Then you'd handle the cases where the model returned valid JSON wrapped in a markdown code block, and the cases where it returned the explanation first and the JSON second, and the cases where it returned a slightly different key name ("confidence_score" instead of "confidence"), and the cases where it returned a string for confidence instead of a float, and the cases where it returned a category that wasn't in the allowed set.

The failure rate per call was small, maybe 2% to 5% depending on the model and the schema complexity, but in a multi-step pipeline, small per-step failure rates compound badly. A three-step pipeline where each step has a 3% failure rate has a roughly 9% end-to-end failure rate. A five-step pipeline at the same per-step rate has nearly 14% failure. The failures required retry logic, fallback prompts, validation layers, and monitoring. The code around the LLM calls grew to be significantly larger than the calls themselves.

Constrained decoding eliminates this failure mode at the output layer. The model's token sampling is constrained by the schema at generation time: at each token position, only tokens consistent with the schema are valid candidates. The output is structurally guaranteed to match the schema, not because the model has understood the prompt's formatting instructions, but because it cannot produce output that violates the schema.

What the reliability curve looks like now

The failure mode that constrained decoding eliminates is "output doesn't parse." That's gone. What remains is a different and more tractable problem: "output parses but is semantically wrong." The JSON is valid, the schema is respected, and the values are wrong.

This is a better problem to have. Semantic errors, such as a confidence score that's poorly calibrated or a category that's technically valid but incorrectly assigned, are the kind of errors that are addressed by better prompting, better few-shot examples, better retrieval context, or fine-tuning. They are model quality issues, not pipeline engineering issues. You can evaluate them systematically, measure improvement, and iterate.

Parse failures were different: they were unpredictable, they cascaded through multi-step pipelines, they were hard to reproduce because they depended on subtle variations in model output, and they required defensive engineering (try/except, retry logic, output normalization) that cluttered the codebase without improving model quality. Eliminating that class of failure improves both development experience and production reliability.

How constrained decoding works

The mechanism matters because it determines the constraints on what schemas you can enforce. At inference time, the model samples the next token from a probability distribution over the vocabulary. In unconstrained generation, all tokens in the vocabulary are candidates, weighted by their probability under the model. In constrained decoding, a mask is applied at each token position that zeros out the probabilities of all tokens inconsistent with the current parse state of the schema.

If the schema says the next token should be a number between 0 and 1, then tokens corresponding to non-numeric characters are masked out. If the schema says the current field is an enum with values ["A", "B", "C"], then after the opening quote, only tokens that begin valid enum values are candidates.

The constraint is enforced at the sampling level, not the post-processing level. This means the model cannot produce a valid JSON object with invalid field values, even if it would have been inclined to. The sampling procedure cannot produce those tokens in the constrained schema state.

The practical limit: constrained decoding enforces structural constraints (types, allowed values, required fields) but not semantic constraints (the value must be logically consistent with the other values, the confidence must reflect the actual model uncertainty). Those remain prompt engineering and evaluation problems.

The downstream effect on pipeline design

Constrained decoding changes pipeline design in a subtle way: it moves the trust boundary. In an unconstrained pipeline, every downstream component that consumes LLM output has to defensively handle the possibility of malformed output. Validation logic, error handling, and retry infrastructure are spread throughout the pipeline because the failure point, malformed output, can occur anywhere a model is called.

In a pipeline built on constrained decoding, the structural guarantee is provided at the inference layer, and downstream components can trust the schema. The validation logic moves to the semantic layer, checking that values are plausible, not just well-formed, which gives the system a cleaner separation of concerns. The pipeline code is simpler, the error handling is more targeted, and the failure modes are easier to characterize.

This is the change that, in practice, has allowed multi-step LLM pipelines to become reliable enough for production use in tasks that require chaining several model calls. A five-step pipeline where each step is constrained to produce a typed output can be built and maintained without the kind of defensive engineering overhead that made earlier pipeline architectures painful.

It also changes how teams should version prompts and schemas. Once a model output is consumed by code, the schema is an interface, not a formatting preference. Adding a required field, renaming an enum value, or narrowing a type can break downstream consumers in the same way that changing an API response can break a client. The mature pattern is to treat schema changes as contract changes: version them, test them against representative inputs, and keep examples tied to the schema version they are meant to exercise.

What this hasn't changed

Constrained decoding doesn't address the semantic quality of model outputs. A model that is poorly prompted, poorly fine-tuned, or operating on inadequate context will produce structured nonsense: valid JSON with wrong values. Constrained decoding provides no protection against this. The move from parse errors to semantic errors is progress; it is not a solved problem.

It also doesn't address latency. Constrained decoding has overhead because the masking operation at each token position adds compute, though the overhead is typically small relative to the model's total inference cost. For high-throughput applications where inference latency is a primary concern, the cost may be meaningful.

And it doesn't change the fundamental challenge of designing schemas that capture what you actually need. A well-designed schema is harder to specify than it looks. The fields need to be the right granularity: coarse enough that the model can fill them reliably, fine enough that they contain the information the downstream system needs. The allowed values for enums need to cover the distribution of cases, not just the cases you thought of. This is a design problem, not a technical one, and constrained decoding provides no guidance on it.

The best schemas I have seen in production are not the most detailed ones. They are the ones whose fields map cleanly to downstream decisions. If no system behaves differently when a field changes, that field is probably commentary masquerading as structure. If two fields always change together, the schema is probably encoding a distinction the pipeline does not actually use. Constrained decoding makes structure reliable, which makes bad structure more expensive to ignore.

The practical upshot

For anyone building multi-step LLM pipelines: adopt constrained decoding for any output that needs to be consumed programmatically. Use a schema library (Pydantic in Python, Zod in TypeScript, or equivalent) to define your output schemas, use a library that integrates constrained decoding at the inference layer, and treat the schema as part of the interface contract of each pipeline step.

The alternative, prompt-based output formatting with defensive parsing, is survivable in small pipelines and a maintenance burden in large ones. The reliability improvement from constrained decoding is not marginal. It's the difference between a pipeline that requires significant defensive engineering to run reliably and one that can be built and reasoned about like other typed software systems.