Structured Output

The intuition

Humans tolerate messy answers. Software does not. If your service expects { "intent": "refund", "priority": 2 }, a polite paragraph or a markdown-fenced almost-JSON object is a production incident.

Structured output is how you make the model speak machine. The weak approach is asking nicely ("respond in JSON"). The strong approach is putting a grammar over the tokenizer so illegal tokens cannot be chosen — like forcing a form-filler to use only the dropdown values, one character at a time, instead of hoping they type the form correctly.

Even perfect form-filling is not perfect truth. A schema can force every field to be present; it cannot know whether age: 250 is right. So production systems combine constrained decoding with validation, retries that feed errors back, and escape hatches (null, unknown) when the honest answer is absence.

Why it exists

Production LLM calls usually feed code, workflows, and tools — not a reading human:

1. Parsers need contracts. Regexing prose for fields is brittle under sampling variance.
2. Prompt-only JSON fails under load. Markdown fences, preambles, trailing commas, and key drift appear exactly when traffic and diversity rise.
3. JSON mode is not enough. Syntactic JSON can still omit keys, invent enums, and wrong-type fields.
4. Tool calling needs typed arguments. Agents dispatch functions; arguments must match schemas or the tool layer becomes an exception factory.
5. Retries need a closed loop. When validation fails, the model needs a precise error, not a blind resample.

Alternatives lose. Prompt-and-pray is fine for prototypes and hostile in prod. Only client-side repair (LLM as free text + heroic parsers) multiplies latency and still fails. Fine-tuning for format can help tone/format at scale but is heavier than grammar constraints for schema adherence. Deterministic temperature does not enforce structure.

On the governed enterprise platform, extraction routes (ticket fields, metadata, tool arguments) use strict schemas; the gateway validates with Pydantic/Zod-class models before side effects; write tools require human approval after parse success, not instead of it.

The core idea

Most production LLM calls feed software, not humans — so the output must be machine-parseable. There are three tiers of getting structure, in increasing order of guarantee:

1. Prompt-and-pray: instruct "respond in JSON", maybe with an example. No guarantee — you'll get markdown fences, preambles ("Here's the JSON:"), trailing commas, or drift under load. Acceptable only for prototypes.
2. JSON mode: a decode-time flag guaranteeing syntactically valid JSON — but not your schema. Keys can be missing, types wrong, enums invented. Considered legacy now that schema-constrained modes exist.
3. Structured outputs (constrained decoding): you supply a JSON Schema; the runtime guarantees output conforms to it — every required key present, types correct, enums valid. OpenAI ships this as response_format: json_schema with strict: true (100% schema compliance in their evals vs ~86% for prompt-based approaches); Anthropic as output_config.format plus strict: true on tool definitions; open-source stacks via Outlines/XGrammar-style grammar-constrained decoding in vLLM.

The fourth pattern is function calling as extraction: define a "tool" whose input schema is your target structure and force the model to call it. Before native structured outputs this was the standard trick (the model emits validated arguments = your extracted object); it remains the right shape when the model genuinely dispatches actions, and strict mode now applies to tool arguments too.

Even with constrained decoding, keep validation + retry in the pipeline: schema conformance ≠ correctness (a valid age: 250 is still wrong), refusals and token-limit truncation bypass the guarantee, and not every provider/model supports every schema feature. The production loop is: call → parse/validate (Pydantic/Zod) → on failure, retry with the error message fed back → after N failures, dead-letter and alert.

How it actually works

Constrained decoding mechanics. The schema is compiled to a grammar (effectively a finite-state machine over tokens). At each decode step, the runtime masks logits: tokens that would violate the grammar get probability −∞, and sampling proceeds over only the legal tokens. The model literally cannot emit an invalid character at any position. Cost profile: first request with a new schema pays a compilation cost (OpenAI caches compiled schemas ~24h); afterwards overhead is minimal. This is also why arbitrary schema features aren't supported — everything must compile to that token-level automaton.

Typical schema restrictions (OpenAI strict mode; Anthropic similar): all objects need additionalProperties: false; every property effectively required (model optionality via union with null); no recursive schemas on some providers; no numeric bounds (minimum/maximum) or string length constraints — SDKs strip those and you re-validate client-side. description fields on properties still matter: the grammar constrains form, the descriptions steer content.

SDK layer. In practice you define a Pydantic model (Python) or Zod schema (TypeScript), and the SDK derives the JSON Schema, sends it, parses the response, and hands back a typed object (client.messages.parse(...) / parsed_output; OpenAI client.responses.parse). The Instructor library popularized this pattern (patching function-calling into "return this Pydantic model" with automatic re-ask on validation error) and remains useful for provider-agnostic code.

Escape hatches that break the guarantee — always check before parsing:

Retry-on-parse-fail pattern (the version worth implementing carefully):

for attempt in range(3):
    raw = call_llm(messages)
    try:
        return TargetModel.model_validate_json(raw)   # schema + semantic validators
    except ValidationError as e:
        messages.append(assistant(raw))
        messages.append(user(f"Invalid: {e}. Return corrected JSON only."))
raise ExtractionFailed  # -> dead-letter queue / human review

Feeding the specific validation error back is what makes retries converge; blind re-tries at temperature 0 mostly reproduce the failure.

The flows

A worked example

Extract expense fields from an employee email on the governed enterprise platform:

Hi, I booked flights to the Berlin conference — €1,240 with Lufthansa
on 12 Sep. Need cost center 8841 if possible. Thanks, Maya.

Target schema (conceptual):

Illustrative good object:

{
  "vendor": "Lufthansa",
  "amount": 1240,
  "currency": "EUR",
  "cost_center": "8841",
  "category": "travel",
  "evidence": "flights to the Berlin conference — €1,240 with Lufthansa"
}

Constrained decoding ensures keys exist and enums are members — OpenAI reported 100% schema compliance vs ~86% for prompt-based approaches in their structured-output launch evals. Client-side validators still check: amount > 0, cost center format, and that evidence is a substring of the source (light grounding).

Now remove the amount from the email and require amount: number with no null. The grammar forces some number — often a plausible invention. That is schema-forced hallucination (hallucination.md).

Common drill-downs

JSON mode vs structured outputs? JSON mode guarantees syntax only — valid JSON, arbitrary shape. Structured outputs guarantee conformance to your supplied schema via constrained decoding. JSON mode is legacy; use schema-constrained mode wherever available.

How does the API guarantee schema conformance? Grammar-constrained decoding: the schema compiles to a token-level automaton; at each step, logits for grammar-violating tokens are masked to −∞ before sampling. Invalid output is unrepresentable, not just discouraged.

When would you use function calling instead of structured outputs for extraction? When the structure represents an action to dispatch (the model decides whether/which tool with what args), when you need the model to choose among several schemas, or on stacks where tool-argument validation (strict: true) is the available guarantee. For "always return this one shape", response-level structured output is simpler.

Structured outputs guarantee the schema — why still validate? Refusals and truncation bypass the guarantee; unsupported constraints (ranges, lengths) aren't enforced server-side; and semantic validity (real dates, cross-field consistency, non-fabricated values) is out of scope for any grammar. Pydantic/Zod at the boundary catches all three.

Design the retry strategy for a parse/validation failure. Retry with the validation error appended to the conversation (self-repair), cap attempts (2–3), consider bumping temperature or escalating to a stronger model on the last attempt, then dead-letter with the raw output logged for review. Track retry rate as a health metric.

How do you handle 'field not present in source' without hallucinated fills? Make the field nullable or add an explicit not_found option, instruct that absence beats guessing, and few-shot an example where the correct output uses null. Constrained decoding without an out forces fabrication.

How does streaming interact with structured output? You receive incremental fragments of the JSON string; you can't fully parse until complete. Options: accumulate then parse; partial/streaming JSON parsers for progressive UI; or stream only for display while a non-streamed call feeds the pipeline.

Schema for classify-into-8-categories — what matters? Enum-constrained field (grammar enforces membership — no invented labels), category descriptions in the schema/prompt, an other/unknown member, optionally a reasoning field before the label, and a threshold/routing plan using logprobs or a confidence field.

Production concerns

• Valid ≠ true. Constrained decoding can force an answer shape when the honest answer is "not present" — the model fills required fields with plausible fabrications. Design schemas with explicit outs: nullable fields, an "unknown" enum member, or a confidence/evidence field. This is the top silent-quality bug in extraction pipelines.
• Schema design is prompt engineering. Field names and descriptions are read by the model. date vs invoice_due_date_iso8601 changes accuracy. Order fields so reasoning-dependent fields come after the fields they depend on (generation is left-to-right); some teams add a leading free-text reasoning field as embedded CoT.
• Latency/cost. Structured output constrains decoding but doesn't shorten it; deeply nested schemas invite verbose outputs. Flat, minimal schemas decode faster. First-call schema compilation adds one-time latency — pre-warm in deploy pipelines if p99 matters. See latency.md and cost.md.
• Versioning. Schema changes are API contract changes: version them, and remember the compiled-schema cache means schema churn also churns latency.
• Cross-provider portability. Strictness levels and supported JSON Schema subsets differ per provider and per model. Abstract behind your own typed layer (Pydantic/Zod at the boundary) so switching providers is a mapping change, and always keep client-side validation even where the provider "guarantees" conformance.
• Metrics to watch: parse-failure rate, retry rate, refusal rate, truncation rate, and field-level accuracy on an eval set. Parse failures ≈ 0 with structured outputs — if you're seeing them, you're hitting an escape hatch. See observability.md.

Test yourself

Go deeper

• OpenAI docs: Structured Outputs — schema mode, strict tool use, refusal handling, supported JSON Schema subset.
• Introducing Structured Outputs in the API — OpenAI — the launch post with the 100%-vs-86% schema-adherence numbers and design rationale.
• Anthropic docs: Structured outputs — output_config.format, strict tool use, messages.parse() with Pydantic/Zod, schema limitations.
• Deep Dive into LLMs like ChatGPT — Andrej Karpathy — background on why free-text generation is unreliable as an interface (sampling, hallucination), which is the case for constraining it.

Where this connects

• Tool Calling — schemas for actions; strict arguments; error handling when tools reject inputs.
• Hallucination — forced fills are a structural hallucination class; design outs and verification.
• Sampling & Determinism — constraints beat temperature for correctness.
• Evals & Testing — measure parse rates and field accuracy separately.