Structured Output

The intuition

Humans tolerate messy answers. Software does not. If your service expects { "intent": "refund", "priority": 2 }, a polite paragraph or markdown-fenced almost-JSON is a production incident.

Structured output is how you make the model speak machine. Asking nicely ("respond in JSON") is weak. The strong approach puts a grammar over the tokenizer so illegal tokens cannot be chosen — like forcing a form-filler to use only the dropdown values, one character at a time.

Even perfect form-filling is not perfect truth. A schema can force every field present; it cannot know whether age: 250 is right. Production systems pair constrained decoding with semantic validation, retries that feed errors back, and escape hatches (null, unknown) when the honest answer is absence.

Why it exists

Most production LLM calls feed parsers, workflows, and tools — not a reading human. Free text drifts into fences, preambles, trailing commas, and invented keys under traffic and input diversity. JSON that merely parses is still not enough if keys are missing, types wrong, or enums invented. Agents make this worse: tool layers become exception factories unless arguments match a schema.

Alternatives lose differently. Prompt-and-pray is fine for prototypes and hostile in production. Client-side repair only multiplies latency and still fails. Fine-tuning for format is heavier than grammar constraints for schema adherence. Temperature 0 does not enforce structure at all.

On the governed enterprise platform, extraction routes use strict schemas; the gateway validates with Pydantic/Zod-class models before side effects; write tools require human approval after parse success, not instead of it.

The core idea

Three tiers of structure, in increasing guarantee:

1. Prompt-and-pray — instruct "respond in JSON," maybe with an example. No guarantee: fences, preambles, key drift under load. Prototypes only.
2. JSON mode — guarantees syntactically valid JSON, not your schema. Keys can still be missing, types wrong, enums invented. Legacy wherever schema mode exists.
3. Structured outputs (constrained decoding) — you supply a JSON Schema; the runtime guarantees conformance (required keys, types, enums). OpenAI: text.format / response_format with type: "json_schema" and strict: true (launch evals: 100% schema compliance vs ~86% prompt-based). Anthropic: output_config.format plus strict: true on tools. Self-hosted: XGrammar (vLLM/SGLang default) or Outlines.

A fourth pattern is function calling as extraction: define a tool whose input schema is your target and force the model to call it — right when the model must choose among actions. Schema conformance is still not correctness. Production loop: call → check stop reason / refusal → parse and validate (Pydantic/Zod) → on failure, retry with the error fed back → after N failures, dead-letter and alert.

How it actually works

Constrained decoding. The schema compiles to a grammar (finite-state or pushdown automaton over tokens). At each step illegal tokens get probability −∞; sampling proceeds only over legal ones — so the model cannot emit an invalid character. That is why schema support is a subset: everything must compile to that automaton. The first request with a new schema pays a compilation cost (simple schemas often under ~10s on OpenAI-class stacks; complex ones longer). Providers cache the artifact — Anthropic documents ~24h from last use; OpenAI caches without a public TTL — then overhead is small.

Schema rules that still bite. Strict mode almost always wants additionalProperties: false and every property required (optionality via union with null). Nesting and recursion have provider caps. Bounds and lengths are not uniform: current OpenAI structured outputs support many (minimum/maximum, pattern, format); Anthropic SDKs often strip those keywords, push them into descriptions, and re-validate client-side. Fine-tunes and older snapshots may support less. Property description fields still matter: the grammar constrains form; names and descriptions steer content.

SDK layer. Define a Pydantic model or Zod schema; the SDK derives JSON Schema, sends it, and returns a typed object (client.responses.parse / client.messages.parse). Instructor popularized provider-agnostic "return this model" with automatic re-ask on validation error.

Escape hatches — check before treating output as typed: safety refusals return refusal text, not schema JSON; max_tokens truncates mid-object (check stop reason first); unsupported features 400 or leave a field unconstrained; streaming yields partial deltas (accumulate then parse).

for attempt in range(3):
    raw = call_llm(messages)
    try:
        return TargetModel.model_validate_json(raw)
    except ValidationError as e:
        messages.append(assistant(raw))
        messages.append(user(f"Invalid: {e}. Return corrected JSON only."))
raise ExtractionFailed  # -> dead-letter / human review

Feed the specific validation error so retries converge. Blind re-tries at temperature 0 mostly reproduce the failure.

The flows

A worked example

Extract expense fields from an employee email on the governed enterprise platform:

Hi, I booked flights to the Berlin conference — €1,240 with Lufthansa
on 12 Sep. Need cost center 8841 if possible. Thanks, Maya.

Target shape: vendor, amount, currency (enum + unknown), cost_center, category (enum + unknown), short evidence — nullable where the source may be silent.

{
  "vendor": "Lufthansa",
  "amount": 1240,
  "currency": "EUR",
  "cost_center": "8841",
  "category": "travel",
  "evidence": "flights to the Berlin conference — €1,240 with Lufthansa"
}

Constrained decoding ensures keys and enums. Client validators still check amount > 0, cost-center format, and that evidence is a substring of the source.

Now remove the amount and require amount: number with no null. The grammar forces some number — often a plausible invention. That is schema-forced hallucination (Hallucination).

Production concerns

Valid ≠ true is the top silent-quality bug. Give optional fields a null or unknown out; prefer evidence on high-stakes extraction; measure field-level accuracy and abstention — not only parse rate.

Schema design is prompt engineering. Names and descriptions steer content (date vs invoice_due_date_iso8601). Generation is left-to-right, so put reasoning-dependent fields after what they depend on; some teams add a leading free-text reasoning field before the final label.

Latency, cost, versioning. Constraints do not shorten generation; flat schemas decode faster. First-call compilation can dominate cold p99 — pre-warm in deploy probes, and avoid schema-string churn that busts the cache. Schema changes are API contracts: version them. Supported subsets differ by provider; abstract behind your own typed layer. Watch parse-failure, retry, refusal, truncation rates, and field accuracy (Latency, Cost, Observability).

Common drill-downs

JSON mode vs structured outputs? JSON mode: syntax only. Structured outputs: schema conformance via constrained decoding. Prefer schema mode.

When function calling instead of response-format structured output? When the structure is an action to dispatch, when the model must choose among several schemas (or call nothing), or when only tool-argument strict mode is available. For one fixed record shape, response-level structured output is simpler.

Why still validate after a schema guarantee? Refusals and truncation bypass it; some constraints never reach the server; semantic validity is out of scope for any grammar.

Missing source fields without hallucinated fills? Nullable fields or unknown/not_found; instruct that absence beats guessing; few-shot a null example. No out → forced fabrication.

Streaming? Accumulate until complete, check stop reason, validate fully, then dispatch tools. Partial parsers are UI-only.

Test yourself

Go deeper

• OpenAI docs: Structured Outputs — schema mode, strict tools, refusals, supported subset.
• Introducing Structured Outputs in the API — OpenAI — 100%-vs-86% adherence numbers and constrained-decoding rationale.
• Anthropic docs: Structured outputs — output_config.format, strict tools, grammar cache, schema limits.
• vLLM structured outputs — XGrammar/Guidance for self-hosted constrained decoding.
• Instructor — Pydantic-first extraction with re-ask across providers.

Where this connects

• Tool Calling — schemas for actions; strict arguments; error handling when tools reject inputs.
• Hallucination — forced fills are a structural hallucination class; design outs and verification.
• Sampling & Determinism — constraints beat temperature for format correctness.
• Evals & Testing — measure parse rates and field accuracy separately.