What is structured output? JSON mode vs strict schemas
Structured output forces an LLM's reply to match a JSON schema by masking invalid tokens during decoding. JSON mode vs strict mode, with a runnable Python demo.

Structured output is an API feature that makes a language model’s reply match a JSON schema you supply, so your code can parse it without retries. It works by constrained decoding: at every step, the API removes any token that would break the schema before the model picks one. That’s different from JSON mode, which only promises syntactically valid JSON, not the keys, types and values you asked for.
Why do LLMs need structured output?
Because a model’s output is a string, and most code wants a typed object. If you ask a model in plain prose to “reply in JSON with keys label and score”, it usually does. Sometimes it adds “Sure, here is the JSON:” in front, wraps it in a Markdown code fence, renames a key, adds a trailing comma or invents an enum value. At one request a minute you fix it by hand. In a pipeline processing a million records, or an agent passing output from one step to the next, each of those is an exception.
OpenAI’s announcement of Structured Outputs on August 6, 2024 put a number on the gap. On its evals of complex JSON schema following, gpt-4-0613 scored “less than 40%”. The new gpt-4o-2024-08-06, trained to follow schemas, reached 93% on its own, and with Structured Outputs turned on it scored “a perfect 100%”. Training got it most of the way; the last 7 points came from engineering.
JSON mode vs structured outputs: what’s the difference?
The same post gives the history. OpenAI introduced JSON mode at DevDay in 2023, and while it improved reliability for valid JSON, “it does not guarantee that the model’s response will conform to a particular schema.” OpenAI’s current docs say it in one line: both produce valid JSON, but “only Structured Outputs ensure schema adherence.”
| Prompting for JSON | JSON mode | Structured outputs (strict) | |
|---|---|---|---|
| Valid JSON | Usually | Yes | Yes |
| Your keys and types | Usually | Usually | Yes |
| Enum values respected | Usually | Usually | Yes |
| How | Instructions | Constrained to JSON syntax | Constrained to your schema |
There are two places to apply it. For the final answer, OpenAI uses response_format with type: "json_schema" in Chat Completions, or text.format in the Responses API, and Anthropic uses output_config.format. For tool calls, both accept strict: true on the tool definition so the arguments always match the tool’s schema, which matters for tool calling in agents.
How does constrained decoding work?
A model generates one token at a time by sampling from a probability distribution over its whole vocabulary. Nothing stops it from picking } where a key should be. Constrained decoding changes one thing: before sampling, it sets the probability of every token that would make the output invalid to zero.
OpenAI’s post explains how it does this. Valid next tokens depend on what has been generated so far (after {"val, a { is no longer allowed), so the set has to be recomputed after every token. To do that, OpenAI converts the JSON Schema into a context-free grammar. It chose a grammar over a finite state machine or regex because grammars can express recursive structures, such as a UI element whose children are UI elements, which FSMs generally cannot. Anthropic’s docs describe the same approach as constrained decoding using “compiled grammar artifacts.”
Here’s the mechanism on a toy scale. The vocabulary is 20 tokens, the schema is {"label": "pos"|"neg"|"neutral", "score": 1..5}, and the “model” strongly prefers the right token at each position but leaves a little probability on every other token, the way a real model does. The script generates 10,000 outputs with and without the mask and validates each one.
# constrained.py: constrained decoding on a toy vocabulary. Python 3 stdlib only.
# The "model" usually prefers a token that fits the schema but sometimes drifts (chatty
# preamble, trailing comma, wrong key). Constrained decoding masks every token the
# schema does not allow at this position, then samples from what is left.
import json, random
VOCAB = ['{', '}', ':', ',', '"label"', '"score"', '"pos"', '"neg"', '"neutral"',
'1', '2', '3', '4', '5', 'Sure', ' here', ' is', ' JSON', '\n', '"Label"']
# Schema: {"label": "pos"|"neg"|"neutral", "score": 1..5}, written as a token-level grammar.
GRAMMAR = [{'{'}, {'"label"'}, {':'}, {'"pos"', '"neg"', '"neutral"'}, {','},
{'"score"'}, {':'}, {'1', '2', '3', '4', '5'}, {'}'}]
def model_weights(pos):
"""Noisy model: 30x preference for tokens that fit, but every token keeps some mass."""
ok = GRAMMAR[pos] if pos < len(GRAMMAR) else {'}'}
return {t: (30.0 if t in ok else 0.08) for t in VOCAB}
def generate(constrained, rng):
out = []
for pos in range(len(GRAMMAR)):
w = model_weights(pos)
if constrained: # the mask: zero out tokens the grammar forbids
w = {t: x for t, x in w.items() if t in GRAMMAR[pos]}
out.append(rng.choices(list(w), weights=list(w.values()))[0])
return "".join(out)
def valid(text):
try:
obj = json.loads(text)
except json.JSONDecodeError:
return False
return (set(obj) == {"label", "score"} and obj["label"] in ("pos", "neg", "neutral")
and obj["score"] in range(1, 6))
rng = random.Random(7)
for mode in (False, True):
outs = [generate(mode, rng) for _ in range(10_000)]
bad = [o for o in outs if not valid(o)]
print(f"constrained={mode!s:<5} valid {1 - len(bad) / len(outs):.1%} e.g. {outs[0]}")
if bad:
print(f"{'':19}a failure: {bad[0]!r}")
Output (Python 3.14.5):
constrained=False valid 69.1% e.g. {"label":"pos","score":3}
a failure: '{"label"2"pos","score":1}'
constrained=True valid 100.0% e.g. {"label":"neutral","score":3}
The unconstrained model puts about 95% to 99% of its probability on a valid token at each of nine positions, which sounds reliable until you multiply: about 7 in 10 outputs come out valid. The failure shown is a 2 where a colon belongs, one bad token that breaks json.loads. With the mask, invalid tokens can’t be sampled, so validity is 100% by construction. Notice the mask doesn’t change which valid token the model prefers; it only removes the invalid ones. A real implementation does this over a vocabulary of a hundred thousand or more tokens, with the grammar compiled once per schema and cached.
What are the limits of structured output?
The guarantee is about shape, not truth. A schema-valid {"label": "pos", "score": 5} can still be the wrong label. Beyond that, the vendor docs list specific limits.
First-request latency. The schema has to be compiled. OpenAI said typical schemas take under 10 seconds on the first request and complex ones up to a minute, then are cached. Anthropic says the same first-request cost applies, with compiled grammars cached for 24 hours from last use.
Refusals. A model can still refuse an unsafe request, and the refusal won’t match your schema. OpenAI returns a separate refusal field; Anthropic returns stop_reason: "refusal". Check before you parse.
Truncation. If the response hits max_tokens, the JSON is incomplete. OpenAI says output is reliable only when there’s no refusal and the response “has not been prematurely interrupted”; Anthropic’s docs say to retry with a higher limit.
Schema subset. Not all of JSON Schema is supported. As of October 2026, Anthropic’s list of unsupported features includes recursive schemas, numeric constraints like minimum and maximum, string length constraints, and additionalProperties set to anything but false, and it caps a request at 20 strict tools. OpenAI’s launch post noted that Structured Outputs was not compatible with parallel function calls at the time.
Small surprises. Anthropic’s docs warn that Claude may change the capitalization of enum values and that required properties are emitted before optional ones regardless of schema order. Compare enums case-insensitively.
Reasoning space. If your schema is only {"answer": ...}, the model has no room to think before committing. Either use a model with a separate thinking channel, as covered in what reasoning models are, or add a short field for supporting evidence before the answer field.
What I would do
Use strict structured output for anything a program reads, and keep prose for anything a person reads. Define the schema with the SDK helpers (Pydantic in Python, Zod in TypeScript), which both OpenAI and Anthropic support, so the type in your code and the schema the model sees can’t drift apart.
Then validate anyway. The constraint guarantees shape; your code still checks business rules: the ID exists, the amount is positive, the date is not in the future. On a data platform that’s the line I’d hold: schema-on-write at the model boundary, semantic checks in code, and anything that fails both gets logged with the input for review. For extraction pipelines over many documents, that combination is what lets you trust the table at the end, which matters for text-to-SQL and evaluating RAG alike.
FAQ
What is JSON mode in LLMs?
A setting that constrains the model to output syntactically valid JSON. It doesn’t enforce your keys, types or allowed values; for that you need structured outputs with a schema.
Does structured output make the model more accurate?
It makes the format reliable, not the content. A schema can help by making the model fill specific fields, but a valid object can still hold a wrong value, so validate the content separately.
Which models support structured outputs?
OpenAI supports it from gpt-4o-2024-08-06 and gpt-4o-mini onward. Anthropic lists support for its current models, including Claude Opus 4.5 and Sonnet 4.5 and later, through output_config.format. Check the provider’s docs for the exact model list.
Is structured output the same as function calling?
They share the mechanism. Function calling produces arguments for a tool your code runs, then the conversation continues. Structured output constrains the final answer itself. Strict mode on a tool applies the same constrained decoding to its arguments.
Sources
- OpenAI, Introducing Structured Outputs in the API, August 6, 2024
- OpenAI, Structured Outputs guide, as of October 31, 2026
- Anthropic, Structured outputs, as of October 31, 2026