Structured outputs vs JSON mode vs tool calling

What are structured outputs, and how do they differ from JSON mode?

If a program is going to read a model’s answer, the answer has to come in the same shape every time. If a field is missing, a number arrives as text or the JSON stops halfway, the program breaks. In the code review prompt post we covered asking for a finding with a verbatim quote, and in the LLM-as-a-judge post recording the judge’s verdict; in both, code reads the model’s answer. This post is about the shape of that answer: the ways to get JSON, what each one guarantees, and what changed on Claude this autumn.

With JSON mode, you ask the model for JSON and the API promises that the answer will be valid JSON. It does not promise which fields will be there. OpenAI’s documentation says so plainly: JSON mode guarantees that the answer parses without errors, not that it matches a schema.

With tool calling (also called function calling), you describe a tool to the model. When the model wants to call that tool, it writes the arguments as JSON. If the tool is not strict, this is best effort, not a guarantee: according to Anthropic’s documentation, with a non-strict tool the model may write "2" instead of 2, or leave out a required field. With a strict tool, the arguments match the schema.

With structured outputs, you give the schema to the answer itself. As the model picks each token, it chooses only among the tokens that fit the schema; this is called constrained decoding. Strict tool use applies the same technique to a tool’s arguments. The result is JSON that is valid and carries the fields in the schema.

So which one is for what? Google’s documentation puts the distinction briefly: structured outputs are for formatting the final answer, function calling is for getting something done in the middle of the conversation. If the answer is going into a database, use structured outputs; if the model wants to read a file or run a command, use tool calling.

How do you turn on structured outputs with the three providers?

All three providers offer the same three ways, but their names, how you turn them on and their limits differ. We read the table from each provider’s own documentation on 20 October 2026:

Source: the documentation of OpenAI, Anthropic and Google, 20 October 2026. “Not stated in the documentation”: in the pages we read that day, we found no sentence that describes this. At Google, response_format is a field of the Interactions API and response_json_schema a field of generateContent. OpenAI’s Responses API turns a function that is not marked strict into strict mode where possible; in Chat Completions, the default is not strict.

At OpenAI, you turn on structured outputs in the Responses API inside text.format, with type: "json_schema" and strict: true. The documentation sets two conditions: every field must be required, and every object must have additionalProperties: false. If you want an optional field, you add null to its type. The schema’s limits are written down too: at most 5,000 fields, 10 levels of nested objects, 1,000 enum values, and 120,000 characters across all names and values together. An unsupported keyword returns an error. The first request with a schema is slower, because the API processes the schema; the requests after it are not.

In tool calling, strict: true makes the arguments match the schema, and OpenAI recommends always turning it on. Forcing the model to call a tool is still an option at OpenAI: if you set tool_choice to required, the model calls at least one function; if you give a function’s name, it calls exactly that one.

At Anthropic, structured outputs come in two halves: output_config.format for the answer (JSON outputs) and strict: true for tools (strict tool use). The two can be used together in the same request. The schema is compiled into a grammar, which is why the first request with a given schema is delayed; the compiled schema is kept for 24 hours after its last use. The limits are per request: at most 20 strict tools, 24 optional fields in total across all schemas, and 16 fields that use anyOf or more than one type; compiling also has a timeout of 180 seconds. An unsupported rule such as minimum or maxLength returns a 400. Fields come in the schema’s order, but required fields are written before optional ones. One more detail: changing output_config.format invalidates the prompt cache for that conversation. We explained why the cache breaks in the prompt caching post.

This autumn’s main change is at Anthropic too. A well-known way to get JSON from Claude was to define a single tool and use tool_choice to force the model to call it. According to the release notes, on Opus 5.5, released on 22 September, and on Sonnet 5.5, released on 28 September, the tool_choice types any and tool return a 400 error. Anthropic’s recommended route is strict tool use with auto, and structured outputs if you need a fixed JSON shape. According to the tool documentation, models that think by default, such as Haiku 5.5, still accept forcing, but on a forced call the model calls the tool directly, without thinking. On Haiku 5.5, released on 7 October, another old method, writing the start of the assistant message yourself (prefill), can return a 400; structured outputs don’t work with prefill anyway.

At Google, the current guide describes the Interactions API: you give response_format a mime_type: "application/json" and a schema. In generateContent, the same is done with response_json_schema. According to the documentation, the model writes syntactically valid JSON that matches the schema and puts the fields in the schema’s order; the API may reject very large or deeply nested schemas. Here there is an important difference: OpenAI and Anthropic reject a schema rule they don’t support with an error, while Gemini ignores it. If your schema has a rule Gemini doesn’t support, the request still succeeds, but the rule is not applied. In function calling, validated mode makes the arguments match the schema; any mode, on the other hand, forces the model to call a function every time.

What don’t structured outputs guarantee?

A schema guarantees the shape of the answer, not its content. Google says so plainly: the values are not guaranteed to be semantically correct. OpenAI also says that structured outputs can still contain mistakes, and adds a warning: if the input has nothing to do with the schema, the model may make things up while trying to fit it. Anthropic, for its part, lists the cases where the schema may not hold at all. What to check:

The two you will meet most often are refusals and cut-off answers. At Anthropic, a request refused for safety reasons does not come back as an error but as a normal answer with stop_reason: "refusal": it returns HTTP 200, the tokens generated are billed, and the answer may not match the schema. At OpenAI, a refusal comes in a separate refusal field. An answer that hits max_tokens is left incomplete with both providers; Anthropic recommends retrying with a higher max_tokens. So your code should first check why the answer ended, and only then parse the JSON.

A trap specific to Claude is a thinking field in the schema. Putting a reasoning field in the schema and asking the model to think there before it answers can lead to a refusal on Claude. Anthropic’s refusals documentation lists a field named reasoning, thinking or trace, in JSON output or in a tool’s input, among the common causes of a reasoning_extraction refusal. Instead, it recommends asking for a short explanation, the evidence behind the result, or a summary of what was done. Since 24 September, a refusal of this kind that arrives before any output has been produced is billed too. The rule is specific to this provider: one of the examples in OpenAI’s guide is a schema that gives the answer with step-by-step explanations. On Claude, how much the model thinks is set by the effort setting, not by the schema; we covered it in the effort post.

How solid the schema is in the provider’s hands is a separate question. In our own records, we sent a strict JSON schema to an endpoint that a routing service’s catalogue listed as supporting response_format, but not as supporting structured outputs. The answer was not an error: HTTP 200 and 1,639 characters of markdown. And the content was right: it named the real bug in the diff we used for the probe. The only thing missing was the JSON. This is the record of a single probe, not a rate. But it shows that an API’s promise of a schema does not replace a check in the code that reads the answer. In diffox, an answer that cannot be parsed as JSON does not count as a success; the call fails, and it goes into the review’s record together with which agent or step dropped out and why.

How do you get JSON from a coding agent on the command line?

The same question comes up with coding agents. If you run an agent in CI or in a script and pass its result on to another step, you need to be able to rely on the shape of the answer. All five tools have a way to run from the command line, but they don’t all give you the same thing: some put the answer inside a JSON envelope, others ask for the answer itself to follow a schema. We read the table from each tool’s own documentation on 20 October 2026:

Source: each tool’s documentation, 20 October 2026. “Not stated in the documentation”: in the pages we read that day, we found no sentence that describes this. Claude Code’s schema validation and retry are described in the Agent SDK documentation; claude -p runs that SDK from the command line. In Cursor CLI, if the run fails, no JSON is written and the process exits with a non-zero code.

The distinction in the table matters. A JSON envelope means the answer comes inside a JSON object; the answer inside it is still free text. If the next step is going to read fields from the answer, you need either a tool with a schema option or a step that validates the answer in code. Claude Code’s documentation is clear on this too: even if the result looks successful, treat it as a failure when the structured_output field is missing. Claude Code notes one more detail: it treats the format keyword in a schema, for example "format": "email", only as an annotation and does not enforce it.

If you still use Gemini CLI, gemini -p --output-format json gives the answer as text in the response field; this documentation does not mention a schema option. For free-tier and Google One users, Google replaced Gemini CLI with Antigravity CLI on 18 June 2026; with paid API keys, Gemini CLI remains available.

How do you get JSON at the end of an agent loop?

In an agent loop, the model first reads, searches and tries things with tools; at the very end it gives a result. To make that result JSON on Claude, it used to be enough to force the model to call a “submit” tool in the last step. Since that route returns a 400 on Opus 5.5 and Sonnet 5.5, the pattern is now:

1. Leave tool_choice at auto. According to Anthropic, the prompt still influences which tool is chosen, and a model that is not forced can think before it calls the tool.
2. Give the model a final tool with strict: true, for example submit, and in the prompt ask it to call that tool exactly once, last, when the work is done. Since the arguments match the schema, you don’t need to parse the result separately.
3. End the loop when that tool call arrives. If the model stops without calling the tool, if the answer is cut off at max_tokens, or if a refusal comes back, don’t count it as a result; raise an explicit error.
4. Put a limit on tool calls. When the limit is reached, don’t run the tool; tell the model to decide now instead. Cut off the turns at an upper limit too.
5. If you want a justification, put a short evidence field in the schema, not reasoning.

You can also use structured outputs for the final answer instead of a tool: Anthropic’s documentation says that strict tools and output_config.format work together in the same request. The tool calls match the schema, and the final text answer comes bound to the schema as well. The example below shows the first route, with Anthropic’s Python SDK:

# Example: tool_choice stays at auto; the loop ends when submit arrives.
import anthropic

client = anthropic.Anthropic()

SUBMIT = {
    "name": "submit",
    "description": "Give the final answer. Call exactly once, last.",
    "strict": True,
    "input_schema": {
        "type": "object",
        "properties": {
            "verdict": {"type": "string", "enum": ["HOLDS", "REFUTED", "UNSURE"]},
            "evidence": {"type": "string"},
        },
        "required": ["verdict", "evidence"],
        "additionalProperties": False,
    },
}


def run(model, messages, tools, run_tool, max_calls=8):
    calls = 0
    for _ in range(max_calls + 3):
        response = client.messages.create(
            model=model, max_tokens=8000, tools=tools + [SUBMIT], messages=messages)
        if response.stop_reason in ("max_tokens", "refusal"):
            raise RuntimeError("no answer: " + response.stop_reason)
        messages.append({"role": "assistant", "content": response.content})
        uses = [b for b in response.content if b.type == "tool_use"]
        for b in uses:
            if b.name == "submit":
                return b.input  # the result, matching the schema
        if not uses:
            raise RuntimeError("the model stopped without calling submit")
        results = []
        for b in uses:
            calls += 1
            out = (run_tool(b.name, b.input) if calls <= max_calls
                   else "Tool budget spent: call submit now.")
            results.append({"type": "tool_result", "tool_use_id": b.id, "content": out})
        messages.append({"role": "user", "content": results})
    raise RuntimeError("turn limit reached without submit")

In diffox, the step that checks a finding against the repository itself, before it is written to the pull request, works this way. The note in the code says it plainly: “A MANUAL LOOP, tool_choice left at auto”. The model can make at most 8 calls with read-only git tools, and the prompt asks it to call the submit_verdict tool exactly once, last, when it is done. submit_verdict is a strict tool: the verdict is one of three values (HOLDS, REFUTED, UNSURE), with an evidence field next to it; there is no thinking field in the schema. When the limit is reached, the tool does not run and the model is told to give its verdict; the turns are cut off at 3 more than the limit. If the model stops without giving a verdict, “no verdict” is written to the record and the finding stays as it is: no finding is counted as refuted by default.

Which one should you use, and when?

The rule is simple: choose by what whoever reads the JSON needs, and still check every answer that comes back. A summary for common cases:

What the cards have in common: the schema guarantees the shape, and the rest is up to your code. Rules a schema cannot express, such as the length of a text, the range of a number or whether one field is consistent with another, are checked in code after the answer arrives. If the check fails, ask once more; if the second try fails too, don’t swallow the error, raise it explicitly. The Claude Agent SDK works the same way: it asks again for output that does not match, and returns an error when the retry limit is reached. We covered checks of this kind, which run after the model’s answer, in the harness engineering post.