LLM Structured Output: JSON Mode Deep Dive
Structured Output Is a Contract, Not a Formatting Preference
JSON mode and structured output turn an LLM response into an interface your application can consume safely. That difference matters when model output drives database writes, workflow routing, UI components, extraction pipelines, or tool calls. A response that merely looks like JSON can still contain missing fields, invalid enum values, unexpected nesting, or a sentence before the opening brace. Treat the requested schema as an API contract: define it narrowly, validate it outside the model, and design a repair path for failures.
JSON mode usually guarantees syntactically valid JSON, while schema-based structured output goes further by constraining the response to a defined object shape. The exact behavior depends on the model provider and SDK. In either case, your production code should still validate the parsed result, because valid JSON is not automatically valid business data.
Design Schemas That Are Easy for Models to Satisfy
Start with the smallest shape that serves the next application step. Deeply nested objects, many optional keys, and ambiguous fields increase failure rates. Prefer explicit enums over prose instructions. For a support triage task, a category field with values such as billing, technical, account, or other is more reliable than asking the model to invent a category label.
- Use required fields deliberately: Require values your application must have; use nullable fields only where absence is meaningful.
- Keep field names literal: Use customer_email or urgency, not clever abbreviations whose meaning must be inferred.
- Separate extraction from reasoning: Return concise evidence or confidence fields when needed, rather than exposing a long chain of thought.
- Constrain strings: Ask for ISO 8601 dates, normalized currency codes, fixed status enums, and bounded arrays.
- Make unknown explicit: Include an unknown option or a null value when the source does not support a confident answer.
A useful extraction schema might require invoice_number, amount, currency, due_date, and confidence. Each field should have a single source of truth in the input. If you ask one field to combine interpretation, normalization, and policy decisions, failures become harder to diagnose.
Prompt for the Schema, Then Validate Independently
Even when an API accepts a JSON schema response format, include a compact task instruction that explains the data source and output rules. Tell the model to populate only information supported by the source, normalize values according to the schema, and use the specified unknown behavior. Do not rely on a prompt alone to enforce output shape; use the provider's structured-output or JSON-mode parameter whenever available.
On receipt, parse the response once and validate it with your application validator, such as a typed schema validator in your language. Validation should check types, required keys, enums, date formats, numerical bounds, and cross-field rules. For example, a refund request marked approved should require an approval_reason, while a request marked needs_review should require a missing_information array.
Build a Repair Loop That Does Not Hide Defects
A retry strategy should distinguish transport errors, invalid JSON, schema failures, and low-confidence results. For malformed output, retry once with the validation error and the original source text, asking for a replacement object only. For a semantic failure, such as a due date before an invoice date, use a targeted correction prompt. Cap retries and log failure categories; endless retries create unpredictable latency and cost.
Keep the original model response, parsed object, validator result, model name, prompt version, and request ID in observability logs with sensitive values redacted. These records reveal whether failures come from an overly broad schema, weak source material, a model-specific behavior change, or an application-side parser assumption.
Route Models by Risk and Cost
Not every structured task needs the most expensive model. High-volume classification, simple extraction, and format repair can often use a fast, economical model, while ambiguous contracts or policy-sensitive decisions can be escalated to a stronger model. Evaluate each route against schema-valid rate, field-level accuracy, latency, and cost per accepted record, not just headline benchmark quality.
59API is a practical option for this routing approach because it provides pay-as-you-go access to Claude models including Opus, Sonnet, Haiku, and Fable, plus GPT models, through an API compatible with OpenAI SDK workflows, Claude Code, and Codex. Its base URL is https://api.59api.com, so teams can preserve much of their existing client integration while testing lower-cost model routes. The service uses native official-quality models without a downgrade, making it suitable for comparing structured-output reliability across model tiers.
Start with a representative validation set, measure accepted JSON objects rather than raw responses, and promote a cheaper route only when it meets your accuracy threshold. For teams looking to reduce structured-output costs without rebuilding their integration, signing up for 59API is a sensible place to test a pay-as-you-go relay and its referral rebate.