← Back to all guides

LLM JSON Mode: Fix Structured Output Failures

Structured output is one of the most useful ways to make an LLM part of a real application. Instead of parsing a paragraph, your code receives predictable fields such as a classification, score, list, or action. However, JSON mode is often misunderstood: it usually guarantees valid JSON syntax, not that the response follows your exact schema.

What is the difference between JSON mode and structured output?

JSON mode asks the model to return a valid JSON object rather than ordinary prose. It can prevent common syntax problems such as trailing commas, unquoted keys, or explanatory text before the object.

Structured output goes further by applying a schema. For example, a schema can require a string field named category, restrict its values to billing or technical, and require a numeric confidence value. If your provider and selected model support schema enforcement, this is generally more reliable than prompting alone.

Why does the model still return invalid or unexpected JSON?

First, confirm that your request actually enables JSON mode. In OpenAI-compatible APIs, this is commonly represented by a response format setting with a JSON object type. The exact parameter can vary by provider, model, and SDK version, so inspect the raw request and response rather than assuming the client applied it.

Second, make the prompt explicit. Tell the model to return one JSON object, name the required keys, define acceptable values, and prohibit commentary outside the object. Some APIs also require the word JSON to appear in the prompt when JSON mode is enabled.

Third, check whether the selected model supports the feature. A compatible endpoint does not mean every model exposes identical capabilities. Structured output may be unavailable, partially supported, or implemented differently across GPT and Claude variants.

How should I write a reliable JSON prompt?

• Describe the output contract before providing the input.
• Specify required fields, data types, allowed enum values, and whether empty values are acceptable.
• State that the response must contain exactly one JSON object with no markdown fences or additional text.
• Include one small example only when it clarifies an ambiguous field.
• Tell the model what to do when information is missing, such as returning null or an empty array.

Do not ask for a JSON object while also requesting a conversational explanation in the same response. Put explanations in a separate field, or use two separate model calls.

What should my application do after receiving the response?

Treat model output as untrusted input. Parse it with your language's JSON parser, then validate the resulting object against a schema. Parsing checks syntax; validation checks meaning. A valid response can still contain a missing field, an impossible number, an unexpected enum value, or a string where your database requires an integer.

Keep the raw response for diagnostics, but avoid logging sensitive user data. When parsing or validation fails, record the model name, request identifier, validation error, and a redacted sample. This makes provider or prompt regressions much easier to identify.

Should I retry invalid JSON?

Yes, but use bounded retries. A practical flow is: parse the response, validate it, then retry once with a short correction message that identifies the exact failure. For example, say that the field confidence must be a number between 0 and 1. Do not blindly resend the same prompt several times, because that increases cost without adding information.

For critical workflows, add a deterministic fallback. You might route failed classifications to a review queue, return a safe default, or use a smaller second model for repair. Never let an unvalidated model response directly trigger payments, permissions, database migrations, or destructive actions.

How can I test structured output before production?

Build a fixture set containing normal requests, missing information, long inputs, non-English text, malformed source data, and adversarial instructions. Assert both JSON parsing and schema validation. Also test refusals, timeouts, truncated responses, and empty output. Measure valid-on-first-attempt rate, repair rate, latency, and token cost rather than judging reliability from a few successful examples.

Is an OpenAI-compatible relay suitable for JSON workflows?

It can be, provided you verify the supported parameters and models. 59API offers pay-as-you-go access through https://api.59api.com, with compatibility for OpenAI SDKs, Codex, and Claude Code. It provides official-quality Claude models including Opus, Sonnet, Haiku, and Fable, as well as GPT models, without a model downgrade. For teams running many validation and retry calls, its low-cost relay pricing can reduce experimentation and production spend. A referral rebate is also available. After confirming your target model's JSON or schema behavior, you can sign up and test the same fixtures against the API base URL before changing your production integration.

Pricing · Docs