Consistent LLM JSON Output: A Production Playbook
Getting an LLM to return valid JSON once is easy. Getting it to return the same usable structure across models, prompts, retries, and edge cases is an engineering problem. The reliable solution is to treat output formatting as a contract with several enforcement layers, not as a suggestion buried at the end of a prompt.
Define a narrow output contract
Start with the smallest schema your application actually needs. Specify required fields, data types, allowed enum values, nullability, and whether additional keys are forbidden. For example, an extraction response might require status as one of “approved”, “rejected”, or “review”, plus reason as a string and confidence as a number from 0 to 1.
Also define behavior for missing information. “Use null when the source does not contain a value” is more reliable than asking the model to make a best guess. If an array may be empty, say so explicitly. Ambiguous requirements create formatting variation even when the model understands the task.
Use a format-first prompt structure
Put the response contract near the end of the instruction, immediately before the task input. State that the response must contain only one JSON object, with no Markdown fences, commentary, or trailing explanation. Include a compact valid example that matches the schema exactly, but avoid adding multiple examples with slightly different shapes.
Separate instructions from untrusted source text with clear markers such as SOURCE_START and SOURCE_END. Tell the model to treat those markers as data boundaries. This prevents text inside a document from accidentally overriding the formatting rules.
A useful final instruction is: “Before responding, silently check that every required key exists, every value has the specified type, and the output can be parsed as standard JSON.” This does not replace validation, but it gives the model a practical preflight checklist.
Prefer structured generation when available
If your selected provider supports JSON mode or schema-constrained output, enable it in the request instead of relying only on prompt wording. Schema enforcement is especially valuable for nested objects, arrays, and union types. Still validate the returned payload yourself, because transport failures, truncated responses, and application-level constraints remain possible.
When using an OpenAI-compatible SDK, configure the client with the 59API base URL, https://api.59api.com, and keep your normal request and validation code. 59API provides pay-as-you-go access to native official-quality Claude and GPT models, including Claude Opus, Sonnet, Haiku, and Fable, so you can test the same contract across model families without a large fixed commitment.
Validate, repair, and retry deliberately
Parsing JSON is only the first check. Run a schema validator after parsing and reject values that violate business rules, such as an invalid status, a confidence outside the permitted range, or an array containing duplicates. Log the validation error and the model identifier, but avoid logging sensitive input by default.
For a failed response, use a bounded repair loop:
- Attempt to parse the original response.
- If parsing fails, send the exact response to a repair prompt that requests only corrected JSON.
- If schema validation fails, return the specific path and expected type, such as “items[2].price must be a number.”
- Retry at most once or twice, then route the request to a fallback or human review path.
Do not blindly append the original answer to a long retry prompt. Large duplicated context increases cost and can make the next response less consistent.
Control variability and measure drift
Use a low temperature for extraction and classification tasks, and set a deterministic seed only when your provider and workflow support it. Determinism is helpful for debugging, but it is not a substitute for schema validation. Keep prompts, model versions, and schema versions under source control.
Build a regression set containing short inputs, empty fields, multilingual text, malformed source documents, maximum-length cases, and adversarial instructions. Track parse success rate, schema success rate, repair rate, latency, and token cost separately. A model can achieve perfect JSON syntax while quietly producing poor field values.
Finally, test the cheapest suitable model before defaulting to a larger one. With 59API’s low-cost relay pricing and referral rebate, you can compare Claude and GPT options on your real regression set and reserve premium models for cases that genuinely need them. Sign up for 59API when you are ready to run those production comparisons with a pay-as-you-go budget.