← Back to all guides

How to Keep LLMs in a Strict Output Format

Why LLMs Drift From the Format You Want

If you have ever asked an LLM for JSON, a bullet list, or a fixed template and received extra commentary, broken brackets, or reordered fields, you are not alone. Large language models are optimized to predict the next token, not to act like a strict parser. That means small changes in temperature, prompt wording, model family, or even a long conversation history can cause output drift.

The good news is that consistent formatting is usually achievable if you combine prompt design, model settings, validation, and retry logic. If you are testing across Claude and GPT models, a relay like 59API can help you do that affordably. It offers pay-as-you-go access to native, official-quality models through https://api.59api.com, works with the OpenAI SDK, and is compatible with tools like Claude Code and Codex, so you can compare behavior without paying premium prices.

Step 1: Be Extremely Specific About the Contract

Do not ask for “JSON” in general. Define the exact structure you want and state what must never appear outside it.

• Good: “Return valid JSON only. Use exactly these keys: status, summary, tags. status must be one of: open, closed, pending.”
• Better: include an example of the final shape, with real key names and sample values.
• Best: explain what to do when data is missing, such as using null, an empty array, or a sentinel value.

Models often fail because the instruction is vague. If you want consistent output, make the contract unambiguous and short enough to survive long prompts.

Step 2: Lower Creativity When Format Matters

Higher temperature and top-p settings increase variation. That may be great for brainstorming, but it is bad for rigid formats. For structured outputs, start with temperature at 0 or close to it. If the model still wanders, reduce the prompt length and remove unrelated context.

Also remember that some models handle strict formatting better than others. With 59API, you can test multiple Claude and GPT options at low cost and find the model that balances format accuracy and response quality for your use case.

Step 3: Use a Schema, Not Just an Instruction

If your application depends on machine-readable output, validate against a schema after every response. A schema gives you a real contract to check, instead of hoping the model behaved.

• For JSON: parse the response and reject anything that is not valid JSON.
• For key/value output: verify each expected field exists and has the right type.
• For tables: count columns and ensure every row matches the same structure.

If your SDK supports structured output or function-calling style features, use them. If not, implement a validation layer in your app and retry when parsing fails.

Step 4: Add a Retry Loop With Targeted Repairs

Even a strong prompt will occasionally fail. Do not treat that as a dead end. Build a retry path that feeds the model the exact parsing error and asks for a corrected version only.

A practical pattern is:

• Send the original instruction.
• Parse the response.
• If parsing fails, return the error message to the model.
• Ask it to fix only the malformed output, without adding extra text.

This works especially well when your first pass is close but has one missing comma, an extra explanation line, or a wrong field name.

Step 5: Trim Context and Control Tool Noise

Long conversations can make output less predictable because the model tries to satisfy too many constraints. If formatting is critical, keep the prompt focused and avoid unnecessary examples, chain-of-thought requests, or unrelated chat history. Also, make sure tool results do not get mixed into the final response unless you want them there.

When you are integrating with the OpenAI SDK or a Claude-compatible client, test the exact request payload your app sends. A tiny change in system instructions can be the difference between clean JSON and a paragraph with JSON inside it.

FAQ: Common Troubleshooting Questions

Why does the model add extra text before or after JSON? Because the instruction did not fully block it, or the model was set to a creative sampling level. Say “output JSON only” and validate the result.

Why does the schema work in one model but fail in another? Different models vary in strictness and instruction following. Test several options and keep the one that is most stable for your task.

Should I rely on prompt wording alone? No. Prompts help, but validation and retries are what make output reliable in production.

How can I test cheaply? Use a low-cost relay such as 59API. Because it is pay-as-you-go and uses native official-quality Claude and GPT models, you can run many format tests without overspending.

Is 59API only for one framework? No. It is designed to be compatible with Claude Code, Codex, and any OpenAI SDK, with the base URL at https://api.59api.com.

Practical Rule of Thumb

If your use case needs human-readable text, flexibility is fine. If it needs machine-readable output, assume the model will eventually break format unless you enforce it. The winning stack is: precise prompt, low temperature, schema validation, retry-on-failure, and affordable model testing.

If you want to experiment with this setup without high API costs, sign up for 59API and use it as your testbed for structured-output workflows. The cheaper your trials, the faster you can harden your prompt and ship a reliable integration.

Pricing · Docs