← Volver a las guías

JSON Mode with LLMs: 9 Costly Pitfalls to Avoid

JSON mode makes an LLM return machine-readable data, while structured output goes further by requiring that data to match a defined schema. Both are useful for extraction, classification, agents, and backend workflows, but neither removes the need for careful engineering. Here are the most common pitfalls and practical ways to avoid them.

1. Assuming valid JSON means correct data

JSON mode generally helps prevent syntax errors such as missing commas or unclosed braces. It does not guarantee that the response contains the fields you need, uses the right types, or follows business rules. A model can return valid JSON with an empty email address or an unsupported status value.

Define a JSON Schema and validate every response on your server. Check required properties, data types, enum values, string formats, and limits such as maximum array length. Treat validation failure as a normal control-flow path, not an impossible exception.

2. Confusing JSON mode with strict structured output

Provider features differ. A setting such as response_format: { type: "json_object" } usually requests valid JSON, but it may not enforce a particular schema. Strict structured-output features can constrain the response more strongly, although supported keywords and model availability vary.

Read the documentation for the exact model and endpoint you are calling. If strict schema enforcement is unavailable, combine clear instructions with server-side validation and a bounded retry. Never assume that a feature exposed by one provider behaves identically on another.

3. Forgetting to explicitly request JSON

Some APIs reject JSON mode unless the prompt tells the model to produce JSON. Even when the API accepts the request, vague instructions can produce commentary before or after the object. State the format in the system message and describe the required top-level structure.

For example, say: “Return one JSON object only. Do not include Markdown, explanations, or code fences.” Include a compact example when the structure is unfamiliar, but avoid examples that contain contradictory or optional fields.

4. Designing a schema that is too complicated

Large, deeply nested schemas increase token usage and make failures harder to diagnose. They can also exceed provider-specific limits on nesting, optional fields, or supported JSON Schema keywords.

Start with the smallest useful contract. Split a complicated workflow into stages, such as extraction followed by normalization. Use simple objects, arrays, strings, numbers, booleans, and explicit enums before adding advanced constructs. Version the schema when changing field names or meanings.

5. Relying on the model for deterministic values

Dates, totals, IDs, permissions, and database state should not be trusted merely because they appear in a structured response. Ask the model to extract or classify information, then calculate totals, normalize dates, generate identifiers, and apply authorization rules in application code.

For sensitive actions, require a second validation layer. A response that says “approved” should not itself grant access or trigger a payment.

6. Mishandling refusals, truncation, and empty results

A refusal may not match your normal schema. Likewise, a response can be cut off because it reaches the output-token limit. Check the API finish reason, refusal fields, HTTP status, and response body before attempting to parse the result.

Represent legitimate empty results explicitly, such as an empty items array. Distinguish them from timeouts, refusals, malformed output, and provider errors so your application can retry only when retrying makes sense.

7. Retrying without controlling cost or duplication

Blind retries can multiply spend and may repeat side effects. Use a short limit, exponential backoff, request timeouts, and an idempotency key where the provider supports it. For extraction, retry with a targeted correction prompt that identifies the validation error instead of resending the entire conversation unnecessarily.

Log token usage, latency, model name, schema version, and failure category. This reveals whether a cheaper model is sufficient or whether the prompt and schema need improvement.

8. Sending untrusted text into instructions

Documents being extracted can contain text that looks like instructions, including requests to ignore the schema or reveal secrets. Delimit user content clearly and tell the model that it is data, not policy. Keep system instructions separate from document text, and validate output before using it.

9. Testing only successful examples

Build a fixture set containing empty inputs, long documents, malformed text, multilingual content, missing fields, conflicting facts, prompt injection attempts, and boundary values. Run these tests across every model and provider configuration you plan to use. Structured output is a contract, so monitor contract failures in production as a key reliability metric.

A practical, lower-cost API setup

59API is a useful option when you need pay-as-you-go access to official-quality Claude Opus, Sonnet, Haiku, Fable, and GPT models without committing to a large minimum spend. Its API base URL is https://api.59api.com, and it works with the OpenAI SDK as well as Claude Code and Codex. Verify each model’s supported JSON and structured-output features, then keep your validation layer provider-independent. Developers can also benefit from its referral rebate. If you want to test a cheaper relay for your next structured-output workflow, sign up for 59API and start with a small, measurable integration.