← Back to all guides

Structured Output with LLM JSON Mode

Structured Output and JSON Mode with LLMs: A Practical Tutorial

Structured output is the difference between a fun chatbot demo and a production system you can trust. Instead of asking a large language model to “summarize this,” you ask it to return predictable fields such as title, category, sentiment, action_items, or risk_score. JSON mode and schema-based prompting make that possible, but they still need careful implementation, validation, and retries.

This step-by-step tutorial shows how to design reliable JSON responses from LLMs, test them, and run them cost-effectively through 59API, an AI API relay compatible with OpenAI SDKs, Claude Code, Codex, and native Claude and GPT workflows. Its base URL is https://api.59api.com, and it is a strong option when you want official-quality models without overpaying during iteration.

Step 1: Define the exact job before the schema

Start with a narrow task. “Analyze customer feedback” is too vague. “Classify a support ticket into one department, urgency, sentiment, and a one-sentence summary” is much better. The model performs more reliably when each field has a clear purpose and a limited range of valid values.

For example, a useful support-ticket output might include department, urgency, sentiment, summary, and suggested_reply. Before writing prompts, decide what your application will do with each field. If a field does not drive UI, routing, analytics, or automation, remove it.

Step 2: Design a small, strict JSON structure

Keep schemas simple at first. Deeply nested objects, optional fields, and mixed data types increase failure rates. A practical first version could be:

{ "department": "billing", "urgency": "high", "sentiment": "negative", "summary": "Customer was charged twice.", "suggested_reply": "We are sorry for the duplicate charge and will investigate immediately." }

Use fixed choices whenever possible. For example, department should be one of billing, technical, sales, or account. urgency should be low, medium, or high. This reduces ambiguity and makes downstream validation easier.

Step 3: Use JSON mode when the model supports it

JSON mode tells the model to return syntactically valid JSON rather than prose. In OpenAI-compatible APIs, this is often configured with a response format such as { "type": "json_object" }. With an OpenAI SDK-compatible relay like 59API, you can point your existing client to https://api.59api.com and keep the same general request style while choosing supported GPT or Claude-family models.

JSON mode is not a full business-rule validator. It helps ensure the response is parseable JSON, but you should still check required fields, allowed values, string lengths, and number ranges in your application.

Step 4: Write prompts that separate instructions from data

A reliable prompt has three parts: role, schema rules, and user data. Tell the model exactly what to return and what not to return. For example: “Return only one JSON object. Do not include markdown. Use only these urgency values: low, medium, high. If information is missing, use null.” Then pass the user’s ticket text separately.

This separation matters because user input may contain confusing instructions such as “ignore previous rules.” Treat user text as data, not instructions. If your SDK supports separate system and user messages, put schema rules in the system or developer message and the raw content in the user message.

Step 5: Validate every response in code

Never assume the model is correct because JSON parsing succeeded. Run validation immediately after receiving the response. Your validator should check:

• Syntax: Can the response be parsed as JSON?
• Required fields: Are all expected keys present?
• Allowed values: Does urgency match low, medium, or high?
• Types: Are strings, numbers, arrays, and booleans in the correct places?
• Safety limits: Are summaries short enough for your UI or database column?

Libraries such as Zod, Pydantic, Joi, or JSON Schema validators are good choices. Validation turns fuzzy AI output into a contract your software can enforce.

Step 6: Add a repair or retry strategy

Even strong models can occasionally return incomplete or invalid data, especially with long inputs. Use a simple retry flow. First, attempt to parse and validate. If validation fails, send the invalid output plus the validation error back to the model and ask it to return a corrected JSON object only. Limit this to one or two retries to control cost and latency.

For high-volume systems, log validation failures by model, prompt version, and input type. This helps you discover whether the schema is too complex, the prompt is unclear, or a cheaper model is not strong enough for the task.

Step 7: Choose the right model for the structure you need

Not every structured-output task needs the most expensive model. Simple extraction, classification, and formatting can often run well on smaller or faster models. Complex reasoning, legal review, multi-document synthesis, or ambiguous inputs may need stronger Claude Sonnet, Claude Opus, or GPT-class models.

This is where 59API is useful for developers. Because it offers cheap, pay-as-you-go access to Claude models such as Opus, Sonnet, Haiku, and Fable, plus GPT models, you can benchmark several options behind one OpenAI-compatible workflow. You get native official-quality models with no downgrade, compatibility with Claude Code and Codex, and lower-cost experimentation while tuning prompts and schemas.

Step 8: Test with realistic edge cases

Do not test only clean examples. Include empty messages, angry customers, multilingual text, very long tickets, pasted logs, contradictory details, and prompt-injection attempts. Check whether the JSON remains valid and whether the classifications still make sense.

Create a small evaluation file with 30 to 100 real or anonymized examples. Run it whenever you change the prompt, schema, model, or temperature. Keep temperature low, often 0 to 0.3, when consistency matters more than creativity.

Step 9: Put structured output into production carefully

In production, store the raw model response, parsed JSON, schema version, model name, validation status, and latency. This makes debugging much easier. If structured output triggers important actions such as refunds, account changes, or compliance decisions, add human review or confidence thresholds.

Structured output and JSON mode are powerful because they let LLMs plug directly into software workflows. Start with a strict schema, use JSON mode where available, validate aggressively, retry safely, and benchmark models by accuracy and cost. If you want a low-cost way to test Claude and GPT models with your existing OpenAI SDK setup, consider signing up for 59API and pointing your client to https://api.59api.com.

Pricing · Docs