← Back to all guides

Temperature and Top_p: A Practical Sampling Guide

Temperature and top_p control how an AI model chooses its next token

Large language models do not select every next word with absolute certainty. For each possible next token, the model assigns probabilities. Sampling parameters decide how broadly the API can draw from those probabilities. The two settings developers encounter most often are temperature and top_p.

These parameters can make the same prompt produce a predictable JSON response, a varied set of product descriptions, or an imaginative story. They are not quality sliders: higher values do not automatically mean better writing, and lower values do not automatically mean more accurate answers. The appropriate value depends on the task and how much variation your application can safely accept.

What temperature changes

Temperature reshapes the model's probability distribution before it samples a token. At a low temperature, likely tokens become even more dominant. At a higher temperature, less likely tokens get a meaningful chance to be selected. A temperature of 0 is generally the most repeatable setting, although exact output can still vary across model versions or backend changes.

• 0 to 0.2: Extraction, classification, code transformations, structured output, and policy-sensitive answers.
• 0.3 to 0.6: Support replies, technical explanations, summaries, and most production chat features.
• 0.7 to 1.0: Brainstorming, marketing variations, fictional dialogue, and alternative phrasing.
• Above 1.0: Experimental creative generation where occasional unusual or inconsistent results are acceptable.

For example, a support application should start at temperature 0.2. That keeps wording stable and reduces the chance that a model invents an unsupported troubleshooting path. A campaign-idea generator might use 0.8 so repeated calls offer genuinely different angles.

What top_p changes

Top_p, also called nucleus sampling, limits choices to the smallest set of tokens whose combined probability reaches a threshold. With top_p set to 0.9, the model samples only from the most probable candidates that together account for 90% of the probability mass. Tokens outside that nucleus are excluded for that sampling step.

A smaller top_p produces more conservative output. A value near 1.0 allows the full distribution. Unlike temperature, top_p filters candidates rather than continuously flattening or sharpening their relative probabilities.

• top_p 0.1 to 0.3: Very constrained outputs where consistency matters more than voice.
• top_p 0.7 to 0.9: A practical range for controlled variety.
• top_p 1.0: No nucleus filter; use temperature as the primary control.

Use one primary sampling control first

Most model providers recommend adjusting either temperature or top_p, rather than aggressively tuning both at once. Changing both makes failures harder to diagnose: when a response becomes repetitive or erratic, you cannot easily tell which control caused it. A straightforward baseline is temperature 0.3 and top_p 1.0 for operational tasks, then change temperature in small increments.

When you specifically need a probability cutoff, keep temperature at 1.0 and adjust top_p instead. For example, use top_p 0.85 to create varied social captions while excluding the long tail of unlikely token choices.

Hands-on example with an OpenAI-compatible client

59API exposes native official-quality Claude and GPT models through an OpenAI-compatible endpoint, so existing OpenAI SDK integrations need only a base URL, API key, and model-name update. Its pay-as-you-go pricing is useful when testing multiple sampling settings, because you can run a realistic evaluation set without committing to a larger platform contract.

Example JavaScript configuration:

import OpenAI from "openai";
const client = new OpenAI({
  apiKey: process.env.FIFTYNINE_API_KEY,
  baseURL: "https://api.59api.com/v1"
});

const response = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Summarize this incident in three factual bullets: ..." }],
  temperature: 0.2,
  top_p: 1
});
console.log(response.choices[0].message.content);

For a creative variant generator, change temperature to 0.8 while leaving top_p at 1. Run the same prompt 10 to 20 times and review outputs for usefulness, factuality, format compliance, and duplication. Store those prompts and expected constraints as a small regression suite before shipping a new setting or model.

Choose settings by measuring the result

Start with the lowest variation that serves the user. Evaluate a fixed prompt set, not one impressive example. For structured responses, validate JSON and count schema failures. For retrieval-based answers, check citations or source grounding. For creative features, ask reviewers to rate novelty alongside relevance. Also set a token limit and clear system instructions; sampling cannot compensate for vague requirements.

Once you have a baseline, 59API lets you compare compatible Claude, GPT, and other supported models through one low-cost relay while retaining the SDK workflows used by Claude Code, Codex, and OpenAI clients. Sign up for 59API when you are ready to test these settings against real application prompts and keep inference spend under control.

Pricing · Docs