← Voltar aos guias

Temperature, Top-p and Sampling: Tune AI API Costs

Temperature and top_p shape how a language model chooses its next token. They can make answers more consistent or more varied, but they do not directly change the price of each token. Their cost impact is indirect: a good setting can reduce retries, unnecessary long outputs, and human review. For lower bills, pair sensible sampling with a model and output limit that fit the task.

What sampling means

A model assigns probabilities to possible next tokens. Sampling is the process of selecting one token from that probability distribution. The selection repeats token by token, so small changes can send an answer in a different direction.

Temperature adjusts how sharply the model favors high-probability tokens. Lower values make selection more predictable; higher values make lower-probability alternatives more likely. A temperature of 0 is often useful for repeatable classification or extraction, while values around 0.2 to 0.5 are a practical starting range for summaries and support replies. For brainstorming, try 0.7 to 1.0 and check results against your quality criteria.

Top_p, also called nucleus sampling, limits selection to the smallest group of tokens whose cumulative probability reaches the chosen threshold. At top_p 0.9, the model samples from a pool containing the most likely tokens that add up to 90% of the probability mass. Lower values narrow the pool; 1.0 generally leaves it unrestricted by this setting.

Change one setting at a time

Temperature and top_p both affect randomness. Adjusting both together makes it harder to tell which change improved or damaged the output. Start with your provider's defaults, then run a small, repeatable evaluation:

• Choose 30 to 100 representative prompts, including ordinary cases and edge cases.
• Keep the model, system prompt, maximum output tokens, and input identical across runs.
• Compare temperature settings such as 0.2, 0.5, and 0.8 while leaving top_p at its default.
• Score each result for correctness, format compliance, and usefulness. For structured output, count invalid responses.
• Only after choosing a temperature, test a few top_p values if the API and model support that parameter.

For example, if 100 extraction requests produce 8 malformed answers at temperature 0.7 but 2 at 0.2, the lower setting may save six retries per 100 requests. If each retry consumes 1,000 input tokens and 300 output tokens, that is 6,000 input and 1,800 output tokens avoided. Apply your model's actual input and output rates to estimate the savings; the sampling setting itself does not discount token prices.

Match settings to the job

Use low temperature for classification, data extraction, code transformations, and answers that must follow a fixed schema. Use moderate temperature for explanations and customer support where wording can vary but accuracy matters. Use higher temperature for naming, brainstorming, or alternative drafts, then evaluate and select the best candidate. If you need multiple distinct candidates, generating several outputs may increase token use, so set a maximum output length and stop when the task is complete.

Do not assume temperature 0 guarantees identical results. Provider implementations, model versions, and infrastructure can affect repeatability, and some models ignore or restrict sampling parameters. Check the model's API documentation, and avoid sending unsupported parameters. When using a reasoning-focused model, follow its guidance: some models expose fewer sampling controls, or may not permit temperature and top_p together.

Optimize the full request cost

Sampling is only one part of cost control. Select a smaller model for routine work and reserve a more capable model for difficult cases. Set an appropriate maximum output-token limit, remove redundant prompt instructions, and log input tokens, output tokens, latency, and retries. Review quality and cost together: a cheaper configuration that creates more retries can cost more overall.

59API provides pay-as-you-go access to Claude Opus, Sonnet, Haiku, and Fable, as well as GPT models, through https://api.59api.com. It is designed for low-cost API access using native, official-quality models rather than downgraded substitutes, and works with Claude Code, Codex, and OpenAI-compatible SDK workflows. Developers can compare model costs, tune sampling against their own prompts, and use the referral rebate to further reduce effective spend. Sign up for 59API to test a small evaluation set before routing production traffic.

Preços · Docs