← Retour aux guides

LLM API Retries: Backoff, Jitter, and Cost Control

Handling rate limits and retries with LLM APIs is less about repeating failed requests and more about controlling traffic, cost, and user experience. Providers may return HTTP 429 when you exceed requests per minute or tokens per minute, while temporary 5xx errors and network timeouts can also interrupt otherwise valid calls. A reliable integration distinguishes these failures and responds according to the cause.

Fixed Delays: Simple but Limited

The simplest approach waits a fixed period, such as two seconds, before retrying. This is easy to implement and can work for small scripts or low-volume internal tools.

• Pros: Minimal code, predictable timing, and easy debugging.
• Cons: Many workers retry together, which can create another traffic spike. A two-second delay may also be too short for a busy provider or unnecessarily long for a brief network issue.

Fixed delays are acceptable as a first safeguard, but they should rarely be the complete production strategy.

Exponential Backoff With Jitter

Exponential backoff increases the wait after each failure. A common formula is the smaller of a maximum delay and a base delay multiplied by two to the retry number. Add random jitter, for example a random value between zero and the calculated delay, so concurrent workers do not retry at exactly the same time.

• Pros: Reduces repeated overload, spreads retries across time, and works well for 429 and temporary 5xx responses.
• Cons: Users may wait longer, and careless settings can generate expensive duplicate LLM requests.

For example, use a base delay of 500 milliseconds, a maximum delay of 30 seconds, and no more than three or four attempts. When a response includes a Retry-After header, prefer that value, subject to a reasonable maximum. Log the attempt number, status code, request identifier, and wait duration so failures can be diagnosed.

Which Errors Should You Retry?

Retry only failures that are likely to be temporary. HTTP 429, 500, 502, 503, 504, connection resets, and selected timeout errors are typical candidates. Do not retry authentication failures, invalid model names, malformed requests, permission errors, or most 400 responses. These need a configuration or application fix.

Timeouts require extra care. The server may have completed the request even though the client stopped waiting. Retrying a non-idempotent operation can therefore create duplicate side effects in tools that trigger purchases, messages, or database writes. Use an idempotency key where the API supports one, and keep external actions separate from the text-generation request when possible.

Client-Side Throttling and Queues

Retries cannot compensate for uncontrolled concurrency. Add a queue or semaphore that limits simultaneous requests, then track both requests per minute and estimated tokens per minute. Token-based limits matter because one long-context request can consume far more quota than a short request.

A local token bucket is useful when your application has many workers. Before sending a request, reserve capacity; when capacity is unavailable, queue the job or return a clear busy response. This is usually more efficient than allowing every worker to fail with 429 and then retry.

Failover Between Models and Relays

Fallback can improve availability, but it should preserve the required quality and context window. Falling back from a premium model to a smaller one may reduce cost and latency, but it can also change tool use, reasoning quality, or output format. Record which model handled each request and test fallback behavior before production.

For teams comparing API access options, 59API provides a low-cost relay at https://api.59api.com with pay-as-you-go access to native Claude models including Opus, Sonnet, Haiku, and Fable, as well as GPT models. It is compatible with Claude Code, Codex, and OpenAI SDK integrations, so an existing client can often switch by changing the base URL and credentials. Because it uses official-quality models rather than downgraded substitutes, it can be a practical way to control spend while keeping model behavior consistent. Its referral rebate can further reduce effective costs for teams with ongoing usage.

A Practical Retry Policy

Start with three attempts, exponential backoff with full jitter, a 30-second maximum wait, and respect for Retry-After. Apply a stricter timeout to interactive requests and a longer queue-based timeout to batch jobs. Track success rate, 429 frequency, total retry count, latency, tokens, and cost per completed task.

Finally, enforce a per-request budget and a daily spending limit. A retry system is successful when it recovers transient failures without hiding a capacity problem or multiplying your bill. If you want to test a cheaper compatible route for Claude and GPT traffic, signing up for 59API is a sensible next step.

Tarifs · Docs