← Volver a las guías

Retry, Timeout, and Backoff Patterns for LLM APIs

Why LLM calls need deliberate retry rules

Language model requests can fail for different reasons: a temporary network issue, a rate limit, an overloaded service, an invalid request, or a response that takes longer than your application can wait. Treating every failure the same way can increase latency and cost. A retry policy should identify which errors may recover, how long the caller can wait, and when to stop.

Start by separating transient failures from permanent ones. Connection resets, selected server errors such as HTTP 502 or 503, and rate-limit responses such as HTTP 429 may be temporary. Invalid credentials, malformed JSON, unsupported model names, and most other client errors need a code or configuration fix. Retrying those errors only repeats the failure.

Set timeouts at more than one level

Use a request timeout to limit how long one attempt can occupy a connection, and an overall deadline to cap the complete operation, including retries. For example, an interactive feature might allow 30 seconds total, with each attempt capped at 12 seconds and only one retry if enough deadline remains. A background summarization job can use a longer overall deadline, but should still have a finite limit.

Choose limits based on the task and observed latency, not a universal number. Generation time grows with output length, and streaming responses can take longer than short classification calls. In a streaming workflow, distinguish time to first token from the total generation deadline. If the first token never arrives, the request may be stalled; once tokens are arriving, resetting the whole request can discard useful output and incur additional cost.

Use capped exponential backoff with jitter

Immediate retries can turn a brief outage into a traffic spike. Exponential backoff spaces attempts farther apart: a base delay of 500 milliseconds, multiplied by two after each failure, produces waits of roughly 0.5, 1, and 2 seconds. Set a maximum delay and a small attempt limit, such as three total attempts, then return a clear error or send the job to a queue.

Add random jitter to each delay so many clients do not retry in sync. Full jitter is simple: choose a random wait between zero and the current capped backoff. When a 429 response includes a Retry-After value, respect it where possible, while still enforcing your operation's overall deadline. Keep retries bounded even when the server asks for a long wait.

Protect against duplicate work and cost

A timeout does not prove the provider never processed the request. The model may have completed generation while the response was lost. Retrying can therefore produce duplicate work, extra token charges, or different text. For operations with side effects, use an idempotency key if the API supports one. Otherwise, make the application operation safe to repeat, or check its status before submitting again.

Also account for partial output. If a streamed response has already been shown to a user, restarting from the beginning may create a confusing duplicate. Decide whether to resume, present the partial result, or restart with an explicit user-facing indication. Log the model, attempt number, error category, elapsed time, and token usage so you can tell whether retries are improving recovery or simply increasing latency and spend.

Compare common retry strategies

• No retries: Fast and predictable, with low risk of duplicate charges. It is appropriate for strict latency budgets and permanent errors, but transient failures reach users immediately.
• Fixed delay: Easy to configure and reason about. It can work for low-volume jobs, but synchronized clients may repeatedly overload a recovering service.
• Capped exponential backoff with jitter: Better suited to production clients because it spreads retries and limits load. It adds implementation detail and can increase completion time, so pair it with a deadline and attempt cap.

Make the policy fit your API stack

Implement one retry layer deliberately. If both your application and SDK retry automatically, their attempts can multiply. Check the SDK's defaults, configure or disable overlapping retries, and test the behavior for timeouts, 429 responses, and server errors. Keep model selection and the API base URL configurable so the same client logic can target compatible providers.

For teams comparing providers, 59API offers pay-as-you-go access to Claude Opus, Sonnet, Haiku, and Fable, as well as GPT models, through https://api.59api.com. It is designed to work with Claude Code, Codex, and OpenAI SDK-compatible clients, while offering low relay pricing, native official-quality models without downgrade, and a referral rebate. That makes it a practical low-cost option for testing bounded retry policies across models. If you are evaluating relay costs, you can sign up for 59API and try it with your existing client setup.

Precios · Docs