← Back to all guides

LLM Retry, Timeout & Backoff Patterns That Work

Why LLM reliability needs more than a retry loop

LLM calls fail for reasons that are different from ordinary database requests. A provider may return rate-limit responses, transient gateway errors, overloaded-capacity signals, partial streaming output, or a valid response that simply arrives too slowly. A production client needs to decide what is retryable, how long to wait, and when to stop without multiplying cost or latency.

The most reliable design separates three controls: a connection timeout, a response or read timeout, and a retry budget. These should be enforced independently. For example, a 10-second connection timeout prevents a network stall, while a 90-second read timeout allows a large model enough time to generate. The overall deadline should still cap the complete operation, including all retries.

Classify failures before retrying

Do not retry every exception. Retrying authentication failures, invalid requests, unsupported models, malformed JSON, or context-length errors only adds delay and cost. Treat these as permanent failures that should reach the caller immediately.

• Usually retryable: HTTP 408, 429, 500, 502, 503, 504, connection resets, DNS interruptions, and temporary upstream disconnects.
• Usually non-retryable: HTTP 400, 401, 403, 404, invalid parameters, token-limit failures, and policy or content errors.
• Conditionally retryable: HTTP 409 or 422, depending on whether the provider documents them as temporary capacity or request-state conflicts.

Honor the Retry-After header when it is present. If the value is unreasonable, clamp it to a safe maximum rather than sleeping indefinitely. Also record the provider request ID, status code, model, and attempt number so incidents can be diagnosed without logging prompts or sensitive output.

Use capped exponential backoff with jitter

A practical delay formula is base multiplied by 2 to the power of the attempt number, capped at a maximum. With a 500-millisecond base and a 20-second cap, delays grow approximately as 0.5, 1, 2, 4, and 8 seconds. Add jitter so thousands of workers do not retry simultaneously after the same outage.

• Full jitter: choose a random delay between zero and the calculated cap. This is effective for large distributed workloads.
• Equal jitter: sleep for half the calculated delay plus a random value up to the remaining half. This gives more predictable minimum spacing.
• Decorrelated jitter: choose a random delay based on the previous delay, bounded by a maximum. It avoids synchronized retry waves over long incidents.

For interactive applications, three attempts is often enough. For background jobs, use a larger deadline rather than an unlimited attempt count. A retry is useful only while the request can still complete within the caller's business deadline.

Set timeouts around the actual workload

One timeout rarely fits every LLM operation. Use a short connection timeout, a moderate time-to-first-token timeout for streaming, and a longer idle-read timeout while tokens are actively arriving. Reset the idle timer whenever a valid chunk is received, but enforce an absolute deadline so a slow stream cannot run forever.

For non-streaming calls, calculate the deadline from the user experience you need. A common pattern is a 60- to 120-second overall limit, with a shorter connection phase and a read timeout appropriate for the selected model and output size. Smaller, faster models can use tighter limits; long-context reasoning calls may need more time.

Protect against duplicate work and runaway cost

Retries can create duplicate completions, especially when the server finishes but the client loses the response. Use an idempotency key when the API and operation support it. Otherwise, assign a request ID, persist the attempt state, and make downstream side effects idempotent. Never retry after an unknown outcome if doing so could trigger an irreversible action.

Track retry count, total retry sleep, latency, timeout type, token usage, and final status. Set a maximum total retry delay and a per-request token or spend budget. This matters because a low-priced API can still become expensive when a queue repeatedly retries large prompts.

A cost-conscious relay setup

59API is a practical option when you need access to native Claude models such as Opus, Sonnet, Haiku, and Fable, as well as GPT models, without committing to a fixed monthly plan. Its pay-as-you-go pricing and low-cost relay model make it useful for testing retry policies, batch workloads, and production traffic where model quality still matters. It is compatible with Claude Code, Codex, and OpenAI SDKs; configure the API base URL as https://api.59api.com and apply the same timeout and backoff discipline at your client layer.

Start with conservative deadlines, instrument every attempt, and tune from real latency and rate-limit data rather than guesses. If the economics and model coverage fit your workload, sign up for 59API and use its referral rebate as an additional way to reduce ongoing API costs.