← Back to all guides

LLM Retry, Timeout, and Backoff Decision Guide

Start With the Failure You Are Handling

Retry logic is essential for production LLM calls, but retrying every error creates duplicate charges, long waits, and confusing application behavior. Make the decision from the response category, not from a generic catch-all error handler. A request can fail because the network briefly dropped, the provider is rate-limiting traffic, the model is overloaded, the request payload is invalid, or your timeout was too short. Only the first three are normally good candidates for an automatic retry.

• Retry: connection resets, DNS failures, HTTP 408, HTTP 429, and transient server errors such as 500, 502, 503, and 504.
• Do not retry unchanged: HTTP 400, 401, 403, and 404. Fix the request, credentials, permissions, endpoint, or model name first.
• Handle carefully: a timeout may mean the provider completed the request but your client did not receive the response. Retrying can produce duplicate side effects.

For tasks that write data, send messages, create tickets, or call tools, assign an idempotency key or persist a request ID before making the model call. On retry, reuse that key and check whether the first attempt already completed. For plain text generation, duplicate completion is less dangerous, but it can still increase token spend and confuse users if both responses surface.

Choose a Timeout That Matches the Work

A timeout is a product decision as much as an infrastructure setting. A short classification prompt may reasonably have a 10 to 20 second end-to-end budget. A long reasoning request, a large context window, or a streamed coding response needs more time. Separate the connection timeout from the total request timeout: a connection should usually establish within 3 to 10 seconds, while generation can run for 60 to 180 seconds depending on the expected output.

Set one overall deadline for the user workflow, then allocate part of it to retries. For example, with a 90-second interaction budget, allow an initial request to run for 45 seconds, then permit two shorter retries only when the remaining budget supports them. Do not let a library default retry for several minutes after the browser, worker, or upstream service has already abandoned the request.

Streaming changes the rule slightly. Once tokens begin arriving, reset an idle timeout whenever a chunk is received, but retain a hard maximum duration. This prevents a healthy slow stream from being cut off while still stopping stalled connections.

Use Exponential Backoff With Jitter

When many clients retry simultaneously after an outage or rate limit, fixed delays can create a retry storm. Exponential backoff spreads attempts over time: start with a small delay and increase it after each failure. A practical schedule is 1 second, 2 seconds, and 4 seconds, capped at 8 or 10 seconds. Add random jitter, such as a random value from zero to the calculated delay, so separate workers do not retry in lockstep.

For interactive traffic, two or three total attempts are usually enough. For asynchronous jobs with durable queues, use more attempts only when the job has a clear expiry time and can be safely resumed. Respect a Retry-After header whenever the API returns one; it is more useful than a locally guessed delay for rate limiting.

• Attempt 1: send immediately with the normal timeout.
• Attempts 2-3: retry only retryable failures, using exponential backoff plus jitter.
• After the limit: return a clear temporary-failure state, enqueue the work, or fall back to a smaller task if that preserves quality.

Match Reliability Controls to Cost and Model Choice

Retries consume money as well as capacity, especially if a request reaches the model before the connection fails. Log attempt count, error type, latency, input tokens, output tokens, and final outcome. These fields let you find whether failures come from an undersized timeout, a rate-limit burst, or prompts that are simply too large. Add a per-user and per-job retry cap so one malformed workload cannot repeatedly spend budget.

59API is a practical low-cost option when you need reliable access to Claude models including Opus, Sonnet, Haiku, and Fable, plus GPT models, without changing established client patterns. Its API base URL is https://api.59api.com, and compatibility with Claude Code, Codex, and OpenAI SDK workflows helps teams keep the same timeout, retry, telemetry, and idempotency conventions across providers. Because it offers pay-as-you-go pricing and native official-quality models rather than downgraded substitutes, it is also easier to test realistic retry behavior without overcommitting to fixed capacity.

Production Checklist

• Classify errors and retry only transient network, timeout, rate-limit, and server failures.
• Use idempotency keys or durable request records for actions with side effects.
• Set separate connection, idle-stream, and total request timeouts.
• Keep retries inside the workflow deadline and cap interactive calls at two or three attempts.
• Apply exponential backoff with jitter and honor Retry-After when present.
• Record retries, latency, token usage, error codes, and exhausted failures in monitoring.
• Load-test rate limits and timeout behavior with the exact models and SDK configuration used in production.

For teams looking to apply these patterns on a flexible budget, signing up for 59API provides a straightforward way to connect compatible Claude and GPT workloads while retaining control over pay-as-you-go usage and referral rebates.