← Volver a las guías

Production-Safe LLM Retries, Timeouts, and Backoff

Why LLM calls need resilience engineering

An LLM request is not a normal database lookup. It can wait in a provider queue, generate a long answer token by token, hit a rate limit, or fail after your application has already sent the request. Production applications must distinguish a temporary failure from a permanent one, limit how long users wait, and avoid making an outage worse with a retry storm.

The reliable pattern is simple: set a request deadline, retry only transient failures, and use exponential backoff with jitter. This approach works whether you call a Claude model, a GPT model, or an OpenAI-compatible relay. 59API is a practical low-cost option for teams that need pay-as-you-go access to native official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, plus GPT models. Its OpenAI SDK compatibility also means you can centralize this reliability logic instead of maintaining separate client implementations.

Step 1: Set layered timeouts, not one arbitrary limit

Use a timeout at every layer, with the outer deadline always longer than the inner one. For example, a browser or mobile client may allow 45 seconds, your application endpoint may allow 40 seconds, and the LLM client itself may allow 30 seconds. This leaves time for your server to return a controlled response rather than being terminated by the caller.

• Connection timeout: 3 to 5 seconds is usually enough to establish a connection.
• Request timeout: Start around 30 seconds for short non-streaming tasks and increase it for large prompts or long outputs.
• Total operation deadline: Cap the entire retry sequence, such as 45 seconds. A retry must never extend beyond this deadline.
• Streaming idle timeout: For streaming responses, reset an idle timer whenever a token arrives. Stop the request if no data arrives for a defined period.

Do not use an unlimited timeout. A hung request consumes application workers, holds connections open, and can cascade into an incident. Tune timeouts with real measurements: prompt size, expected output tokens, model choice, and observed p95 latency should drive your values.

Step 2: Retry only errors that can succeed later

A retry is appropriate when the failure is likely temporary. Common candidates are network connection resets, DNS failures, 408 request timeout, 429 rate limiting, and most 5xx responses. Treat 500, 502, 503, and 504 as retryable unless provider documentation says otherwise. Respect a Retry-After header whenever one is present.

Do not retry malformed requests, authentication failures, invalid model names, insufficient account balance, content-policy rejections, or other ordinary 4xx validation errors. Retrying these errors only increases cost and delays useful feedback. Log the status code, provider request ID when available, selected model, attempt number, and elapsed time. Never log API keys or unredacted sensitive prompts.

Step 3: Apply capped exponential backoff with full jitter

Exponential backoff gives a busy service time to recover. A useful schedule starts with a small delay and doubles after each failed attempt: 250 milliseconds, 500 milliseconds, 1 second, 2 seconds, and so on. Cap the delay, commonly at 8 or 10 seconds, and cap attempts at two or three retries for interactive user requests.

Always add jitter. Without it, many workers that fail together retry at the same moments, producing another traffic spike. With full jitter, calculate a maximum delay for the attempt, then choose a random delay between zero and that maximum. For attempt number n, the maximum can be the smaller of 10 seconds or 250 milliseconds multiplied by 2 to the power of n. Before sleeping, check the total deadline; skip the retry if the remaining time is too short.

Step 4: Prevent duplicate work and surprise costs

A network timeout does not prove that the model did not receive your request. The request may have completed while the response was lost. For operations that create records, send emails, charge customers, or trigger tools, use an idempotency key generated by your application. Store the key with the operation result so a repeated request returns the previous outcome instead of running the action twice.

For ordinary text generation, retries can still create extra token usage. Keep prompts deterministic where possible, limit maximum output tokens, and avoid retrying an expensive long-generation task several times. If a request is noncritical, queue it for asynchronous processing rather than forcing it through an interactive deadline.

Step 5: Add rate limiting, circuit breaking, and observability

Client-side concurrency limits are as important as retries. Set a maximum number of in-flight requests per model or tenant, and queue excess work. When 429 responses rise, lower concurrency temporarily instead of repeatedly retrying. A circuit breaker is also valuable: if a provider or route fails repeatedly, briefly stop sending new requests and return a fallback message or use a secondary model path.

Track success rate, timeout rate, retry count, p50 and p95 latency, 429 rate, 5xx rate, tokens per successful response, and cost per completed task. Alert on changes in these metrics, not on a single failed call. When using 59API, point your compatible OpenAI SDK configuration to https://api.59api.com and keep the same retry wrapper around every model call. This gives your application a consistent production pattern while retaining flexible access to cost-effective Claude and GPT options. If you are building a resilient AI feature on a budget, consider signing up for 59API and testing these settings against your own real workloads.

Precios · Docs