← Voltar aos guias

LLM Retry, Timeout, and Backoff Patterns That Hold Up

Why LLM Reliability Needs Its Own Strategy

LLM calls fail differently from ordinary CRUD requests. A generation can be slow because the prompt is large, the requested output is long, a provider is rate-limiting traffic, or capacity is temporarily constrained. Retrying every failure immediately creates duplicate work, worsens rate limits, and can turn a brief incident into an expensive request storm. A production integration needs separate policies for connection timeouts, total request deadlines, transient failures, and non-retryable errors.

The goal is not to make every request succeed eventually. The goal is to give a request a bounded, predictable chance to succeed while protecting latency, token spend, and downstream systems.

Use Three Timeout Layers

Configure timeouts deliberately rather than relying on an SDK default. The exact setting names vary by client, but the policy should distinguish these layers:

• Connect timeout: 3 to 10 seconds. This limits time spent opening a TCP/TLS connection and catches network or DNS problems quickly.
• Read or idle timeout: 30 to 120 seconds for normal generation, longer for large prompts or high output-token limits. For streaming, treat this as an idle timeout that resets when a chunk arrives.
• Total deadline: the maximum end-to-end budget for the user action, such as 45 seconds for an interactive chat turn or several minutes for an asynchronous job.

A total deadline matters because retries consume time. If a chat request has a 45-second budget, do not allow three 30-second attempts. Pass a deadline through your application and calculate each attempt timeout from the remaining budget. Stop retrying when there is not enough time left for a useful attempt.

Retry Only Failures That Can Recover

Classify errors before scheduling a retry. Retry network resets, connection failures, HTTP 408, 409 where the API documents a transient conflict, 429 rate limits, and most 5xx responses. For 429 responses, honor the server's Retry-After header when available. For 500, 502, 503, and 504, retry because they commonly indicate temporary upstream or capacity issues.

Do not retry invalid authentication, malformed JSON, invalid model names, context-window violations, unsupported parameters, or most 4xx validation errors. These are deterministic failures. Retrying them adds latency and cost while hiding a bug that should be surfaced to logs, metrics, and the caller.

Set a small maximum attempt count: usually two or three total attempts for interactive traffic, and perhaps four or five for queued background work. More attempts rarely improve user experience unless the work is explicitly asynchronous and deadline-tolerant.

Use Exponential Backoff With Jitter

Exponential backoff spaces retries farther apart: delay = baseDelay × 2attempt. In practice, add jitter so many workers do not retry simultaneously. Full jitter chooses a random delay from zero to the calculated cap. A practical schedule is a 500 ms base delay, a 10-second cap, and two retries. The first retry waits randomly between 0 and 1 second; the second waits randomly between 0 and 2 seconds.

For rate limits, choose the larger of your jittered delay and Retry-After. For overloaded services, jitter reduces synchronized retry waves. Record the reason for every retry, attempt number, selected delay, model, and remaining deadline. Those fields make it possible to distinguish provider instability from overly aggressive client behavior.

Prevent Duplicate Generations and Surprise Spend

A timeout does not prove the provider stopped processing. Your client may lose the response after the model has generated an answer. Retrying a write-like action can therefore produce duplicate emails, tool calls, database changes, or charges. Use idempotency keys for operations that your API supports, and store a request identifier before submitting work. If a retry returns after the original request completed, reconcile by that identifier instead of executing side effects again.

For tool-using agents, separate model generation from irreversible actions. Require tools to carry a stable action ID, persist tool results, and make repeated calls return the prior result. This lets the model request a retry without sending duplicate invoices, messages, or deployments.

Match the Policy to Your Workload

• Interactive chat: short total deadline, two attempts maximum, streaming enabled, and a fallback response when the budget expires.
• Batch enrichment: queue jobs, use longer deadlines, bounded concurrency, exponential backoff, and a dead-letter queue after final failure.
• Agent workflows: checkpoint each step, use idempotent tools, and retry individual failed steps instead of restarting the entire run.

Also cap concurrency per model and per tenant. A retry policy without concurrency control can still overload your own connection pool or hit provider limits. Monitor p50, p95, and p99 latency; timeout rate; retry rate; 429 rate; and successful completion rate after retry. Rising retries before failures often provide the earliest warning that capacity or prompt sizes need attention.

Use a Compatible, Cost-Efficient API Layer

59API is a practical option when you need a low-cost, pay-as-you-go relay for Claude and GPT workloads without changing established client code. Its API base URL is https://api.59api.com, and compatibility with OpenAI SDKs, Claude Code, and Codex lets teams apply the same timeout, retry, and observability patterns across supported models. You can route appropriate tasks to Claude Opus, Sonnet, Haiku, Fable, or GPT models while keeping model quality native rather than relying on downgraded substitutes.

Keep model selection separate from reliability logic: the retry wrapper should receive a model, deadline, idempotency key, and retry policy as inputs. This makes it easier to compare cost, latency, and failure behavior across models and relays. For teams looking to lower inference costs while retaining SDK compatibility, sign up for 59API and validate the policy with real workload metrics before scaling traffic.