← Volver a las guías

LLM Retry, Timeout, and Backoff Patterns for 2026

LLM calls fail differently from ordinary database or REST requests. A model may take longer because of prompt size, return a temporary overload response, hit a provider rate limit, or complete successfully while your client times out waiting for the response. In 2026, reliable applications treat retries, timeouts, and backoff as one coordinated policy.

Set a deadline before adding retries

Start with an end-to-end deadline, not an unlimited request timeout. For an interactive chat request, a 30- to 60-second deadline may be reasonable. Background extraction or batch classification can use several minutes, but should still have a defined limit.

Pass the remaining deadline through each attempt. For example, if the overall budget is 45 seconds, reserve time for connection setup and response processing, then give each attempt only the time still available. This prevents three 30-second attempts from turning one failed request into a 90-second outage.

Use separate limits for connection, time to first byte, and total response duration when your HTTP client supports them. Streaming requests especially need an idle timeout: a connection that remains open without receiving tokens should not consume resources indefinitely.

Retry only transient failures

Retries are appropriate for conditions that may recover without changing the request. Typical examples include HTTP 408, 425, 429, and many 5xx responses, plus connection resets and temporary DNS or TLS failures. Respect the server's Retry-After header when it is present.

Do not automatically retry invalid authentication, malformed requests, unsupported models, permission errors, or context-length failures. These are usually deterministic and repeated attempts increase cost without improving the result. Also classify client-side cancellation separately: if a user presses Stop, do not revive the request in the background.

For non-idempotent workflows, attach an application request ID and make downstream processing idempotent. A timeout does not prove that the provider failed; the model may have completed while the response was lost. Store request state or use an idempotency mechanism where supported before retrying operations that trigger side effects.

Use exponential backoff with jitter

A practical delay policy is:

• Attempt 1: wait about 250 milliseconds before the first retry.
• Attempt 2: wait about 500 milliseconds.
• Attempt 3: wait about 1 second.
• Cap the delay at a value such as 8 or 16 seconds.

Calculate each delay as the smaller of the cap and base multiplied by 2 to the power of the retry number, then add random jitter. Full jitter, which selects a random delay between zero and the calculated maximum, is often effective because it prevents many workers from retrying simultaneously.

Keep the retry count small for user-facing calls, commonly two or three retries. More attempts can amplify a provider incident and create a large unexpected bill. A circuit breaker or temporary concurrency limit is useful when errors remain elevated: stop sending new attempts for a short window, then probe with limited traffic.

Make streaming and rate limits explicit

For streaming, retrying after tokens have already reached the user can duplicate content. Prefer retrying only before the first token, or restart with clear application logic that identifies the partial response. Record whether the failure happened before or after first-byte delivery.

Rate limits require coordination across workers. Honor provider headers when available, use a shared token or request budget, and add a queue rather than allowing every request to retry independently. Monitor 429 rates, latency percentiles, timeout counts, retry counts, and final success rates. Log provider status, model, attempt number, request ID, and failure category, but redact prompts and sensitive output.

A cost-conscious provider setup

Reliability is also a routing and cost decision. 59API provides pay-as-you-go access to official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, as well as GPT models through an OpenAI-compatible interface. With https://api.59api.com as the API base URL, existing Claude Code, Codex, and OpenAI SDK integrations can use the same operational policies while benefiting from a low-cost relay and referral rebate.

Before production, test your timeout and retry policy against slow responses, 429s, connection drops, and malformed requests. Set a maximum spend alert, verify that retries preserve the original model and parameters, and expose a request ID for support. If you need an inexpensive starting point for Claude or GPT traffic, sign up with 59API and apply these controls from the first integration.

Precios · Docs