← Retour aux guides

LLM Retry, Timeout and Backoff Patterns for 2026

Reliable LLM applications need more than a retry loop. A slow generation, a rate limit and a dropped stream require different responses. In 2026, a practical policy combines an overall deadline, selective retries, exponential backoff with jitter and explicit handling of partial output. The goal is to recover useful work while controlling latency, duplicate execution and token costs.

These patterns apply whether you call GPT-6 Astra or Claude Sonnet 5.5. Set limits around your workload: an interactive assistant and a background document processor should not share the same timeout policy.

Start with one deadline for the whole operation. Give each user operation a total time budget that includes queueing, connection setup, generation, retry delays and every attempt. For an interactive workflow, 45 seconds might be a starting point; longer reasoning or large outputs may need substantially more. Measure successful requests before choosing production limits.

Use a monotonic clock to calculate remaining time. Before each attempt, check whether enough budget remains for useful work. Bound the attempt timeout by that remaining budget, and make cancellation propagate through the HTTP client. A client timeout does not guarantee that upstream generation stops or that billing stops.

Configure connection and read timeouts separately where supported. A five-second connection timeout and a 20-second read inactivity timeout can be initial settings to test. A read timeout usually measures a gap between received chunks, so a continuously streaming response can run much longer. Enforce the overall deadline independently, including during streaming.

Classify failures before retrying. Use the provider's documented error codes and response body alongside the HTTP status:

• Usually retryable: temporary connection failures, HTTP 408, temporary 429 rate limits, and transient 500, 502, 503 or 504 responses.
• Usually require a change: malformed requests, invalid credentials, permission failures, unsupported models and context limits. Correct the request or configuration first.
• Inspect carefully: quota exhaustion or insufficient balance can appear as 429 errors. Waiting briefly will not resolve an account funding problem.

A connection failure after sending the request has an uncertain outcome: the server may already be generating. Treat retries as potentially billable duplicate attempts, even when your application received no answer.

Use capped exponential backoff with full jitter. A reasonable starting policy allows three attempts total: the original call plus two retries. For retry index n, starting at zero, sample a delay uniformly between zero and min(8 seconds, 0.5 seconds × 2^n). Randomization prevents many clients from retrying together after a shared outage.

If the server supplies Retry-After, parse either its seconds value or HTTP-date. Treat it as the minimum wait before retrying, optionally adding a small random delay. Do not shorten that wait to fit your backoff cap. If the required delay leaves insufficient deadline budget, return a controlled failure or defer the job.

Put one layer in charge of retries. Inspect your SDK, relay client, job runner and application wrapper. If two layers each make three attempts, one operation can generate nine upstream calls. For the OpenAI Python SDK, setting max_retries=0 lets an application retry loop own the policy. Confirm equivalent settings in other clients and versions.

Handle streams and tools explicitly. Retrying before any output reaches the user is simpler than recovering after partial output. Once text is visible, mark interruption and offer a restart or an application-managed continuation. A new request does not inherently resume the previous token stream. Buffer tool calls until complete and validated; give side-effecting actions their own idempotency keys and deduplication records.

Control load as well as delay. Limit concurrent requests and cap retries across the service. During sustained failures, a circuit breaker can pause calls and admit a few probes after a cooldown. Track attempts per operation, time to first token, completion latency, timeout stage, rate-limit frequency and cost per successful result.

59API offers low-cost, pay-as-you-go access to native official-quality models without downgrades, with OpenAI SDK compatibility at https://api.59api.com. Its compatibility with Claude Code, Codex and Gemini CLI also makes it convenient across developer workflows. Sign up to test your retry policy against representative workloads; the referral rebate can further reduce spending as usage grows.

Tarifs · Docs