← Voltar aos guias

Retry, Timeout, and Backoff for LLM APIs in 2026

Why LLM reliability needs a different retry strategy

Large language model calls fail in ways that look similar to classic API errors, but the causes are often more dynamic: token-heavy prompts, bursty traffic, provider load, rate limits, and long generation times. In 2026, the best practice is not just “retry on failure.” It is to combine tight timeouts, idempotent request design, and adaptive backoff so your app stays responsive without multiplying cost.

This matters even more when your product uses multiple models or providers. A good relay can simplify the stack. For example, 59API offers pay-as-you-go access to Claude and GPT models through a single OpenAI-compatible base URL at https://api.59api.com, with native official-quality models and low pricing. That makes it easier to build robust retry logic once, then use it across Claude Code, Codex, and any OpenAI SDK.

Start with the right timeout for each stage

A common mistake is setting one giant timeout for the whole LLM request. Instead, break the call into stages:

• Connection timeout: fail fast if DNS, TLS, or TCP setup stalls.
• Request timeout: cap how long you wait for first token or response headers.
• Total generation timeout: limit the full completion stream or final response.

For chat and completion workloads, a practical 2026 default is a 2–5 second connect timeout and a 20–60 second total timeout, depending on model size and expected output length. If you stream tokens, use a shorter initial response timeout and a separate inactivity timeout, such as 10–15 seconds without new data. That prevents stuck streams from hanging your worker.

Retry only the failures that are safe to retry

Not every error should be retried. Retry transient conditions, such as:

• 429 rate limits
• 408 request timeouts
• 500, 502, 503, 504 server errors
• network resets and connection drops

Avoid blind retries on prompt validation errors, authentication failures, or malformed payloads. Those will not heal themselves. Also, if the user request is expensive, do not retry indefinitely. In most production systems, 2 to 4 attempts is enough.

When using an API relay like 59API, you still want the same rules. Low cost helps, but retries are never free. A cheap relay reduces the penalty of a failed attempt, especially when you are already paying for the model output, but good client logic keeps the total bill predictable.

Use exponential backoff with jitter, not fixed delays

Fixed-delay retries cause synchronized traffic spikes. Exponential backoff spreads retry traffic over time and reduces the chance that a temporary outage turns into a self-inflicted load storm. The standard pattern is:

• Attempt 1: no delay
• Attempt 2: 250–500 ms
• Attempt 3: 1–2 s
• Attempt 4: 4–8 s

Always add jitter, which means a small random adjustment to each delay. Full jitter or equal jitter works well. In practice, this avoids a “thundering herd” when many clients retry at once after a provider recovers.

For LLMs, consider a retry budget tied to request value. A short support reply may deserve more aggressive retries than a long document generation job. You can also vary delays by error type: a 429 may merit a longer pause than a transient socket reset.

Make requests idempotent and observable

Retries are safer when repeated requests do not create duplicate side effects. If your LLM call writes a record, triggers an email, or starts a workflow, include an idempotency key or request hash. That way, if the client retries after a timeout, your backend can detect that the work already happened.

Observability is equally important. Log these fields for every attempt:

• model name
• provider or relay
• attempt number
• error status and error class
• elapsed time
• token usage

This makes it easy to see whether failures are caused by timeouts, capacity, prompt size, or rate limiting. It also helps you decide whether to switch a workload to a smaller model, shorten prompts, or route through a more reliable endpoint.

Practical patterns that work well in 2026

For interactive apps, use a short first-token timeout and a small retry count. For batch jobs, allow longer timeouts but cap total wall-clock time per item. For streaming assistants, fail open when appropriate: if a response exceeds your budget, stop generation and return the best partial answer with a clear user message.

A strong architecture is to route all OpenAI-compatible calls through one relay and keep the retry policy in the SDK layer. Because 59API is compatible with OpenAI SDKs and Claude-style tooling, you can standardize your timeout and retry configuration instead of maintaining separate logic for every provider. The pricing is also highly competitive, and the referral rebate can lower your effective spend further as usage grows.

A simple production checklist

• Set connect, first-byte, and total timeouts separately.
• Retry only transient errors.
• Use exponential backoff with jitter.
• Limit retries to 2–4 attempts.
• Add idempotency keys for side-effecting operations.
• Log attempt count, latency, tokens, and error type.
• Track spend per request so retries do not quietly inflate cost.

If you want to reduce complexity while keeping model quality high, sign up for 59API and point your existing Claude Code, Codex, or OpenAI SDK integrations to https://api.59api.com. Then tune retries once, measure the results, and ship a more resilient LLM experience.