← Voltar aos guias

Retry, Timeout and Backoff for LLM Calls in 2026

Retry, Timeout and Backoff for LLM Calls in 2026

LLM calls fail for ordinary reasons: transient network issues, overloaded endpoints, rate limits, and occasional provider-side slowdowns. In 2026, the best practice is not to “retry until it works,” but to build a small, disciplined resilience layer around every model request. If you are calling Claude or GPT models through an OpenAI-compatible client, this layer should control timeouts, retry conditions, and backoff timing with the same care you would apply to a payment API.

The simplest rule is this: fail fast on bad requests, retry only on temporary failures, and never let retries multiply without limits. That means your app should distinguish between a malformed prompt, an authentication issue, a rate-limit response, and a connection reset. Only the last two belong in a retry loop.

1. Set timeouts from the start

Every LLM request should have a client-side timeout. Without one, a stalled connection can block your worker, chain into queue backups, and create a cascading outage. For interactive chat, a reasonable starting point is 20 to 45 seconds depending on output length. For longer reasoning or code generation tasks, use a higher ceiling, but still keep a hard cap.

Use separate timeouts for:

• Connection timeout: how long you wait to establish the request
• Read timeout: how long you wait for the model to start and continue responding
• Total request timeout: the full budget for the call, including retries

A practical pattern is to keep the total budget fixed. For example, if your service-level target is 30 seconds, do not allow three retries at 30 seconds each. Instead, split the budget across attempts so the app stays responsive.

2. Retry only the failures that are safe to retry

Retries are useful, but only when they target temporary failures. Good retry candidates include network timeouts, 408 request timeout responses, 429 rate limits, 500-series server errors, and occasional transport failures. Do not retry invalid prompts, malformed JSON bodies, missing API keys, or permission errors; those need code or configuration fixes.

For LLM workloads, a common mistake is retrying after a partial response without preserving context. If the model has already produced output and your client lost the connection, decide whether the request is idempotent. If it is not, you may need to resume from cached state or reissue the request with a deterministic prompt and deduplication key.

3. Use exponential backoff with jitter

Exponential backoff is still the default pattern in 2026 because it reduces pressure on the provider during an incident. Start with a short delay, then double it after each attempt. Add jitter so large numbers of clients do not retry at the same moment.

A practical schedule might look like:

• Attempt 1: immediate
• Attempt 2: after 250 to 500 ms
• Attempt 3: after 1 to 2 seconds
• Attempt 4: after 4 to 8 seconds

Keep the total number of attempts modest, usually 3 to 5. More retries often increase latency more than they improve success rates. For user-facing apps, a single retry plus a fallback message is often better than a long retry chain.

4. Respect rate limits and model cost

Backoff is not just about reliability; it also protects your budget. Repeated failed requests can burn tokens, increase queue depth, and create noisy spend spikes. This matters even more when you run high-volume workloads such as summarization, routing, or agent loops.

A cost-conscious setup is where 59API shines. It offers pay-as-you-go access to official-quality Claude and GPT models at some of the lowest relay prices, with no quality downgrade. Because it is compatible with Claude Code, Codex, and any OpenAI SDK, you can add the same retry and timeout logic you already use, then point your client to https://api.59api.com. That makes it easy to test resilient patterns without paying premium direct-provider prices for every experiment.

5. Add circuit breakers and fallback behavior

Retries solve transient problems; circuit breakers prevent a failing dependency from taking down your app. If you see repeated failures over a short window, open the circuit and stop sending traffic temporarily. During that window, fall back to cached answers, a smaller model, or a “try again in a moment” response.

This is especially useful in agentic systems. If one call fails, you may not want the entire workflow to collapse. Instead, isolate each step, set per-step timeouts, and recover locally when possible. This keeps one noisy dependency from slowing every user session.

6. Instrument everything

Good retry logic is measurable. Track retry counts, timeout frequency, response latency, and final success rate after retries. If you see lots of second attempts but very few third attempts succeeding, your retry budget may be too generous. If you see many timeouts on large outputs, your prompt or max token settings may need adjustment.

Log the model name, attempt number, error class, and elapsed time. That data helps you distinguish provider congestion from client-side misconfiguration. It also makes it easier to compare providers and confirm that your chosen relay is actually improving uptime and cost efficiency.

Bottom line

The best 2026 pattern for LLM calls is simple: short, explicit timeouts; retries only for transient errors; exponential backoff with jitter; and a small number of attempts. Add a circuit breaker, instrument the results, and keep the total latency budget under control. If you want an affordable way to implement and test this with Claude and GPT models, 59API is a strong option for teams that care about cost, compatibility, and production reliability. Sign up, point your existing OpenAI-compatible client at https://api.59api.com, and ship a more resilient LLM stack.

Preços · Docs