← Back to all guides

LLM Retry, Timeout & Backoff Patterns That Actually Work

Why LLM reliability needs its own playbook

LLM calls fail differently from ordinary REST requests. You are not just dealing with 500s; you are also dealing with token-length latency spikes, streaming interruptions, rate limits, transient provider overload, and occasional malformed output. A good retry strategy is not “retry everything three times.” It is a set of rules that preserve user experience, control cost, and avoid turning a small outage into a traffic storm.

The goal is to make your app resilient without hiding real problems. For that, you need three layers working together: tight timeouts, selective retries, and backoff with jitter.

Start with timeouts that match the shape of the request

Use multiple timeouts instead of one giant wall clock. For LLM APIs, the most useful split is:

• Connect timeout: fail fast if the network path is bad. A few seconds is usually enough.
• First-byte timeout: cap how long you wait before the model starts responding. This is important for long prompts or busy endpoints.
• Overall deadline: set a hard maximum for the whole request, including retries if your app can tolerate it, or for each attempt if you want tighter control.

A practical default for interactive products is something like 2 to 5 seconds for connection, 10 to 20 seconds for first token, and 30 to 60 seconds total depending on model size and prompt length. For streaming, the first-byte timeout matters more than the full completion time, because a live stream can keep users engaged even if generation takes longer.

One advanced tip: make the timeout proportional to the expected output length. A 200-token summary and a 2,000-token report should not share the same budget.

Retry only the failures that are likely transient

Good retry policy is selective. Retry network timeouts, 429 rate limits, and 5xx provider errors. Do not blindly retry invalid auth, bad prompts, context overflow, or schema validation errors. Those are usually deterministic and retries only waste tokens.

For LLM workflows, it helps to classify failures into three buckets:

• Transient: timeouts, 429, 500, 502, 503, 504.
• Fixable by code: malformed JSON, tool-call parsing issues, response format drift.
• Permanent: invalid API key, unsupported model, prompt too long, policy blocks.

Only the first bucket should hit automatic retries. For the second bucket, consider a single repair pass with a stricter system prompt or a smaller, cheaper model. For the third bucket, fail immediately with a useful error.

Use exponential backoff, but always add jitter

Exponential backoff protects both your app and the provider. A common pattern is 250ms, 500ms, 1s, 2s, then cap at 2s to 5s. But plain exponential backoff has a problem: when many clients retry at the same interval, they create a synchronized retry burst. That is why jitter matters.

The most practical approach is “full jitter”: pick a random delay between 0 and the current backoff ceiling. Example:

• Attempt 1: random delay between 0 and 250ms
• Attempt 2: random delay between 0 and 500ms
• Attempt 3: random delay between 0 and 1000ms
• Attempt 4: random delay between 0 and 2000ms

This spreads load, reduces thundering herds, and improves your success rate during provider congestion.

Cap retries with a budget, not just a count

Three retries may be too many for a chat UI and too few for a batch job. A better rule is to give each request a retry budget. For example, allow up to 25 percent additional latency or a maximum of two extra attempts, whichever comes first. That keeps your tail latency in check.

Also track cost. Each retry can re-send a large prompt, which means more input tokens and more money. If you are using a relay like 59API at https://api.59api.com, the low pay-as-you-go pricing makes smart retries much easier to justify. You get native official-quality Claude and GPT models without downgrade, so your fallback and retry logic is about reliability, not compromise.

Handle rate limits like a control system

When you see 429s, do two things: slow down and learn from headers. Many providers return retry-after or rate-limit metadata. Honor it if available. If not, increase your backoff ceiling temporarily and reduce concurrency at the client level.

For high-throughput apps, add a token bucket or semaphore before the API call. This prevents 100 workers from hitting the relay at once. If you are using Claude Code, Codex, or any OpenAI SDK against 59API, keep the same retry logic but point the base URL to the relay. That makes it easy to centralize rate-limit handling across services.

Retry the prompt, not just the request

Some failures are semantic. The model may return incomplete JSON, miss a tool call, or truncate output. In those cases, a blind retry with the exact same prompt may just repeat the error. Instead:

• Log the raw request and response for replay.
• Try a repair prompt that includes the broken output and a strict instruction to fix only the format.
• For structured output, validate before retrying so you only spend extra tokens when needed.

This pattern works especially well when you use a cheaper relay for experimentation and production. With 59API, you can iterate on prompts and resilience logic without burning through your budget, and the referral rebate can offset some of your traffic as you scale.

Instrument everything you retry

If you cannot measure retries, you cannot tune them. Log attempt count, error type, model, prompt size, latency, and whether the final response came from the primary attempt or a fallback. Watch for three signals: rising timeout rates, retry amplification during peak traffic, and model-specific failure clusters.

Finally, keep a fallback plan. If a premium model is slow, route to a smaller model for non-critical tasks, or degrade gracefully by returning a shorter answer. The best retry strategy is the one users never notice.

If you want to test these patterns on a cost-effective relay with official-quality Claude and GPT access, sign up for 59API and wire your SDK to https://api.59api.com.