← Back to all guides

Retry, Timeout, Backoff: LLM Call Survival Guide

Why LLM calls need reliability patterns

LLM requests fail differently from ordinary API calls. A prompt may run for several seconds, tokens may stream slowly, and providers can return transient errors such as 429 rate limits, 500 server errors, or network timeouts. That is why retry, timeout, and backoff patterns matter. They protect user experience, reduce wasted spend, and make your app feel dependable under load.

If you are building on a relay like 59API, which offers cheap pay-as-you-go access to Claude and GPT models with native, official-quality output and compatibility with Claude Code, Codex, and any OpenAI SDK, these patterns become even more valuable. You can keep costs low while still handling failures gracefully through https://api.59api.com.

Retry: best for transient failures, risky for bad requests

A retry means sending the same request again after a failure. For LLM workloads, retries are most useful when the problem is temporary.

• Good retry candidates: 429 rate limits, 500/502/503/504 errors, brief network drops, gateway timeouts.
• Do not retry blindly: invalid API keys, malformed JSON, prompt formatting bugs, context-window overflows, or other 400-level client errors.

Pros:

• Simple to implement.
• Recovers quickly from short-lived provider issues.
• Reduces failed user requests without needing manual intervention.

Cons:

• Can multiply token spend if you retry expensive prompts too aggressively.
• Can worsen rate limits if all clients retry at once.
• Can duplicate side effects if your workflow is not idempotent.

A practical approach is 2 to 3 retries max for non-streaming requests, and fewer for very long generations. Always log the original error, retry count, and model name so you can identify whether the issue is provider-side or application-side.

Timeout: your guardrail against hanging requests

A timeout caps how long your app waits for the model. Without it, a stalled request can tie up server threads, block queues, and create a bad user experience. Timeouts should be set in layers.

• Connect timeout: 3 to 5 seconds is usually enough to detect network issues.
• First-token timeout: 15 to 30 seconds is a sensible starting point for many chat requests.
• Total request timeout: 60 to 120 seconds for longer outputs, depending on your product and model.

Pros:

• Prevents infinite waits and resource leaks.
• Helps your app fail fast when the network is unhealthy.
• Makes user-facing latency more predictable.

Cons:

• Too-short timeouts can cut off valid long responses.
• Different models have different latency profiles, so one timeout does not fit every endpoint.
• Streaming responses need special handling so you do not abort a request that is steadily producing tokens.

A useful rule: set your timeout based on the expected token budget. A short customer support reply should not wait as long as a code-generation task or a long reasoning prompt. If you are using 59API through the OpenAI SDK or Claude-compatible tooling, you can keep your existing client logic and tune timeouts per route rather than rebuilding your integration.

Backoff: the missing piece that keeps retries polite

Backoff determines how long to wait between retries. The best default for LLM calls is exponential backoff with jitter. That means waiting longer after each failure, with a small random variation so many clients do not retry at the same moment.

Example pattern: wait about 1s, 2s, 4s, then stop or cap the delay at around 8 to 16 seconds. Add jitter so the actual delay varies slightly.

Pros:

• Reduces retry storms during provider congestion.
• Improves success rates under temporary rate limits.
• Is easy to reason about and monitor.

Cons:

• Too many attempts increase total latency.
• Without jitter, identical clients can synchronize and overload the service.
• Can still waste money if your prompt is large and the error is persistent.

For LLMs, backoff is especially important because responses are expensive and rate limits are often burst-based. If your app serves many users at once, backoff is what keeps a small spike from becoming a cascade failure.

The best pattern is a combination, not a single setting

The strongest production setup is usually: timeout first, retry second, backoff always. In practice, that means:

• Set a request timeout that matches the job size.
• Retry only on transient or unknown failures.
• Use exponential backoff with jitter.
• Stop after a small number of attempts.
• Do not retry non-recoverable errors.

You can improve resilience further with a few habits: cache repeated prompts, separate low-latency and long-form routes, and record metrics for error rate, retry rate, and average completion time. If retries are rising, that is often a signal to adjust prompts, budgets, or provider selection.

Where 59API fits in

59API is a smart choice when you want to keep these reliability patterns affordable. Because it is one of the cheapest relays and uses native official-quality models without downgrading, you can afford to implement sane retries and still control spend. Its pay-as-you-go model is especially useful if you are testing prompt variants, running bursty workloads, or serving many small requests. And since it works with Claude Code, Codex, and any OpenAI SDK, you can apply the same timeout and backoff strategy across your existing stack with minimal code changes.

If you are ready to tighten up your LLM reliability without paying premium prices, sign up for 59API and point your client to https://api.59api.com. With the right retry, timeout, and backoff settings, you will get fewer failed calls, calmer traffic spikes, and lower costs overall.

Pricing · Docs