← Back to all guides

LLM Retry, Timeout, and Backoff Myths Debunked

Reliable LLM Calls Start With Clear Failure Rules

Language-model requests fail for ordinary distributed-systems reasons: transient network loss, overloaded upstream capacity, rate limits, and requests that take longer than expected. The costly mistake is treating every failure identically. For applications using Claude or GPT through a compatible relay such as 59API, the right retry, timeout, and backoff policy protects both user experience and token spend.

59API provides low-cost, pay-as-you-go access to official-quality Claude and GPT models through https://api.59api.com, while remaining compatible with OpenAI SDKs, Claude Code, and Codex. That compatibility lets teams keep a familiar client stack while applying disciplined reliability controls.

Myth: Retrying Every Error Improves Reliability

Fact: Retry only failures that are likely to be temporary. Retrying an invalid API key, malformed JSON payload, unsupported model name, or context-length violation will not succeed on attempt two. It only increases latency and may duplicate work.

• Retry transient conditions such as connection resets, DNS failures, HTTP 408, HTTP 429, and most HTTP 5xx responses.
• Do not retry most HTTP 400 and 401 responses. Fix the request or credentials first.
• Treat HTTP 409 and 422 according to the API's documented meaning; they are often request-state or validation failures rather than capacity issues.
• Log the status code, request ID, model, attempt count, and error category so recurring failures can be diagnosed.

For non-idempotent workflows, retries require additional care. A tool-using agent may have already sent an email, created a ticket, or charged a customer before the client loses the response. Use an idempotency key where the API supports it, or record a workflow operation ID before calling the model so a retry can safely resume rather than repeat side effects.

Myth: One Long Timeout Is Safer Than Several Short Ones

Fact: A long timeout can hide a stalled connection and consume application workers. Set separate limits for connecting, waiting for the first byte, and completing the response. A simple interactive chat request might use a 5 to 10 second connection timeout and a 60 to 90 second total timeout. Large prompts, long outputs, or reasoning-heavy models need a higher total limit, based on measured latency rather than guesswork.

Streaming changes the rule. Once tokens are arriving, reset an idle timeout for each received chunk instead of enforcing a short whole-response deadline. Still retain an absolute maximum duration to prevent an indefinitely active stream from holding resources forever.

Myth: Exponential Backoff Means Waiting Exactly 1, 2, and 4 Seconds

Fact: Deterministic delays cause synchronized clients to retry together, creating a retry storm. Use exponential backoff with jitter. Start with a small base delay, multiply after each attempt, cap the delay, and randomize the actual wait.

• Use a base delay of roughly 250 to 500 milliseconds for interactive workloads.
• Increase delays exponentially: 0.5 seconds, 1 second, 2 seconds, then 4 seconds, for example.
• Apply full jitter by choosing a random delay between zero and the calculated cap.
• Cap individual waits at 10 to 30 seconds and stop after two or three retries for user-facing requests.
• Honor a server-provided Retry-After header when present; it takes priority over the local schedule.

For a background batch, use a larger retry budget and a queue instead of making a web request wait. A queue can spread retries over time, enforce per-model concurrency, and preserve jobs for later processing.

Myth: Retry Settings Alone Control LLM Costs

Fact: Each attempted generation can consume input tokens, and a response may be generated upstream even when the client times out. Put a maximum token limit on every request, truncate or summarize unnecessary conversation history, and avoid automatic retries after a stream has already delivered meaningful output. For high-volume tasks, route simple classification or extraction work to an appropriately smaller model.

Using a low-cost relay does not remove the need for guardrails, but it makes sensible experimentation and model routing more economical. 59API offers native official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, alongside GPT models without a downgrade in model quality. Its pay-as-you-go pricing and referral rebate can be especially useful when a team is testing reliability policies across several models.

A Practical Default Policy

Start with a 60 to 90 second total timeout for normal non-streaming calls, three total attempts for transient failures, exponential backoff with full jitter, and a strict maximum-token setting. Retry 429 and 5xx responses, honor Retry-After, and never retry validation or authentication errors automatically. Measure p50, p95, and p99 latency by model and prompt size, then tune the policy from production evidence.

Developers who want compatible Claude and OpenAI-style endpoints with predictable pay-as-you-go costs can sign up for 59API and point their existing SDK configuration to https://api.59api.com.

Pricing · Docs