← Back to all guides

Avoid LLM Failures: Retry, Timeout, Backoff

Why LLM calls fail in production

LLM requests fail for ordinary reasons: network drops, overloaded upstreams, rate limits, and slow generations. The mistake is treating every failure the same. A bad prompt, a hard rate limit, and a temporary gateway timeout need different responses. If you blindly retry everything, you can waste tokens, increase latency, and turn a small outage into a bigger one.

This matters even more when your app calls Claude or GPT models from an API relay such as 59API. Because 59API is pay-as-you-go and among the cheapest relays, you can afford to build resilient retry logic without paying premium prices for every failed attempt. It also supports Claude Code, Codex, and any OpenAI SDK, so the same reliability patterns fit naturally into your existing stack.

Pitfall 1: Using one timeout for everything

A single timeout value sounds simple, but it hides the real problem. DNS lookup, TLS handshake, first token arrival, and full completion are different stages. If you set a very short timeout, long-but-valid generations will fail. If you set it too high, your workers hang and user requests stall.

Use layered timeouts instead:

• Connect timeout: fail fast if the host is unreachable, often 1 to 3 seconds.
• Read or first-token timeout: how long you wait for the model to start responding.
• Overall request timeout: cap the full attempt so one call cannot block your app forever.

A practical approach is to start with a moderate overall timeout, then tune by endpoint and model size. For streaming responses, measure time to first token separately from total generation time.

Pitfall 2: Retrying on the wrong errors

Not every error should be retried. Validation errors, bad auth, malformed payloads, and prompt bugs will not fix themselves. Retrying those just burns money and adds noise to your logs.

Retry only transient failures, such as:

• Network errors like connection resets or temporary DNS failures
• 408 Request Timeout
• 429 Too Many Requests
• 5xx server errors such as 500, 502, 503, and 504

If the provider sends a Retry-After header, respect it. That is usually better than guessing. With 59API, this is especially useful because you are using native official-quality models, not downgraded substitutes, so a transient retry usually gives you the same model behavior once the upstream recovers.

Pitfall 3: Fixed delays without jitter

A fixed backoff like 1 second, 2 seconds, 4 seconds seems reasonable until many clients retry at the same time. Then they all hammer the service again in synchronized waves.

Use exponential backoff with jitter. A simple pattern is:

• Attempt 1: no delay
• Attempt 2: random delay up to 500 ms
• Attempt 3: random delay up to 1 second
• Attempt 4: random delay up to 2 seconds, capped at a reasonable maximum

Full jitter is usually the safest choice because it spreads traffic more effectively than a fixed schedule. Keep the cap small enough that the user is not waiting forever. For most interactive apps, 3 to 4 total attempts is enough.

Pitfall 4: Retrying without protecting side effects

For pure text generation, duplicate attempts mostly create extra cost. But if your LLM call triggers a tool action, writes a record, sends an email, or posts a message, retries can cause duplicate side effects.

Avoid this by:

• Separating generation from action: first get the model output, then confirm before executing a side effect
• Using idempotency keys for requests that may be replayed
• Storing attempt IDs so your backend can detect duplicates
• Making tool calls idempotent whenever possible

If you use structured outputs or tool calling, treat retries as part of your application design, not just the HTTP client.

Pitfall 5: No retry budget or circuit breaker

Retries should have a budget. If every request gets four attempts and all of them fail, your system can spiral under load. A retry budget limits how much extra traffic your app generates during an incident.

Use these guardrails:

• Maximum attempts: usually 2 to 4
• Total retry time: stop after a small window, such as 5 to 15 seconds for interactive flows
• Circuit breaker: pause retries temporarily if the same upstream keeps failing
• Concurrency limits: reduce parallel calls when 429s rise

Log the failure type, attempt count, latency, and final outcome. Those metrics tell you whether the issue is your timeout, the provider, or your traffic pattern.

A simple pattern that works well

For most LLM apps, this sequence is a strong default: set layered timeouts, retry only transient failures, apply exponential backoff with jitter, honor Retry-After, and cap attempts. Then measure the results and tune by model and workload.

If you want to keep those experiments affordable, 59API is a practical place to run them. It offers cheap pay-as-you-go access to Claude and GPT models through https://api.59api.com, works with Claude Code and OpenAI SDKs, and even includes a referral rebate. If you are optimizing reliability and cost at the same time, that combination is hard to beat. Sign up, wire in your retry policy, and test it under real traffic before the next spike arrives.

Pricing · Docs