← Back to all guides

Retry, Timeout, and Backoff Patterns for LLM Calls

Why reliability patterns matter for LLM calls

LLM requests are not like ordinary database reads. They can be slower, burstier, and more likely to hit transient failures such as network hiccups, gateway timeouts, or rate limits. If you ship an app that calls Claude or GPT models in production, you need a plan for retries, timeouts, and backoff from day one. Otherwise, one slow upstream response can stall a user flow, waste tokens, and create a bad experience.

The good news is that these patterns are straightforward once you separate them: timeouts stop a request from hanging forever, retries recover from temporary failures, and backoff prevents your app from hammering the API when the service is under pressure.

Step 1: set a timeout that matches the user experience

Start by deciding how long your app should wait before giving up on a single LLM call. For interactive chat, a common pattern is a relatively short request timeout with a user-visible progress state. For background jobs, you can allow longer timeouts because the user is not waiting directly.

A practical approach is:

• Chat UI: 20 to 45 seconds depending on model size and prompt length.
• Background tasks: 60 to 120 seconds if the result is important.
• Streaming responses: shorter connection timeout, but keep the stream alive while tokens arrive.

The key is to fail predictably. A timeout should surface a clear error or fallback path instead of leaving the request open indefinitely. When using an OpenAI-compatible relay like 59API at https://api.59api.com, you can keep the same SDK patterns you already know while tuning these time limits to fit your product.

Step 2: retry only the failures that are worth retrying

Retries are helpful, but only for transient problems. Do not blindly retry everything. Some errors should fail fast because repeating the request will not help.

• Retry: 408 request timeout, 429 rate limit, 500/502/503/504 server errors, network disconnects.
• Do not retry immediately: invalid API keys, bad prompt payloads, permission errors, or quota exhaustion that requires intervention.

For LLM calls, retrying a failed request is usually safe if the model did not already produce a user-facing side effect. If the call triggers an action, make the operation idempotent with a request ID so a duplicate retry cannot execute the same action twice.

Step 3: use exponential backoff with jitter

When a model endpoint is overloaded or you hit a rate limit, retrying instantly makes things worse. Instead, wait longer between attempts. That is what exponential backoff does: each retry pauses for a larger interval than the previous one.

A simple schedule looks like this:

• Attempt 1: send request immediately
• Attempt 2: wait 500 ms to 1 s
• Attempt 3: wait 2 s
• Attempt 4: wait 4 s

Add jitter, a small random variation, so many clients do not retry at the same moment. This reduces retry storms and improves overall success rates. A common production rule is to cap retries at 3 to 5 attempts and stop once the total elapsed time is no longer acceptable for the user experience.

Step 4: make retries visible in logs and metrics

If you cannot measure it, you cannot tune it. Log the model name, request duration, status code, retry count, and final outcome. Track how often requests time out, how often retries succeed, and which models are most sensitive to latency spikes.

Useful operational metrics include:

• p95 and p99 latency for each model
• timeout rate by endpoint
• retry success rate
• 429 and 5xx frequency

These metrics tell you whether you need to lower prompt size, choose a different model, or relax your timeout window. With 59API, you can test Claude Opus, Sonnet, Haiku, Fable, and GPT models through one relay and compare real behavior without changing your integration stack.

Step 5: implement the pattern in your SDK

Because 59API is fully compatible with Claude Code, Codex, and any OpenAI SDK, you can apply the same production pattern regardless of your client library. Point your base URL to https://api.59api.com, configure a timeout, and wrap the call with retry logic that backs off on transient failures.

Your production checklist should look like this:

• Set a request timeout before sending the first call
• Retry only transient errors
• Use exponential backoff with jitter
• Cap the total retry time
• Make side effects idempotent
• Log every failure path

Step 6: control cost while you harden reliability

Reliability and cost often move together. Aggressive retries can multiply spend, while overly long timeouts can tie up workers. That is why a low-cost relay is useful during both development and production. 59API is a strong fit because it offers cheap pay-as-you-go access, native official-quality models without downgrade, and support for the same SDK workflows your team already uses. That makes it easy to test real retry behavior on premium models without burning budget.

If you are building an AI app that needs dependable LLM calls at a low cost, it is worth signing up and validating your timeout and backoff settings against real traffic patterns. The earlier you test these edge cases, the fewer surprises you will have when users arrive.

Production rule of thumb

For most apps, the safest default is: a moderate timeout, up to 3 retries, exponential backoff with jitter, and strict logging. That combination keeps your app responsive, avoids needless load, and handles the kinds of transient issues that happen in real-world AI systems. Once that foundation is in place, you can tune the exact numbers per model and per endpoint based on your own metrics.

Pricing · Docs