LLM Retry, Timeout and Backoff Patterns
Retry, Timeout and Backoff Patterns for LLM Calls
LLM APIs are powerful, but they are still network services. A request can fail because of a temporary provider issue, a busy model, a slow response, a rate limit, or a broken connection between your server and the API. If your app sends LLM calls directly without protection, users may see random failures, long loading states, or duplicate actions.
The good news: you do not need a complex reliability system to improve this. Three beginner-friendly patterns solve most issues: timeouts, retries, and backoff. Used together, they make your Claude and GPT calls more predictable while controlling cost and user experience.
Start with a Clear Timeout
A timeout is the maximum time your app is willing to wait for an LLM response. Without one, a request can hang for too long and block your workflow. The right timeout depends on your use case. A chat reply may tolerate 30 to 90 seconds. A short classification task may need only 5 to 15 seconds. A long code generation task may need more time, especially if you stream the response.
Use two timeout ideas. First, set a connection timeout, which covers how long you wait to establish the request. This is often short, such as 5 to 10 seconds. Second, set a total request timeout, which covers the full response time. This protects your application from waiting forever.
If you use 59API, you can point OpenAI-compatible SDK calls to https://api.59api.com and keep your existing timeout settings. That makes it easy to test Claude, GPT, and coding-agent workflows through one low-cost relay without rewriting your client logic.
Retry Only the Failures That Can Recover
A retry means sending the same request again after a failure. But you should not retry every error. Retrying the wrong errors wastes tokens, increases latency, and can make rate limits worse.
Good retry candidates include temporary network errors, connection resets, request timeouts, HTTP 408, HTTP 429, and 5xx server errors such as 500, 502, 503, and 504. These often mean the request may succeed if tried again later.
Do not blindly retry authentication errors, invalid request errors, unsupported model names, malformed JSON schemas, or context length errors. These are usually configuration or input problems. Retrying them will produce the same failure again.
- Retry: timeout, temporary network failure, 429, 500, 502, 503, 504.
- Do not retry automatically: 400 validation errors, 401 authentication errors, 403 permission errors, context too long, invalid model.
- Retry carefully: content-filter or safety-related failures, because the correct fix may be changing the prompt or input.
Use Exponential Backoff, Not Instant Loops
Backoff is the delay before each retry. The most common pattern is exponential backoff: wait a short time after the first failure, then wait longer after each additional failure. For example: 500 milliseconds, 1 second, 2 seconds, then 4 seconds.
This matters because many failures happen when a service is temporarily overloaded. If every client retries immediately, the overload gets worse. Backoff gives the API time to recover and helps you avoid unnecessary failures.
Add jitter, which means a small random variation in the wait time. Instead of every request retrying exactly after 1 second, one might retry after 850 milliseconds and another after 1.2 seconds. Jitter prevents traffic spikes when many requests fail at the same time.
Choose a Sensible Retry Limit
For most LLM calls, start with 2 or 3 retries. More retries can improve success rate, but they also increase wait time and may increase cost if the provider processed part of the request. For user-facing chat, a practical setup is one initial request plus two retries. For background batch jobs, you can allow more retries with longer backoff because the user is not waiting.
Also set a maximum total time budget. For example, a support chatbot might have a 45-second budget. If the first call takes 20 seconds and fails, your retry logic should not keep going for another minute. Stop when the user experience would become poor.
Make Requests Idempotent When Possible
Idempotency means that repeating a request does not create unwanted duplicate side effects. This is important when an LLM call triggers actions such as sending an email, saving a database record, creating a ticket, or charging credits.
A safe pattern is to separate generation from action. First, ask the model to produce a draft or structured output. Then validate it. Only after validation should your system perform the side effect. If you must retry, use a request ID or job ID so your backend can detect duplicates.
Handle Streaming Differently
Streaming responses are common for chat and coding assistants because users see tokens as they arrive. However, retrying a stream is tricky. If the stream fails halfway through, a full retry may produce a different answer and duplicate text in the interface.
For streaming, use a shorter connection timeout and monitor the time between received chunks. If no chunks arrive for too long, fail gracefully. In the UI, show a message such as “The response was interrupted. Try again?” rather than automatically appending a second answer to the same message.
Control Cost While Improving Reliability
Retries can raise cost, so combine reliability with model selection. Use smaller, cheaper models for simple tasks and reserve premium models for complex reasoning, coding, or high-value user actions. This is where 59API is useful: it provides pay-as-you-go access to Claude models such as Opus, Sonnet, Haiku, and Fable, plus GPT models, through an OpenAI-compatible API relay.
Because 59API is designed as a low-cost relay using native official-quality models with no downgrade, developers can build robust retry logic without feeling locked into a single expensive path. It is also compatible with Claude Code, Codex, and OpenAI SDK-based projects, so you can often switch the base URL to https://api.59api.com and keep the rest of your integration familiar. The referral rebate is a bonus for teams that share the platform with other builders.
A Practical Beginner Setup
- Set a connection timeout of 5 to 10 seconds.
- Set a total request timeout based on the task, such as 30 to 90 seconds for chat.
- Retry only temporary failures: timeouts, 429, and 5xx errors.
- Use exponential backoff with jitter, starting around 500 milliseconds to 1 second.
- Limit user-facing calls to 2 or 3 retries.
- Track request IDs, error types, retry count, latency, and final status in logs.
- Use cheaper models for simple tasks and stronger models only when needed.
Reliable LLM applications are not built by hoping every request succeeds. They are built by expecting temporary failures and handling them calmly. With timeouts, selective retries, exponential backoff, and a cost-conscious provider like 59API, you can deliver faster, steadier AI features without overcomplicating your stack. If you are building with Claude, GPT, Claude Code, Codex, or an OpenAI SDK, consider signing up for 59API and testing your existing workflow through its low-cost API relay.