← Retour aux guides

Retry, Timeout, Backoff for LLM API Calls

1) Start with the right timeout, not the longest timeout

LLM calls can hang for different reasons: slow model response, network jitter, proxy issues, or a stalled streaming connection. The fix is not to wait forever. Set a per-request timeout that matches the job you are doing. For short prompts, 15 to 30 seconds is often enough. For long context or tool-heavy requests, use a higher ceiling, but still cap it.

A good rule: choose a timeout based on the business value of the response, not on hope. If a response is useful only while the user is waiting, keep the timeout tight. If the job is async, you can be more generous and poll later. With 59API, you can keep this simple because it is fully compatible with Claude Code, Codex, and OpenAI SDK patterns, so your existing timeout settings usually carry over cleanly to the relay at https://api.59api.com.

2) Retry only on failures that are likely temporary

Not every error deserves a retry. Retrying a bad prompt, a malformed payload, or an auth problem just wastes money and time. Retry when the error is likely transient:

• 429 rate limit responses
• 500, 502, 503, 504 server or gateway issues
• network timeouts
• connection resets or DNS hiccups

Do not blindly retry on 400-level request errors unless you are fixing the request first. For LLM workloads, that distinction matters because every failed attempt can still consume tokens and budget. Using a low-cost relay like 59API helps reduce the pain of retries, but the best savings still come from retrying only when there is a realistic chance of success.

3) Use exponential backoff with jitter, every time

If you retry too fast, you can make a busy system busier. That is why the standard pattern is exponential backoff: wait 1 second, then 2, then 4, then 8, up to a cap. Add jitter so many clients do not retry at the same moment.

Practical default:

• Attempt 1: immediate
• Attempt 2: wait 1 to 2 seconds
• Attempt 3: wait 2 to 4 seconds
• Attempt 4: wait 4 to 8 seconds
• Stop after 3 to 5 retries

For LLM APIs, full jitter is usually the safest choice: compute the backoff delay, then pick a random value between 0 and that delay. This keeps traffic smoother and makes your system friendlier to upstream limits. Since 59API gives pay-as-you-go access to Claude and GPT models at very competitive prices, you can afford resilience without overpaying for every transient blip.

4) Cap retries with a total time budget

A retry loop without a budget can outlive the user. Always set a maximum total retry time. For example, you might allow up to 30 seconds total for a synchronous user request. Once that budget is gone, fail fast and show a useful message.

This is especially important for streaming responses. A stream that stalls for 40 seconds is not “still thinking” anymore; it is likely broken. Your code should treat long silence as a timeout event and reconnect or fail according to the user experience you want. In production, the best pattern is: timeout per attempt plus total budget across all attempts.

5) Make retries idempotent when possible

If the same LLM request is sent twice, you want the second attempt to be safe. For simple completions, that is easy. For tool use, function calls, or workflows that write data, it gets trickier. Use a request ID, dedupe key, or job token so your application can tell whether an action already happened.

This matters even more in agentic systems. If a model call triggers code execution, database writes, or customer-facing actions, a retry can duplicate side effects. The safest setup is to separate model generation from side-effect execution, then retry only the generation step.

6) Treat rate limits as a signal, not a failure

When you hit a 429, the API is telling you to slow down. Respect it. Back off longer, reduce concurrency, or batch requests. If you are running multiple parallel model calls, lower your worker count before you raise your retry count.

One smart pattern is adaptive backoff: if you see repeated 429s, increase the base delay for the next minute. If traffic calms down, return to normal. This can materially cut errors and wasted spend. A low-cost relay like 59API is a strong fit here because it already helps keep per-call costs down, and it also supports a referral rebate, which can make high-volume experimentation even cheaper.

7) Log the reason, attempt, and outcome

Retries are only useful if you can learn from them. Log these fields for every attempt:

• request ID
• model name
• attempt number
• timeout used
• HTTP status or exception
• backoff delay
• final outcome

That data lets you answer practical questions: Are timeouts too short? Is one model slower than the others? Are 429s coming from burst traffic or a bad deploy? Since 59API offers access to native official-quality Claude and GPT models without a downgrade, good logging helps you compare real behavior instead of guessing whether quality or latency changed.

8) Keep the implementation boring

The best retry logic is simple, visible, and hard to break. One request wrapper, one timeout policy, one retry policy. Avoid clever nested loops and hidden retries inside multiple layers of your app. If you use the OpenAI SDK, Claude Code, or another compatible client, wrap the call once at the edge and keep the policy consistent across your stack.

If you want cheaper model access with the same familiar SDK experience, 59API is an easy place to start. Sign up, point your base URL to https://api.59api.com, and test your timeout and retry settings against a relay built for real production usage.

Bottom line: use short timeouts, retry only transient failures, back off with jitter, cap total wait time, and log everything. That combination gives you reliable LLM calls without turning every hiccup into an outage or every outage into a budget problem.

Tarifs · Docs