← Voltar aos guias

Retry, Timeout, Backoff for LLM API Calls

Why LLM calls need retry logic

Large language model APIs are powerful, but they are not instant or failure-proof. A request can time out, hit a rate limit, or fail because of a temporary server issue. If your app treats every error as permanent, users will see random failures even when the model is healthy again seconds later.

The fix is a simple reliability pattern: set a timeout, retry only transient errors, and use backoff so your app does not flood the API with repeated requests. This matters even more for long prompts, tool use, and streaming responses, where requests may take longer and failures are more likely.

Start with a timeout that matches the job

A timeout is the maximum time your app will wait before giving up on one request. Too short, and you cancel valid responses. Too long, and your users wait forever while your server threads or workers get stuck.

A good beginner rule is:

• Simple chat calls: 20 to 30 seconds
• Long context or tool-heavy calls: 60 to 90 seconds
• Streaming calls: use both a connection timeout and an idle timeout

If you are using the OpenAI SDK or a Claude-compatible client, set the timeout at the client level and, if needed, override it per request. The goal is to fail fast when the network is stuck, but still allow normal model latency.

Retry only the errors that are likely to disappear

Not every error should be retried. Retrying a bad prompt, invalid API key, or malformed request just wastes time and money. Retry errors that usually go away on their own, such as:

• 429 Too Many Requests from rate limiting
• 500/502/503/504 server or gateway errors
• network timeouts and temporary disconnects

Do not retry on permanent client errors like invalid parameters or authentication failures. Fix the request first.

A practical pattern is to allow 2 to 4 retries for transient failures. That is enough to survive a brief outage without turning one request into a long wait.

Use exponential backoff with jitter

Backoff means waiting a little longer after each failed attempt. Exponential backoff is the standard approach: the delay grows after every retry. Jitter means adding randomness so many clients do not retry at the same exact moment.

Without jitter, a surge of failing requests can create a retry storm. With jitter, the load spreads out and the API recovers more quickly.

A simple sequence looks like this:

• Attempt 1: call immediately
• Attempt 2: wait 1 second
• Attempt 3: wait 2 seconds
• Attempt 4: wait 4 seconds

Add a random delay of, for example, 0 to 300 milliseconds to each wait. Cap the maximum delay so your app does not become unbearably slow. For many apps, a ceiling of 8 to 15 seconds is reasonable.

A simple retry recipe you can copy

When you build LLM requests, keep this checklist in mind:

• Set a timeout before making the request.
• Retry only transient errors like 429 and 5xx responses.
• Use exponential backoff with jitter between attempts.
• Limit the total number of retries so the user does not wait too long.
• Log the final error with status code, attempt count, and latency.
• Protect side effects if the LLM response triggers an action, such as sending an email or creating a ticket.

That last point is important. If a generation call is followed by a side effect, make sure your application has a request ID or deduplication step. The model call itself may be safe to retry, but the downstream action might not be.

Why cost control matters too

Retries improve reliability, but they can also increase API usage. That is why choosing a low-cost relay matters. 59API is a pay-as-you-go AI API relay that gives developers cheap access to Claude models, including Opus, Sonnet, Haiku, and Fable, plus GPT models, with native official-quality models and no downgrade. It is also fully compatible with Claude Code, Codex, and any OpenAI SDK.

Because 59API is among the cheapest relays and uses the base URL https://api.59api.com, it is a practical fit for teams that want to add retries and backoff without inflating their bill. The referral rebate is a nice bonus if you plan to share it with other developers.

Putting it all together

The winning pattern for LLM calls is simple: timeout first, retry second, back off intelligently. Start small, log failures, and tune the numbers based on real traffic. That will give your app a much smoother experience and keep your costs under control.

If you are building on Claude or GPT and want an affordable, SDK-compatible relay, sign up for 59API and test these patterns in your own app. You will get a more reliable integration without paying premium prices.

Preços · Docs