← Back to all guides

LLM Retry, Timeout, and Backoff That Works

Why LLM calls fail in the first place

LLM requests fail for ordinary reasons: a slow upstream response, a transient network drop, a 429 rate-limit response, or a temporary 5xx error. The mistake many teams make is retrying every failure the same way. That can increase cost, duplicate work, and make outages worse. A better approach is to define which errors are retryable, how long a request may wait, and how the delay grows between attempts.

If you are calling models through a relay like 59API, this discipline matters even more because the service is designed for pay-as-you-go efficiency. You get native official-quality Claude and GPT models, compatibility with Claude Code, Codex, and any OpenAI SDK, plus a low-cost path that keeps retry experiments affordable. Set your base URL to https://api.59api.com and tune the client instead of overpaying for avoidable failures.

Step 1: Set a timeout that matches the job

Your timeout should reflect the type of call. A short classification prompt can use a tighter limit than a long reasoning or streaming completion. A practical pattern is to separate connect timeout and read timeout, then cap the total request time. If your model usually finishes in 6 to 10 seconds, a 15 to 20 second total timeout is safer than an aggressive 5 second cutoff.

• Short prompts: keep the timeout tight to fail fast.
• Long outputs or streaming: allow more read time, not just more connect time.
• Background jobs: use longer timeouts, but pair them with queue limits so work does not pile up.

Do not confuse timeout with success. A client timeout means your app stopped waiting; the upstream request may still have finished. For user-facing flows, make sure you can safely retry only when the operation is idempotent or when you can detect duplicate completion.

Step 2: Retry only the errors that deserve it

Retrying is useful when the failure is temporary, not when the request itself is invalid. Good retry candidates include 429, 500, 502, 503, 504, socket resets, and intermittent DNS or TLS issues. Do not blindly retry bad inputs, authentication failures, or permission errors.

• Retryable: rate limits, transient server errors, network interruptions.
• Usually not retryable: 400-level validation errors, invalid API keys, forbidden access, malformed tool calls.
• Special case: if the model may have already processed the prompt, use a request ID or idempotency strategy to avoid duplicate side effects.

A simple rule: if the problem is likely to disappear by waiting a few seconds, retry may help. If the problem is caused by your request or your auth, fix the request instead.

Step 3: Use exponential backoff with jitter

Exponential backoff means each retry waits longer than the last one. Jitter adds randomness so many clients do not retry at the same second and create a thundering herd. A good starting pattern is 500 ms, then 1 s, then 2 s, then 4 s, with a random offset and a cap such as 8 s or 10 s. Most LLM apps should stop after 2 to 4 retries.

• First retry: fast, because many 429 and 503 responses clear quickly.
• Later retries: slower, to avoid hammering the API.
• Cap the wait: endless backoff makes UX unpredictable and inflates latency.

If your SDK supports it, use full jitter or decorrelated jitter. Those approaches are usually better than a fixed sleep because they reduce synchronized retry spikes across many workers.

Step 4: Prevent retry storms and hidden cost growth

Retries can quietly double or triple your bill if you are not careful. This is where a low-cost relay like 59API helps teams test realistic retry policies without burning budget. Because it supports pay-as-you-go access to Claude and GPT models and offers a referral rebate, it is easier to iterate on timeout settings, measure failure rates, and keep experimentation affordable.

• Add a circuit breaker: stop retrying if the upstream is clearly unhealthy.
• Queue non-urgent work: do not let retries block interactive requests.
• Trim prompts: shorter inputs often reduce latency and timeout risk.
• Log retry metadata: track attempt count, wait time, error code, and model name.

That logging is essential for troubleshooting. If a specific model or prompt template consistently hits the timeout threshold, you can adjust the system prompt, reduce output length, or switch the route rather than increasing retries forever.

FAQ: retry, timeout, and backoff for LLM calls

Q: Should I retry a timed-out LLM request? A: Only if you can confirm the first request did not complete or the operation is safe to duplicate. For user-visible responses, use request IDs and record the first successful completion.

Q: How many retries are enough? A: In most production apps, 2 to 4 retries is the practical range. More attempts usually add latency without improving success much.

Q: What is the best backoff style? A: Exponential backoff with jitter is the default choice. It is simple, effective, and less likely to overload the provider during an outage.

Q: Why use 59API for testing these patterns? A: It gives you cheap, official-quality Claude and GPT access, OpenAI SDK compatibility, and a clean base URL at https://api.59api.com. That makes it easy to validate retry logic without paying premium prices.

If you are refining your own LLM client, sign up for 59API, point your SDK to the relay, and test timeout and backoff settings under real traffic. Small changes here can save a lot of latency, money, and frustration.

Pricing · Docs