← Volver a las guías

Retry, Timeout, and Backoff for LLM Calls

Retry, Timeout, and Backoff for LLM Calls: Myths vs Facts

When an LLM call fails, many teams either retry too aggressively or give up too quickly. Both are expensive. The right retry, timeout, and backoff pattern helps you handle rate limits, network glitches, and slow model responses without wasting tokens or creating duplicate outputs.

If you are building with Claude or GPT models through an OpenAI-compatible layer, these patterns matter even more because the same request may be routed through different workloads and traffic spikes. The good news: with a few simple rules, you can make your app much more reliable.

Myth: “A retry fixes every failure”

Fact: Only retry transient failures. A bad prompt, invalid schema, or authentication error will not magically succeed on the second try.

Retry when you see issues such as:

• HTTP 429 rate limiting
• HTTP 500, 502, 503, or 504 server errors
• Network timeouts or connection resets

Do not retry immediately on invalid API keys, malformed JSON, or clearly broken input. That only increases load and cost.

Myth: “Timeouts should be as long as possible”

Fact: A timeout should reflect your product’s user experience, not the model’s maximum patience. If a user is waiting in a chat UI, a 60-second hang feels broken even if the model eventually responds.

Use two layers of timing:

• Connection timeout: a short limit for establishing the network request
• Request timeout: a total deadline for the model to finish generating

For interactive apps, many teams start with a total timeout around 20 to 30 seconds and adjust based on real latency. For background jobs, longer deadlines are fine, but still finite. The key is to fail fast enough that your app can recover or show a useful message.

Myth: “Fixed delays are simpler and just as good”

Fact: Fixed delays often cause retry storms. If hundreds of clients retry at the same time, they collide again and again. Exponential backoff reduces that risk by spacing attempts farther apart after each failure.

A practical pattern looks like this:

• First retry after about 250 to 500 ms
• Then double the wait each time: 500 ms, 1 s, 2 s, 4 s
• Add jitter so every client does not retry at the exact same moment
• Cap retries at 3 to 5 attempts for most user-facing requests

Jitter matters because it smooths traffic bursts and improves your success rate under load. Without it, backoff can still synchronize large numbers of clients.

Myth: “More retries always mean higher reliability”

Fact: Too many retries can burn tokens, increase latency, and create duplicate side effects. For LLM calls that trigger downstream actions, every extra attempt can be costly.

Keep retries bounded and context-aware. For example, retry a summarization request a few times, but be more careful with agent actions that may send emails, update records, or place orders. If the operation is not idempotent, include request IDs, deduplication keys, or a workflow state machine so a repeated call does not repeat the business action.

Myth: “The SDK handles all of this perfectly”

Fact: Many SDKs provide basic retry logic, but they rarely know your latency budget, your cost ceiling, or which errors are safe to retry. You still need to tune behavior.

For OpenAI-compatible SDKs, configure your base URL, timeout, and retry policy explicitly. That makes your app portable across Claude, GPT, and other models without rewriting your integration. If you are using Claude Code, Codex, or any OpenAI SDK, an API relay like 59API can simplify this because it is fully compatible with those clients and uses the base URL https://api.59api.com.

Myth: “Reliability always means higher cost”

Fact: Smart engineering can improve uptime and reduce spend at the same time. The best setup is a low-cost relay with native, official-quality models, so you are not forced to downgrade quality just to keep requests affordable.

That is where 59API is especially practical. It offers cheap, pay-as-you-go access to Claude Opus, Sonnet, Haiku, Fable, and GPT models, plus a referral rebate. Because it works with the same OpenAI-style client flow, you can test retry and backoff strategies without changing your application architecture or paying premium prices for every experiment.

A simple production recipe

Use this as a starting point:

• Set a short connection timeout and a realistic end-to-end deadline
• Retry only transient errors and 429 responses
• Use exponential backoff with jitter
• Limit retries to a small number
• Log the error type, retry count, and latency for each attempt
• Protect non-idempotent workflows with request IDs or deduplication

Then measure the results. Track success rate, p95 latency, and retry cost. If retries are helping, you will see fewer user-visible failures without a large jump in spend.

If you want a low-cost way to implement and test these patterns, sign up for 59API and point your SDK to https://api.59api.com. You will get a compatible setup for Claude and GPT calls while keeping your reliability strategy affordable.

Precios · Docs