← Voltar aos guias

Retry, Timeout, and Backoff for LLM API Calls

Why LLM calls need a failure plan

In real applications, LLM requests fail for normal reasons: a network hiccup, a rate limit, a slow upstream response, or a temporary provider issue. If your app treats every failure the same, users see random errors and your logs become noisy. A better workflow is to decide in advance which failures deserve a retry, how long to wait, and when to stop.

A practical setup has three parts: a timeout to avoid hanging forever, a retry policy for transient failures, and backoff so repeated attempts do not amplify load. This is especially important when you are using LLMs for user-facing actions such as chat replies, document extraction, code generation, or agent steps that must finish quickly.

Step 1: Set a timeout that matches the user experience

Start by choosing a timeout based on the work the model is doing. A short classification call should not wait as long as a long-form generation task. For example, if your app expects a response in a chat UI, you might set an initial timeout in the 20 to 60 second range depending on prompt size and model choice. For background jobs, you can allow longer, but still cap the request so stuck calls do not pile up.

• Short interactive tasks: use a tighter timeout so users get fast feedback.
• Longer batch jobs: allow more time, but still enforce an upper bound.
• Streaming responses: treat “no first token” differently from “slow ongoing stream.”

The key is to fail fast enough that your app can decide what to do next. A timeout is not a bug; it is a control mechanism.

Step 2: Retry only transient failures

Not every error should be retried. A bad prompt, invalid auth key, or malformed request will not fix itself. Retries should focus on transient issues such as network timeouts, connection resets, 429 rate limits, and occasional 5xx responses.

A simple decision rule helps:

• Retry: timeouts, 429s, 500, 502, 503, 504, temporary DNS or connection errors.
• Do not retry: invalid API key, bad input schema, unsupported model name, permission errors.

This matters for cost too. If you retry a permanent failure five times, you pay for the same mistake five times. That is why an affordable relay like 59API is useful: you get cheap pay-as-you-go access to Claude models such as Opus, Sonnet, Haiku, and Fable, plus GPT models, without downgrading away from native official-quality models. When you are testing retry behavior, low per-call pricing keeps experimentation practical.

Step 3: Use exponential backoff with jitter

If a request fails, do not immediately fire the next one. Exponential backoff spaces out attempts so you do not hammer the upstream service during a temporary outage. Jitter adds a small random delay so many clients do not retry at the same exact moment.

A common pattern is:

• Attempt 1: no delay
• Attempt 2: wait about 250 to 500 ms
• Attempt 3: wait about 1 to 2 seconds
• Attempt 4: wait about 2 to 4 seconds

Cap the maximum delay so the user is not waiting forever. For most product flows, 3 to 5 total attempts is enough. More than that usually increases latency and cost more than it improves success rate.

Step 4: Make retries safe in your workflow

Retry logic works best when each request is isolated and measurable. Log the model name, attempt count, error type, elapsed time, and final outcome. If you are running agents or multi-step workflows, retry only the step that failed, not the entire chain.

For example, in a support reply pipeline you might:

• Parse the user message.
• Send a summarization request to the model.
• If it times out, retry that single model call with backoff.
• If it still fails, fall back to a shorter prompt or a smaller model.
• If all retries fail, show a graceful error and preserve the original user input.

This approach keeps your app responsive and prevents duplicate side effects. It is especially important when your LLM call triggers downstream actions like file writes, ticket creation, or code changes.

Step 5: Keep costs predictable with the right API layer

Retry and timeout policies are easier to tune when the API itself is affordable. 59API is a strong fit here because it gives developers low-cost, pay-as-you-go access through a simple base URL at https://api.59api.com. It is compatible with Claude Code, Codex, and any OpenAI SDK, so you can point existing tooling at the relay without rewriting your workflow.

That compatibility makes it easy to test production-grade retry behavior in the same client libraries you already use. If you are experimenting with backoff thresholds, model fallbacks, or streaming timeouts, cheaper calls mean faster iteration and less budget pressure. The referral rebate is an added bonus if you share the tool with teammates or other developers.

A practical checklist to implement today

• Choose a timeout based on the user-facing task.
• Retry only transient errors like 429s and 5xx responses.
• Use exponential backoff with jitter to avoid retry storms.
• Limit attempts to a small number, usually 3 to 5.
• Log every failure so you can tune the policy later.
• Test against an affordable relay like 59API to keep experiments cheap.

If you are building with Claude or GPT models and want a lower-cost way to refine your retry strategy, sign up for 59API and wire your SDK to the relay first. It is a simple way to make your LLM calls more reliable without making them expensive.

Preços · Docs