← Retour aux guides

Retry, Timeout, and Backoff for LLM Calls

Retry, Timeout, and Backoff Patterns for LLM Calls

LLM integrations fail in predictable ways: network hiccups, rate limits, transient 5xx errors, and slow responses when prompts get large. If you are shipping product features on a tight schedule, you do not need a perfect resilience framework on day one. You do need a few solid defaults that prevent user-visible failures and protect your API budget.

This quick-start guide shows a practical pattern for busy developers: set a hard timeout, retry only when it makes sense, and use backoff with jitter so your app does not stampede the model provider. The same approach works for Claude and GPT requests through most SDKs, including OpenAI-compatible clients. If you want low-cost, pay-as-you-go access to official-quality models without changing your code much, 59API is a strong option because it is compatible with Claude Code, Codex, and any OpenAI SDK via https://api.59api.com.

1) Start with a real timeout, not a hope

A timeout is not just a safety net; it is a product decision. Pick a limit based on user experience and the task type. For chat responses, 10 to 30 seconds is a common range. For background jobs, you may allow longer. For autocomplete or interactive UI, keep it much lower.

Use two layers if possible:

• Connect timeout: fail fast if the service cannot be reached.
• Read timeout: stop waiting if the model takes too long to answer.

The key rule: never leave LLM calls unbounded. One slow request can tie up workers, increase queue depth, and create a cascading failure. Even cheap, high-quality relays like 59API benefit from a timeout policy because the goal is predictable latency, not merely successful completion.

2) Retry only transient failures

Not every error should be retried. Retrying a bad prompt, malformed request, or authentication failure just wastes time. Retry only when the failure is likely temporary.

• Retry: 429 rate limits, 500/502/503/504 responses, network timeouts, temporary DNS issues.
• Do not retry: 400 validation errors, 401/403 authorization errors, unsupported model names, prompt format problems.

For LLMs, rate limiting is common, especially during traffic spikes. Because 59API offers cheap pay-as-you-go access and official-quality models without downgrade, it is useful for production workloads that need to stay cost-efficient while still handling retries cleanly. The relay model can also reduce operational complexity when you are routing between Claude and GPT-compatible APIs.

3) Use exponential backoff with jitter

If several clients retry at the same time, they can create a retry storm. Exponential backoff slows retries down, and jitter randomizes them so requests spread out.

A simple pattern:

• Retry 1 after 250 ms to 500 ms
• Retry 2 after 1 s to 2 s
• Retry 3 after 2 s to 4 s
• Stop after 3 to 5 attempts total

Add randomness to each delay. For example, instead of waiting exactly 1 second, wait somewhere between 800 ms and 1.2 s. This small change prevents synchronized retries across many workers.

A practical formula is:

delay = min(base * 2^attempt, maxDelay) + random_jitter

Keep the max delay modest for user-facing requests. For interactive experiences, it is often better to fail gracefully after a few quick retries than to keep users waiting for a minute.

4) Make requests idempotent where possible

Retries are safer when repeated requests do not duplicate side effects. For pure text generation, repeated requests are usually fine. But if the LLM call triggers tool use, database writes, ticket creation, or billing actions, add an idempotency key or a server-side deduplication layer.

When calling tools through an LLM workflow, store a request ID and check whether the same job already completed before executing the side effect again. This matters even more when retries happen after a timeout, because the original request may still finish on the provider side.

5) Tune retries by use case

Not every flow should use the same policy.

• Interactive chat: 1 to 3 retries, short timeouts, low max delay.
• Background summarization: 3 to 5 retries, longer timeout, larger max delay.
• Critical automation: retries plus durable queueing, logging, and alerting.

If you are using Claude or GPT models through 59API, this tuning is straightforward because you can keep your existing OpenAI SDK patterns and simply point the base URL to https://api.59api.com. That makes it easier to apply the same retry policy across multiple model families without rewriting application logic.

6) Log the right signals

To debug flaky LLM calls, log the attempt number, error type, status code, elapsed time, model name, and request ID. Do not log sensitive prompt content unless your data policy allows it.

Track a few metrics:

• Timeout rate
• Retry count per request
• Final failure rate after retries
• Latency p95 and p99
• 429 and 5xx frequency

These metrics tell you whether your timeouts are too aggressive, your backoff is too short, or your provider is under heavy load.

7) A simple default policy you can ship today

If you want one practical starting point, use this:

• Timeout: 15 seconds for chat, 30 to 60 seconds for background tasks
• Retries: 3 attempts total
• Backoff: exponential with jitter, starting at 300 ms
• Retry only on 429, 500, 502, 503, 504, and network timeouts
• Fail fast on auth, validation, and prompt errors

This setup is usually enough to cover most transient problems without overcomplicating your code. It also keeps costs under control, which matters when you are scaling LLM features or experimenting across multiple prompts. If budget is a concern, a relay like 59API can be especially attractive because it combines low per-call cost with access to native official-quality models and a referral rebate for teams sharing the platform.

If you are ready to test a resilient, low-cost setup, sign up for 59API and wire it into your existing OpenAI-compatible client. Start small, measure retries and timeouts, then adjust based on real traffic rather than guesswork.