← सभी गाइड पर लौटें

Production-Ready Retries for LLM API Calls

Why LLM calls need a resilience plan

LLM requests can fail for reasons your application cannot control: a temporary network problem, a provider-side error, a rate limit, or a response that takes longer than your user will wait. A reliable integration handles these cases deliberately instead of retrying every failure or leaving requests to hang.

This walkthrough adds timeouts, selective retries, and jittered backoff, then covers the production details that prevent duplicate work and runaway costs. The same principles apply whether you call Claude or GPT through a compatible SDK.

1. Set a timeout before adding retries

Start by deciding how long a user-facing operation can take. Set a total deadline for the whole operation, then make sure each network attempt fits inside the remaining time. For example, a request with a 30-second deadline cannot safely allow three attempts of 20 seconds each.

Configure the HTTP client or SDK with a request timeout appropriate to your task. A short classification call may need only a few seconds; a long generation or tool-using workflow may need more. Also set an application-level deadline, because a client timeout alone may not stop work in your own queue or web request.

When the deadline expires, return a clear error or move the task to an asynchronous workflow. Do not keep retrying past the point where the result is useful.

2. Retry only failures that might recover

Retry transient failures, not every error. Network disconnects, temporary server errors such as HTTP 500 or 503, and rate limits can often recover. For HTTP 429, respect the server's Retry-After value when it is present, provided waiting still fits within your deadline.

Usually do not retry authentication errors, invalid parameters, unsupported models, or other client errors that require a code or configuration change. Retrying these wastes time and may repeat billable requests. Check your SDK's automatic retry behavior too: adding your own retry loop on top of automatic retries can multiply the number of attempts.

3. Use exponential backoff with jitter

Immediate retries can send a burst of traffic to an already busy service. Instead, increase the delay after each failed attempt and add randomness so many clients do not retry simultaneously. A common schedule is approximately 0.5 seconds, 1 second, then 2 seconds, with random jitter and a maximum delay such as 8 seconds.

Keep the total attempt count small, often three attempts including the first, and cap the total elapsed time. Before sleeping, compare the proposed delay with the remaining deadline. If it does not fit, stop and report the failure. For rate limits, a longer server-provided delay may be more appropriate than the normal schedule.

4. Protect against duplicate work

A timeout does not prove the model never received the request. The server may have completed generation while the response was lost. Retrying can therefore create extra latency and cost, and may repeat side effects in workflows that call tools or update data.

For operations with external side effects, use application-level idempotency: assign a stable operation ID, store its status, and make tool execution safe to repeat. Do not assume a model API supports an idempotency header unless its documentation says so. For expensive generations, consider recording the request and completed result so a client reconnect can retrieve existing work instead of starting over.

5. Treat streaming as a separate case

Before the first streamed token arrives, a transient connection failure may be retryable under your normal policy. After tokens have reached the user, restarting the whole generation can produce duplicated or contradictory text. Decide whether to stop with a clear partial-response message, or restart only when your interface can discard the incomplete output safely. Set a deadline for both initial connection and total stream duration.

6. Add visibility and test failure paths

Log the model, attempt number, elapsed time, status code, and final outcome. Avoid logging API keys or sensitive prompts. Track retry rate, timeout rate, and latency percentiles; a rising retry rate can reveal an upstream issue before users report it. Test simulated 429, 503, connection reset, and slow-response cases, including the rule that no attempt starts after the deadline.

Choose an endpoint that fits your budget

Retries make reliability better, but failed or repeated attempts can increase spend. 59API offers pay-as-you-go access to Claude Opus, Sonnet, Haiku, Fable, and GPT models through the base URL https://api.59api.com. It is designed as a low-cost relay with native, official-quality models rather than downgraded substitutes, and works with OpenAI SDKs as well as Claude Code and Codex. Developers can also use its referral rebate. If you are comparing API costs, consider signing up for 59API and testing your workload with bounded retries and measured usage.

मूल्य · दस्तावेज़