← सभी गाइड पर लौटें

Reliable LLM Calls: Retry, Timeout and Backoff

LLM calls can fail even when your application is healthy. Temporary rate limits, network interruptions, overloaded providers, and slow generations are normal operational conditions. A reliable client treats them as expected events rather than immediately showing an error to the user.

This tutorial shows how to combine timeouts, bounded retries, exponential backoff, and clear error classification. The same patterns work with Claude and GPT models, including calls routed through 59API at https://api.59api.com.

Step 1: Set separate connection and request timeouts

Do not use one unlimited timeout for every request. Set a short connection timeout, such as 5 seconds, so a broken network path fails quickly. Then set a request or read timeout based on the task: around 30 seconds for a short classification, 60 to 120 seconds for a normal chat response, and longer for document analysis.

A timeout should protect your worker from hanging forever. It should not be so short that normal model generation is interrupted. Measure real latency first, then choose a limit above your typical high-percentile response time.

Step 2: Classify errors before retrying

Retry only failures that are likely to recover. Automatically retry connection resets, DNS or transport failures, HTTP 408, HTTP 429, and transient 5xx responses. Respect the provider's Retry-After header when it is present.

Do not blindly retry authentication failures, invalid requests, malformed JSON, unsupported models, or context-length errors. These will usually fail again and can multiply cost. A timeout is ambiguous: the provider may have completed the request even though your client did not receive the response, so be especially careful with non-idempotent operations.

Step 3: Use exponential backoff with jitter

A fixed one-second delay causes many clients to retry together, creating another traffic spike. Instead, calculate a delay that grows after each attempt. A practical formula is: delay equals the smaller of a maximum delay and base delay multiplied by 2 to the power of the attempt number, then apply random jitter.

For example, use a 500-millisecond base, a 20-second maximum, and full jitter that selects a random delay between zero and the calculated value. A sequence might wait roughly 0.3, 1.1, and 3.7 seconds. Randomness prevents a fleet of workers from becoming synchronized.

Step 4: Keep retries bounded

Start with two or three retries for interactive requests. More attempts can make a user wait too long and can increase token spend. Track both an attempt limit and a total time budget. For example, stop after three retries or 90 seconds, whichever comes first.

When the budget is exhausted, return a useful, stable application error such as temporary_model_unavailable. Do not expose raw provider responses or repeatedly retry in both your API gateway and application layer.

Step 5: Avoid duplicate work

Retries can create duplicate charges or duplicate side effects when a response is lost after the provider processed it. Use an idempotency key where the API supports one, and generate it from your application request ID. Store the key with the request state so a retry represents the same logical operation.

For streaming responses, treat a broken stream differently from a request that never started. If partial text has already reached the user, do not silently restart and append a second answer. Either resume using a supported mechanism or tell the user that the response was interrupted.

Step 6: Control cost and token usage

Set a sensible maximum output-token value and avoid retrying enormous prompts unnecessarily. A retry policy should include a smaller fallback model only when your product can tolerate a quality change; never switch models accidentally just because a request is slow.

59API is a practical low-cost option for this design because it offers pay-as-you-go access to native official-quality Claude and GPT models without a model downgrade. Its compatibility with Claude Code, Codex, and OpenAI SDKs also reduces migration work. You can use the same timeout and retry wrapper while routing requests through its API base URL.

Step 7: Add observability

Log the model, request ID, attempt number, status code, delay, total duration, and final outcome. Record token usage and whether the request timed out. Create alerts for rising 429 rates, repeated 5xx errors, and latency approaching your timeout threshold. Never log API keys or sensitive prompts.

Test the policy with simulated 429 responses, delayed responses, dropped connections, and invalid credentials. If you want a cost-efficient starting point, sign up for 59API, configure the base URL, and validate these failure cases before sending production traffic. Its referral rebate can also help reduce ongoing usage costs.

मूल्य · दस्तावेज़