← सभी गाइड पर लौटें

LLM Retry, Timeout and Backoff: A 15-Minute Setup

A reliable LLM integration needs more than “retry three times.” Without deadlines and backoff, one slow request can trigger duplicate generations, unexpected charges, and a queue of frustrated users. This quick-start guide gives you a practical baseline for retry, timeout and backoff patterns for LLM calls.

1. Set a deadline before adding retries

Start with the maximum time your application can spend on one user operation, including every attempt and wait. For short, non-streaming responses, try these initial settings, then tune them using production latency:

• End-to-end deadline: 60 seconds.
• Connection timeout: 5 seconds.
• Read timeout: 25 seconds of inactivity.
• Maximum attempts: 3 total, including the first request.

A read timeout usually measures inactivity, not total request duration; check your HTTP client's semantics. Enforce the overall deadline separately with cancellation. Before each attempt, calculate the remaining budget and cap that attempt's duration accordingly. Long outputs and reasoning-heavy requests may need larger budgets.

2. Retry transient failures, not every error

Classify the failure before scheduling another call:

• Usually retry: temporary connection failures, connection resets, and transient HTTP 500, 502, 503 or 504 responses.
• Conditionally retry: HTTP 429 when it indicates temporary rate limiting. Exhausted credits or a hard quota require intervention, not repeated requests.
• Usually stop: invalid requests, authentication failures, permission errors, unsupported parameters and context-length errors.

A timeout is ambiguous: the server might still be generating after your client stops waiting. Retrying can therefore incur another generation charge. Invalid structured output is different again; a bounded repair attempt may help, but repeating the identical prompt is not a transport-retry strategy.

3. Use exponential backoff with full jitter

For retry number n, starting at zero after the first failed attempt, calculate a delay ceiling as min(8 seconds, 0.5 seconds × 2^n). Then choose a random delay uniformly between zero and that ceiling. Your first two retry waits therefore range from 0–0.5 seconds and 0–1 second.

This is full jitter. It spreads retry traffic rather than making every worker reconnect simultaneously. If a response includes Retry-After, parse either its seconds value or HTTP date and do not retry before that time. Wait at least the greater of your jitter delay and the server's requested delay.

If that wait leaves insufficient time for a useful attempt, stop or schedule background work instead. Do not shorten the server's requested wait merely to squeeze another attempt into your deadline.

4. Keep one owner for retries

Check whether your SDK already retries requests. Three application attempts wrapped around three SDK attempts can produce nine upstream calls. Either let the SDK manage retries within your deadline or disable its retries and centralize the policy yourself.

The control flow should be simple: check remaining time, make one bounded attempt, classify any failure, check the attempt limit, calculate the wait, and retry only if the budget permits. Propagate cancellation when the user disconnects, while recognizing that cancellation may not stop upstream computation.

5. Protect streaming and side effects

Once a stream has delivered tokens, do not silently append a restarted response: it can repeat or contradict earlier text. Mark the stream as interrupted and offer an explicit restart. A read timeout may also fire during a long gap between chunks, so tune streaming inactivity limits separately.

For tool calls that send emails, create orders or modify data, deduplicate execution using an application operation ID. Do not assume the model endpoint supports idempotency keys unless its documentation explicitly says so.

6. Apply the policy through 59API

59API offers low-cost, pay-as-you-go access to native, official-quality models without downgrades, with OpenAI SDK compatibility and an API base URL of https://api.59api.com. Benchmark models such as GPT-6.1 Sol and Gemini 2.5 Flash separately: latency and output length affect suitable deadlines.

Log model, attempt count, status, latency, backoff time and available request IDs without recording secrets. Track retry success and spend per successful operation; low token prices cannot compensate for uncontrolled duplicate calls. Sign up for 59API when you are ready to test this policy with a small, representative workload.

मूल्य · दस्तावेज़