← सभी गाइड पर लौटें

Cut LLM Retry Waste: Timeouts and Backoff Math

Why Retry Logic Is an LLM Cost-Control Problem

Retries improve reliability, but uncontrolled retries can quietly multiply LLM spend. Unlike many lightweight APIs, an LLM request may generate hundreds or thousands of tokens before a connection drops, a client timeout fires, or a gateway returns a temporary error. If your application immediately resends the full prompt three times, it may pay for substantially more input and output tokens while delivering only one useful answer.

Use a simple planning formula: expected request cost = base request cost × average attempts per completed task. If 10,000 tasks cost $0.02 each at one attempt, the baseline is $200. An average of 1.20 attempts raises that to $240. Reducing the average to 1.05 attempts saves $30 per 10,000 tasks, before accounting for lower infrastructure and support costs.

Set Separate Connection, First-Token, and Total Timeouts

A single 30-second timeout is usually too blunt. LLM calls have different stages, and each needs a realistic budget. Set a connection timeout of 3 to 5 seconds, a time-to-first-token timeout of 15 to 30 seconds for interactive requests, and a total timeout based on expected output length.

For example, if a model typically produces 800 tokens at 40 tokens per second, generation alone may take about 20 seconds. Add 10 seconds for queueing and prompt processing, then set a 35- to 45-second total timeout. For a 4,000-token research response, a 120-second total timeout is more reasonable. A timeout that is too short creates avoidable duplicate calls; one that is too long ties up workers and delays fallback decisions.

Track p50, p95, and p99 latency by model, prompt size, and requested output size. Set interactive timeouts near the p99 of successful requests plus a small buffer, not at an arbitrary round number. Revisit these values after model or workload changes.

Retry Only Errors That Can Recover

Retry transient failures, including network resets, HTTP 408, HTTP 429, and most HTTP 500, 502, 503, and 504 responses. Do not retry authentication failures, malformed requests, unsupported parameters, content-policy blocks, or requests that exceed a known context limit. Those errors need a code or prompt fix, not more API calls.

• Attempt 1: send the request normally.
• Attempt 2: retry only eligible failures after a delay.
• Attempt 3: use a longer delay or a cheaper fallback model if quality requirements allow.
• After 3 attempts: return a graceful error, queue the job, or route it for later processing.

For many production workloads, cap retries at two additional attempts. A three-attempt maximum limits worst-case API exposure to roughly 3× the original call, rather than allowing a temporary outage to create an expensive retry storm.

Use Exponential Backoff With Jitter

Exponential backoff prevents thousands of clients from retrying at the same moment. A practical schedule is 1 second, 2 seconds, and 4 seconds, with random jitter of up to 500 milliseconds. In other words, the second attempt waits roughly 1.0 to 1.5 seconds, the third waits 2.0 to 2.5 seconds, and so on. For HTTP 429 responses, honor the server’s Retry-After header when available.

Do not retry immediately after a rate limit. If 200 workers all retry instantly, they can extend the outage and waste requests. Add concurrency limits as well: if your application can process 100 jobs at once, reserve only 70 to 80 concurrent slots for a provider during elevated error rates.

Prevent Duplicate Work Before It Starts

Idempotency is essential for actions such as sending emails, creating tickets, or updating records from model output. Store a request ID and task state before calling the model. If the client times out but the server completed the request, your application can check the task state instead of blindly generating the answer again.

Also separate generation from side effects. First obtain and validate the model response; then execute the action exactly once. This design avoids the classic failure mode where a retry sends two customer emails or creates two database entries.

Choose a Cost-Efficient, Compatible API Path

Model routing works best when it is paired with disciplined retries. Use a fast, lower-cost model for classification, extraction, and simple rewrites, then reserve premium models for difficult reasoning or high-value final output. For example, routing 80% of 50,000 monthly tasks to an economical model can make a larger difference than shaving a few milliseconds from every retry delay.

59API is a practical low-cost option for teams that want native official-quality Claude and GPT model access without changing their existing integration pattern. Its pay-as-you-go relay supports Claude models including Opus, Sonnet, Haiku, and Fable, works with Claude Code, Codex, and OpenAI SDK-compatible clients, and uses the API base URL https://api.59api.com. That compatibility makes it easier to keep one retry, timeout, and observability layer while routing requests efficiently. It also offers a referral rebate, which can further improve effective costs for eligible users.

Start by measuring attempts per successful task, timeout rate, and token spend by error code for one week. Then set explicit budgets: fewer than 1.08 average attempts, under 1% timeout-driven duplicates, and a hard maximum of three total attempts. If you want a compatible, pay-as-you-go endpoint for implementing this plan, consider signing up for 59API and testing the configuration against your real workload.

मूल्य · दस्तावेज़