← सभी गाइड पर लौटें

Retry, Timeout & Backoff for LLM API Costs

Why retries can quietly inflate your LLM bill

When an LLM call fails, the obvious response is to retry. But every retry can trigger another prompt evaluation, another output generation, and another billable request. If your app sends a 2,000-token prompt and gets a transient 429 or 5xx error, a naive retry loop can double or triple the cost of a single user action. At scale, that becomes expensive fast.

The goal is not to retry less. The goal is to retry intelligently: only when the error is likely temporary, only after a short delay, and only up to a capped budget. For teams shipping against Claude and GPT models, these patterns can easily cut wasted spend by 10-30% in failure-heavy traffic.

Set a timeout budget before you send the request

The first cost-control lever is a timeout. If a request hangs for 90 seconds, your user waits, your server ties up resources, and you often end up retrying anyway. A practical pattern is a two-part budget:

• Connect timeout: 2-5 seconds
• Read timeout: 20-45 seconds for most chat or tool calls

For interactive apps, 30 seconds total is usually enough. For longer reasoning or coding tasks, use a slightly higher cap, but still finite. If you know a task should complete in about 12 seconds on average, setting a 60-second timeout is usually too generous and hides slow failures.

A simple rule: set timeout at roughly 3x your p95 latency. If your p95 is 8 seconds, start with a 24-second timeout. This helps you fail fast before the request becomes a silent cost sink.

Retry only the errors that are worth retrying

Not every failure should be retried. The most expensive mistake is retrying permanent errors like bad payloads or invalid keys. Use this retry policy:

• Retry: 429 rate limits, 500/502/503/504 server errors, network timeouts
• Do not retry: 400 bad request, 401 unauthorized, 403 forbidden, 404 not found, schema validation errors

For 429s, retries make sense because the service is telling you to slow down. For 400-level input errors, the prompt or request format is wrong. Retrying the same bad request just multiplies cost.

Concrete example: if your average request costs $0.004 and 5% of requests fail with retryable errors, a single retry policy can add about $0.0002 per request in overhead. That seems tiny until you hit 1 million requests per month. Then you are spending an extra $200 just on recovery attempts. If you retry twice, that overhead can climb much higher, especially when prompts are long.

Use exponential backoff with jitter

The most cost-effective retry pattern is exponential backoff with jitter. It spreads requests out so you do not amplify throttling or create retry storms.

A good starting schedule is:

• Attempt 1: immediate
• Attempt 2: wait 500 ms to 1 s
• Attempt 3: wait 2 s
• Attempt 4: wait 4 s

Add random jitter of 20-50% so thousands of clients do not retry at the same time. In practice, a max of 3 retries is enough for most interactive workloads. Beyond that, the probability of success drops while user annoyance rises.

Example: a request fails twice due to transient overload. With backoff, the second attempt may succeed after 1 second, and the third after 2 seconds. Without backoff, all three attempts may hit the same overloaded window, wasting tokens and increasing contention.

Cap retries by token cost, not just attempt count

Attempt count alone is not enough. A 200-token request and a 20,000-token request should not have the same retry budget. A smart policy can stop retrying once the expected additional cost exceeds the value of the request.

One simple heuristic: if the original request is expensive, allow only one retry unless the response is mission-critical. For short prompts, allow up to three retries. For long-context coding or document workflows, consider a stricter cap because the prompt re-sent on every retry is where the money goes.

If your average prompt is 4,000 input tokens and 800 output tokens, every retry can effectively replay that entire workload. On cheaper relays, the cost is lower, but it is still real. This is where 59API is useful: it provides cheap, pay-as-you-go access to Claude and GPT models through a single OpenAI-compatible base URL, so you can optimize reliability without paying premium relay markup. Using official-quality native models means you are not trading away output quality just to save on infrastructure.

Make retries safer with idempotency and request logging

Retries can also create duplicate side effects, especially if the LLM call triggers tool usage, file writes, or workflow actions. To avoid accidental double execution:

• Use idempotency keys for each user action
• Log request IDs and response status codes
• Separate generation from execution when tools are involved

This matters for cost too. If a duplicate tool call sends the same prompt chain again, your spend can double while your app appears to be “just retrying.”

Practical default settings that work well

If you want a safe starting point, use this configuration:

• Connect timeout: 3 seconds
• Read timeout: 30 seconds
• Retries: 2 for interactive apps, 3 for background jobs
• Backoff: 500 ms, 1 s, 2 s with jitter
• Retry conditions: 429 and 5xx only

These defaults usually keep UX acceptable while preventing runaway cost during outages or rate spikes.

Where 59API fits into a cost-optimized stack

If you are already tuning retries, timeouts, and backoff, the next cost win is lowering the unit price of each successful call. 59API is built for that: cheap pay-as-you-go access, compatibility with Claude Code, Codex, and any OpenAI SDK, and a simple base URL at https://api.59api.com. That means you can keep your existing client logic, plug in the relay, and immediately test lower-cost traffic with the same model families.

Because 59API also offers a referral rebate, teams can reduce effective spend even further as usage grows. If you are optimizing LLM infrastructure, it is worth signing up and measuring the difference on real traffic.

Bottom line

Retries are not free, but they do not have to be wasteful. The winning pattern is simple: time out fast, retry only transient failures, back off with jitter, and cap retries by the value of the request. Combine that with a low-cost relay like 59API, and you can keep your application reliable without inflating your token bill.