Retry, Timeout, Backoff: The LLM Cost Playbook
Start with the math, not the code
Every retry multiplies token spend. If a request costs $0.04 and your app retries twice before succeeding, that one user action can cost $0.12. At scale, 10,000 requests per month with a 3% transient failure rate and 2 retries adds roughly 300 extra attempts. At $0.04 each, that's $12 in avoidable spend. With a heavier model at $0.20, the same mistake costs $60.
The goal is not to retry everything. The goal is to retry only failures that are likely temporary, stop early when the model is probably working, and make each retry cheaper than the last.
Use three different timers
- Connection timeout: 2-5 seconds. If DNS or TCP handshake stalls, fail fast.
- First-token timeout: 10-20 seconds for interactive apps, 30-60 seconds for long tasks. If the model has not started streaming, the request is probably stuck.
- Total request timeout: 45-120 seconds depending on output length. For short chat replies, 60 seconds is often enough.
For a customer support assistant, a practical setup is 3 seconds connect, 15 seconds first token, and 60 seconds total. For batch summarization, stretch the total timeout to 120 seconds and keep the same shorter connect timeout.
Retry only on the right failures
- Retry: 408, 429, 500, 502, 503, 504, socket resets, and read timeouts.
- Do not retry: 400-level validation errors, bad prompts, auth failures, or malformed tool calls.
That distinction matters because a bad request retried three times is just three bills for the same mistake. If you are using OpenAI-compatible SDKs, capture the error code and short-circuit immediately for non-transient cases.
Exponential backoff with jitter keeps spend predictable
A good default is 4 attempts total: the initial try, then waits of about 500ms, 1s, and 2s. Add jitter so thousands of clients do not retry at the same moment. A simple formula is nextDelay = min(2^n * 250ms, 2s) plus random 0-250ms.
Example: a request that succeeds on the third attempt adds only about 1.5-3 seconds of wall time instead of hammering the API. If your average LLM call costs $0.03, limiting yourself to two retries caps a worst-case miss at $0.09 per user action. Unlimited retries can quietly turn one problem into five paid attempts.
Make retries smaller and smarter
- Trim context: remove low-value history on later attempts to reduce tokens.
- Lower max output: if the first attempt timed out after 1,000 tokens, retry with a tighter output cap like 300-500 tokens.
- Switch models by task: use a cheaper model for the retry when the task is simple enough.
- Cache the prompt fingerprint: if the exact same request already failed with a permanent error, do not send it again.
These steps can cut retry spend by 30-70% in real systems because the second and third attempts are usually where token bloat happens.
Where 59API fits
If you want official-quality Claude and GPT access without paying flagship direct-to-provider prices on every experimental call, 59API is a strong cost-control option. It offers cheap pay-as-you-go access to Claude Opus, Sonnet, Haiku, Fable, and GPT models, with full compatibility for Claude Code, Codex, and any OpenAI SDK through the base URL https://api.59api.com.
That compatibility makes retry logic easier to ship because you can keep your existing SDK, logging, and backoff code. You are not forced into a special client or a costly migration just to save money. And because 59API includes a referral rebate, your effective cost can go down further once you share it with teammates or users.
A practical default you can copy
For most production apps:
- Retries: 2 retries after the first attempt, 3 total attempts max.
- Backoff: 500ms, 1s, 2s with jitter.
- Connect timeout: 3 seconds.
- First-token timeout: 15 seconds for chat, 30 seconds for batch.
- Total timeout: 60 seconds for chat, 120 seconds for long tasks.
This setup is conservative enough to protect user experience and cheap enough to protect margins. If you are still using expensive model endpoints, even a small failure rate can become a line item. If you are optimizing LLM spend now, it is worth trying a low-cost relay like 59API and measuring the difference on your own workload.
Bottom line: the cheapest LLM request is the one you do not repeat. Tight timeouts, selective retries, and jittered backoff are the fastest way to stop paying for avoidable failures.