Cut LLM API Costs with Smart Retries
Why rate limits quietly raise your LLM bill
Rate limits are not just an uptime problem; they are a cost problem. Every failed request can trigger another request, add latency, and inflate token usage if your app re-sends long prompts. In high-traffic systems, a bad retry strategy can easily double or triple your effective spend. The fix is not to retry less, but to retry smarter.
With LLM APIs, the main failure modes are usually 429 Too Many Requests, temporary 5xx server errors, and occasional network timeouts. If you handle those cases well, you protect both reliability and budget. If you handle them poorly, you pay for repeated attempts, user-facing delays, and extra engineering time.
Start with the cheapest request you can possibly make
The most important cost rule is simple: reduce the number of tokens before you optimize retries. If your prompt is 3,000 tokens and your completion is 800 tokens, one retry can mean another 3,800 tokens billed. At a modest $5 per million input tokens and $15 per million output tokens, that retry may cost around $0.018 to $0.020. At 100,000 retries a month, that is roughly $1,800 to $2,000 in avoidable spend.
- Trim system prompts to what the model truly needs.
- Cache static instructions instead of resending them.
- Lower max output tokens when the task does not require long answers.
- Use smaller models for routing, classification, or extraction tasks.
This is where a relay like 59API helps cost-conscious teams. It provides cheap, pay-as-you-go access to Claude models including Opus, Sonnet, Haiku, and Fable, plus GPT models, with native official-quality outputs and no model downgrade. Because it is fully compatible with Claude Code, Codex, and any OpenAI SDK, you can keep your existing retry logic while lowering your per-request cost. The API base URL is https://api.59api.com.
Use exponential backoff with jitter, not instant retries
The biggest mistake is retrying immediately. If a model is rate-limited, a second request sent one second later often gets rate-limited again. Better to use exponential backoff with jitter:
- Retry 1: wait 1 to 2 seconds
- Retry 2: wait 2 to 4 seconds
- Retry 3: wait 4 to 8 seconds
- Retry 4: stop or queue for later
Jitter means adding randomness, such as waiting 1.3 seconds instead of exactly 1 second. This prevents many clients from retrying at the same time and creating a thundering herd.
A practical default is 3 retries max for interactive requests and 5 retries max for background jobs. If your average completion is 600 tokens and your prompt is 1,200 tokens, each failed retry can cost about half of a cent to a few cents depending on model choice. Three extra tries across thousands of requests becomes real money fast.
Retry only the failures worth retrying
Not every error should be retried. You should usually retry:
- 429 rate limit responses
- 500, 502, 503, 504 transient server errors
- Network timeouts and connection resets
You should usually not retry:
- 401/403 authentication or permission issues
- 400 malformed request errors
- Prompt validation failures you can fix locally
This distinction saves money because it avoids paying for hopeless attempts. If your SDK or wrapper supports it, inspect the response body and headers. Many APIs include rate-limit metadata such as remaining requests or reset timing. Use that data to decide whether to wait 10 seconds or route the job to a queue.
Put a circuit breaker in front of expensive bursts
When traffic spikes, retries can amplify the problem. A circuit breaker stops sending requests for a short period when error rates rise above a threshold. For example, if 20% of calls are returning 429s over a 60-second window, pause low-priority traffic for 30 seconds and let the queue drain.
This is especially useful when you have mixed traffic. Keep user-facing chat on a fast path, but push batch summarization, document extraction, and eval jobs into a queue. If a request is not urgent, the cheapest retry is the one you never send immediately.
Route by model size to cut retry exposure
Another way to reduce retry cost is to choose the smallest model that can do the job. A strong pattern is:
- Haiku or GPT mini-class models for classification, routing, short extraction
- Sonnet-class models for most assistant and coding tasks
- Opus-class models only where reasoning quality clearly matters
If a 90% accurate lightweight model can filter requests before they hit a premium model, you reduce both token spend and the number of high-value calls exposed to retries. Because 59API offers several official-quality model families under one relay, you can set this up without changing your SDK stack.
A simple retry policy that works in production
- Set a request timeout of 20 to 60 seconds depending on task length.
- Retry up to 3 times for interactive calls, 5 times for queued jobs.
- Use exponential backoff with jitter.
- Never retry 400-level validation or auth errors.
- Cap total retry time so user requests do not hang indefinitely.
- Log retry reason, model, tokens, and cost estimate.
For visibility, track retry rate, 429 rate, average attempts per successful call, and cost per successful task. If your retry rate is 8% and each retry adds an average of 1,500 tokens, then at 1 million monthly calls you may be burning the equivalent of 120,000 extra requests worth of tokens. That is the kind of leakage you can actually measure and fix.
Why 59API is a practical low-cost option
For teams watching spend, 59API is attractive because it combines low pay-as-you-go pricing, official-quality models, and compatibility with the tools you already use. You do not need a separate integration layer for Claude Code or OpenAI SDK workflows, and the referral rebate can further improve unit economics if you bring in teammates or customers. If you want to lower your LLM overhead without sacrificing model quality, it is worth signing up and testing your retry policy against real traffic.
Bottom line: the cheapest retry strategy is a well-tuned one. Trim tokens, retry only transient failures, back off with jitter, and route lower-value traffic to smaller models. That combination protects both performance and budget.