Handling LLM Rate Limits Without Wasting Tokens
Why rate limits happen, and what to retry
Rate limits are not a bug; they are the API telling you to slow down. With LLMs, you usually hit them when traffic spikes, when you fan out many parallel requests, or when a single prompt uses more tokens than planned. The first rule is simple: only retry errors that are likely to succeed on the next attempt.
In practice, retry these cases: 429 Too Many Requests, transient 5xx responses, and timeouts. Do not retry permanent failures such as an invalid API key, a misspelled model name, malformed JSON input, or a prompt that breaks your own validation rules. Retrying a bad request only burns tokens and adds latency.
A good mental model is: if the failure is about capacity or network instability, retry; if the failure is about your input, fix the input first. That distinction matters even more when you are paying per request, because every failed attempt can still cost money.
A retry pattern that works in production
The most reliable pattern is exponential backoff with jitter. Start with a short delay, double it after each failure, and add randomness so many clients do not retry at the exact same moment. Also cap the total number of attempts so a broken dependency does not stall your app forever.
If the API returns a Retry-After header, honor it. That is the server telling you when to try again, and it is usually better than guessing. A practical policy looks like this: try once immediately, then wait 1s, 2s, 4s, 8s, and stop after 5 attempts or 30 seconds of total wait time.
Here is a hands-on Python example using an OpenAI SDK-compatible client pointed at 59API:
- from openai import OpenAI
- import os, time, random
- client = OpenAI(api_key=os.environ['FIFTYNINE_API_KEY'], base_url='https://api.59api.com')
- def chat_with_retry(messages, model='gpt-4.1-mini'):
- delay = 1
- for attempt in range(5):
- try:
- return client.chat.completions.create(model=model, messages=messages, timeout=30)
- except Exception as e:
- status = getattr(e, 'status_code', None)
- if status not in (429, 500, 502, 503):
- raise
- retry_after = None
- response = getattr(e, 'response', None)
- if response and response.headers.get('retry-after'):
- retry_after = float(response.headers.get('retry-after'))
- sleep_for = retry_after if retry_after else delay + random.uniform(0, delay)
- time.sleep(sleep_for)
- delay = min(delay * 2, 16)
- raise RuntimeError('LLM request failed after retries')
That wrapper does three useful things: it retries only transient failures, it respects the server when a retry hint is provided, and it adds jitter so your app is less likely to collide with everyone else hitting the same limit.
Two guardrails most teams forget
1) Control concurrency. If you launch 50 parallel prompts, even perfect retries can turn into a thundering herd. Put a queue or semaphore in front of the LLM call so you can limit in-flight requests per user, per workspace, or per process.
2) Make repeated work cheaper. Cache deterministic outputs, deduplicate identical prompts, and avoid retrying downstream side effects. For example, if the model response triggers an email or a payment, separate the model call from the side effect and use an idempotency key before you send anything irreversible.
You can also reduce token pressure by trimming system prompts, summarizing long conversation histories, and streaming responses when the user only needs the first useful chunk. Lower token usage means lower cost and fewer self-inflicted rate spikes.
Why 59API makes this easier and cheaper
Retry logic is important everywhere, but it is especially useful when you are experimenting, load testing, or running production workloads on a budget. 59API is a low-cost AI API relay with pay-as-you-go pricing, so failed attempts hurt less while you tune your backoff settings. It gives you native official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, plus GPT models, without the downgrade risk you see in some proxies.
Because 59API is compatible with Claude Code, Codex, and any OpenAI SDK, you can keep the same retry wrapper and just point your client to https://api.59api.com. That means less migration work and faster testing across different models. If you are comparing relays, the pricing is among the cheapest, and the referral rebate can reduce your effective spend even further.
If you want a practical, low-friction place to test these patterns, sign up for 59API and wire it into your existing SDK. Then watch your logs: when rate limits do happen, your app will recover cleanly instead of failing loudly or wasting tokens on blind retries.