Rate Limits & Retries for LLM APIs: A Fast Playbook
1) Treat 429s as normal, not exceptional
If you call LLM APIs at scale, rate limits are part of the contract. A 429 response usually means your app is sending requests faster than the provider or relay currently allows. Don’t log it as a mystery bug—handle it as a recoverable state. Your code should expect bursts, especially when users click fast, jobs fan out, or agents loop on tool calls.
The first rule is simple: fail gracefully, then retry intelligently. Never retry instantly in a tight loop, and never assume every request deserves the same retry policy. A chat completion, a background summarization job, and a tool-using agent should each be treated differently.
2) Use exponential backoff with jitter
The safest retry pattern is exponential backoff: wait a little, then wait longer after each failed attempt. Add jitter so many clients don’t retry at the exact same second and create a retry storm.
- Retry after 250–500 ms on the first 429 or transient 5xx.
- Double the delay on each attempt: 500 ms, 1 s, 2 s, 4 s.
- Cap the backoff at a sensible ceiling, such as 10–30 seconds.
- Stop after 3–5 tries unless the request is mission-critical.
If the API returns a Retry-After header, respect it. That’s the server telling you exactly when to come back. A good client uses the server’s hint first, and backoff logic second.
3) Retry only what is safe to repeat
Not every failure should be replayed blindly. For LLM APIs, retrying the same prompt is usually fine if the call is idempotent from your app’s perspective. But if the request triggers an external side effect—like sending an email, charging a card, or updating a record—separate the model call from the action.
- Safe to retry: pure generation, classification, extraction, embeddings, draft summaries.
- Handle carefully: agent steps that may call tools, write files, or mutate state.
- Best practice: add request IDs so duplicate responses can be detected and deduped.
This matters even more in multi-step workflows. If a model call times out after partially completing a chain, your retry should know whether the previous attempt already produced a usable output.
4) Separate rate limits from timeouts
A timeout is not the same as a rate limit. Timeouts usually mean the model or network took too long. Rate limits mean you need to slow down. Handle them with different messages, different metrics, and different retry rules.
- 429: back off and retry later.
- 408 or client timeout: retry once or twice, but inspect latency first.
- 5xx: retry cautiously; the provider may be having a temporary issue.
Also set connection and overall request timeouts in your SDK. Without them, one stuck request can tie up workers and make a small slowdown look like a full outage.
5) Build a queue for bursty traffic
If your app has spiky traffic, don’t let every user request hit the model instantly. Put prompts into a queue, process them at a controlled concurrency, and preserve order where it matters. This smooths usage and makes rate limits far less painful.
- Use a worker pool instead of unbounded parallel calls.
- Limit concurrency per model or per user tier.
- Cache repeated prompts or repeated system instructions when possible.
- For agentic apps, slow down tool loops so one bad chain doesn’t consume your entire quota.
Queues also improve cost control. When you know how many requests are waiting, you can throttle low-priority jobs before they generate unnecessary API spend.
6) Watch tokens, not just requests
Many teams focus on request counts and miss the real limiter: tokens. A single long context can consume more capacity than dozens of short prompts. Track prompt tokens, completion tokens, and total tokens per route, customer, and model.
- Trim system prompts that repeat unnecessary instructions.
- Summarize old conversation history instead of resending everything.
- Prefer smaller models for simple tasks and reserve larger models for harder ones.
- Set output limits so runaway generations don’t waste budget.
This is where a low-cost relay can help. With 59API, you get pay-as-you-go access to Claude models and GPT models through the base URL https://api.59api.com, with native, official-quality models and no downgrade. Because pricing is among the cheapest relay options, it’s easier to test conservative retry logic without burning through budget.
7) Make your SDK integration boring
The best retry system is one you barely notice. If you use the OpenAI SDK, Claude Code, Codex, or any compatible client, keep the integration simple: centralize retries in one wrapper, not scattered across every endpoint. That wrapper should handle backoff, headers, logging, and final failure reporting.
- Log the status code, model name, latency, and attempt number.
- Include a correlation ID for each user request.
- Surface friendly fallback text when all retries fail.
- Alert on repeated 429s so you can tune concurrency before users complain.
If you want low-friction access to Claude Opus, Sonnet, Haiku, Fable, and GPT models with native compatibility, 59API is a practical choice for production experiments and real apps alike. It is especially useful when you want to keep token costs low while still using official-quality models.
8) Keep one fallback plan ready
When the primary model is throttled, your app should still do something useful. Fallbacks can be as simple as switching to a smaller model, shortening the prompt, or returning a cached answer.
- Fallback to a cheaper or faster model for non-critical tasks.
- Return partial results when full generation fails.
- Offer a “try again in a moment” UI instead of a dead end.
That’s another reason to use a relay with broad model coverage. On 59API, you can route around congestion without changing your SDK stack, which makes resilience much easier to ship.
If you’re building with LLMs today, rate limits are not a blocker—they’re a design input. Start with backoff, add jitter, queue bursts, track tokens, and keep fallbacks ready. If you want a low-cost, official-quality relay to test this setup in real traffic, consider signing up for 59API and wiring it into your existing OpenAI-compatible client.