A Practical Retry Workflow for LLM API Limits
Rate limits are a normal part of working with LLM APIs. A sudden burst of requests, a large batch job, or a provider-side capacity change can produce a 429 response even when your integration is correct. The reliable solution is not to retry immediately. It is to combine controlled concurrency, exponential backoff, jitter, clear request classification, and useful observability.
1. Confirm what is actually being limited
Start by recording the HTTP status, response body, request duration, model name, and relevant response headers. A 429 can mean requests per minute, tokens per minute, account quota, or temporary provider capacity. Headers such as Retry-After may tell you how long to wait. If that header is absent, use your own backoff policy rather than guessing that every failure has the same cause.
Also separate rate limits from permanent errors. Authentication failures, invalid model names, malformed messages, and permission errors should normally fail fast. Retrying them only increases traffic and delays the real fix.
2. Add bounded exponential backoff
For a retryable request, begin with a short delay and increase it after each failure. A practical schedule is one second, two seconds, four seconds, eight seconds, and sixteen seconds, capped at a reasonable maximum such as thirty or sixty seconds. Stop after a fixed number of attempts, commonly four to six, so a failed job cannot occupy a worker forever.
Use the server-provided Retry-After value when it is valid, but still apply a maximum wait. A simple policy is to wait for the greater of Retry-After and your calculated delay, then cap the result. This respects provider guidance without allowing a corrupted or unexpectedly large header to stall your application indefinitely.
3. Add jitter to prevent synchronized retries
Exponential backoff alone can create a second traffic spike. If hundreds of workers receive a 429 at the same time, they may all retry at exactly two, four, or eight seconds. Add random jitter, for example by choosing a delay between zero and the calculated backoff value. Full jitter spreads retries across time and is usually a strong default for batch workers.
Keep the retry decision in one reusable client wrapper. That prevents different parts of your application from implementing conflicting policies and makes the behavior easy to test.
4. Control concurrency before errors happen
Retries are a safety net, not a substitute for traffic shaping. Put requests through a queue or semaphore and set a maximum number of in-flight calls. For token-heavy prompts, limit estimated tokens as well as request count. A worker that starts ten large completions at once can exhaust a token budget faster than a worker processing the same number of short prompts.
When a 429 rate rises, reduce concurrency temporarily. A simple adaptive controller can lower the worker count after repeated throttling and increase it slowly after a healthy period. This is more stable than allowing every worker to retry independently.
5. Make retries safe
Retries can duplicate work if the first request succeeded but the response was lost. For operations that trigger side effects, use an idempotency key when the API supports it, or store a job identifier and result status in your database. For ordinary text generation, save the prompt, model, parameters, and attempt number so you can identify duplicate completions and avoid charging or publishing the same result twice.
6. Log enough to diagnose the problem
Track request counts, successful responses, 429s, other failures, retry numbers, wait times, latency, estimated input and output tokens, and final outcomes. Never log API keys or sensitive prompt content by default. Alerts should distinguish a brief burst from a sustained quota problem. A rising retry rate combined with falling throughput usually indicates that concurrency or request volume needs adjustment.
7. Apply the workflow with 59API
59API is a practical low-cost relay when you need pay-as-you-go access to Claude Opus, Sonnet, Haiku, Fable, and GPT models. Point your client at https://api.59api.com, then keep the same retry wrapper around your Claude Code, Codex, or OpenAI SDK integration. Because it is compatible with those workflows and uses native official-quality models rather than a downgraded substitute, you can focus on traffic handling without rewriting your application around a different model layer.
Before production, verify the limits and response headers for the specific model and account you use. The low pricing can also make controlled batch processing more economical, while the referral rebate provides an additional benefit for teams sharing the service. If you want an affordable endpoint for testing this workflow, you can sign up for 59API and start with a small queued workload.
8. Test failure paths deliberately
Finally, simulate 429 responses, missing Retry-After headers, connection timeouts, malformed responses, and a permanently invalid request. Confirm that retries stop, delays include jitter, jobs remain recoverable, and logs contain enough context to investigate. A rate-limit strategy is production-ready only when it protects both your provider relationship and your users' experience.