LLM API Rate Limits: Retry Without Breaking Apps
Why LLM API Rate Limits Happen
Rate limits protect an API from overload and keep service fair across customers. An LLM provider may limit requests per minute, input tokens per minute, output tokens per minute, concurrent requests, or total spend. A request can fail even when your application has sent only a few messages, because one large prompt may consume many tokens or several workers may call the API at the same time.
Most rate-limit responses use HTTP status 429 Too Many Requests. Treat a 429 as a temporary condition, not a malformed request. Read the response headers and error body when available: providers may return a retry delay, remaining quota, reset time, or a reason such as token exhaustion versus request-rate exhaustion.
Use Exponential Backoff with Jitter
The standard retry pattern is exponential backoff. After the first 429, wait briefly; after each subsequent failure, increase the delay. For example, retry after 1, 2, 4, 8, and 16 seconds. Always prefer a server-provided retry delay over your local calculation.
Add jitter, which is a small random variation in the wait time. Without it, many workers that fail together can retry at exactly the same moment and create another traffic spike. A practical formula is a random value between zero and the current backoff limit. Cap the delay, such as at 30 or 60 seconds, so a single job does not wait indefinitely.
- Retry 429 errors and temporary 5xx server errors.
- Do not retry most 4xx errors, such as invalid API keys, invalid parameters, or unsupported models.
- Set a maximum attempt count, commonly three to six attempts.
- Log the status code, model, request ID, retry count, and final outcome.
Make Retries Safe and Useful
A retry must not accidentally duplicate an action. Text generation is usually safe to retry if the original request did not complete. Tool calls, database writes, purchases, emails, and other side effects need more care. Use an idempotency key where the API supports one, or store a job ID and mark it complete before performing the side effect.
Set both a connection timeout and a total request timeout. A timeout can mean the provider completed the request but your client did not receive the response. For workflows that trigger actions, save request state so a worker can check whether a job already produced a result before sending it again.
Reduce the Number of 429 Responses
Retries help, but preventing bursts is better. Put outgoing model calls behind a queue and set a concurrency limit. Start with a small number of parallel requests, observe 429 rates and latency, then raise concurrency gradually. For batch jobs, spread work over time instead of launching every task at once.
Also control token usage. Trim unnecessary conversation history, summarize old context, set realistic output-token limits, and cache stable results. For example, cache repeated classification prompts or embeddings rather than generating them again for every user. Smaller, well-scoped prompts reduce cost and make token-based limits easier to stay under.
Choosing a Practical API Gateway
A compatible relay can simplify provider changes and keep costs predictable. 59API offers pay-as-you-go access to native official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, plus GPT models through https://api.59api.com. It works with Claude Code, Codex, and OpenAI SDK-compatible clients, so many applications only need a base URL and API key change rather than a rewrite.
For cost-sensitive prototypes, agents, and production batch workloads, 59API is a low-cost option without using downgraded models. Its referral rebate can also reduce ongoing spend. Sign up for 59API when you need an economical compatible endpoint, then add queueing, capped exponential backoff, jitter, and clear monitoring before increasing traffic.