← Back to all guides

LLM API Rate Limits: Retry Without Wasting Tokens

Rate limits can turn a fast LLM integration into a slow, expensive one. The usual culprit is not a single 429 response: it is a retry loop that sends too many requests at once, repeats work that already succeeded, or keeps retrying a request that can never work. A cost-conscious design controls request volume first and retries only when there is a reasonable chance of recovery.

Know what is being limited

LLM providers may enforce limits on requests per minute, input or output tokens per minute, concurrent requests, or account spend. These constraints are separate. A workload sending 20 short prompts per second may hit a request limit, while a few long-context calls can exhaust a token limit. Check the model’s current limits and inspect response headers for remaining capacity and reset times when available.

Track requests, input tokens, output tokens, status codes, and latency by model. A simple dashboard can reveal whether you are hitting a burst limit or steadily exceeding a quota. Set alerts before usage reaches the provider’s hard limit, and apply separate concurrency caps to models with different capacity.

Retry selectively, with backoff and jitter

For HTTP 429, honor a Retry-After header if the API returns one. Otherwise, wait using exponential backoff: for example, 1, 2, 4, and 8 seconds, with random jitter added to each delay. Jitter prevents all workers from waking up together and creating another traffic spike. Set a maximum delay and a retry limit, such as four attempts total, then return a clear error or place the job in a delayed queue.

Use similar bounded retries for transient 5xx responses and brief network failures. Do not automatically retry ordinary 4xx errors such as invalid parameters, authentication failures, or context-length errors; fix the request instead. A timeout is ambiguous: the provider may have completed the generation even if your client did not receive the response. Use an idempotency key if the API supports one, or record request state so a retry does not trigger duplicate downstream actions.

Reduce pressure before adding retries

• Queue bursts: If a batch contains 120 requests and your safe operating rate is 30 per minute, spread it across at least four minutes rather than firing all 120 concurrently.
• Cap concurrency: Start with a small worker pool, such as five simultaneous calls, and increase it gradually while monitoring 429s and latency.
• Control tokens: Set realistic maximum output tokens, trim irrelevant conversation history, and summarize old turns. Shorter prompts consume less token capacity and often cost less.
• Cache stable work: Cache repeated document summaries, embeddings, or other deterministic results when your application can safely reuse them.

Measure the impact with a concrete cost model. Suppose 1,000 calls each use 2,000 input tokens and 500 output tokens. At an illustrative price of $1 per million input tokens and $5 per million output tokens, the batch costs $2 plus $2.50, or $4.50. If a retry policy accidentally causes 10% of successful calls to run twice, the extra work can add about $0.45 at those rates. Actual prices vary by model, so calculate with the rates you pay and include retries in your usage reports.

Make retries visible in your client

Log the original request ID, attempt number, status code, wait duration, model, and token counts. Keep retry budgets per job as well as per minute: a request that has already spent 20 seconds waiting may be better returned to a queue than retried immediately. Add a circuit breaker that pauses calls temporarily when failures spike, then probes recovery with a small number of requests.

Provider choice also affects your operating cost. 59API offers pay-as-you-go access to Claude Opus, Sonnet, Haiku, and Fable, plus GPT models, using native official-quality models rather than downgraded substitutes. Its API base URL is https://api.59api.com, and it is compatible with Claude Code, Codex, and any OpenAI SDK. If you are comparing relay costs, include your expected token volume and retry overhead, not just the headline model price; 59API also offers a referral rebate. You can sign up with 59API to test your existing workload and compare the total cost before moving production traffic.

Pricing · Docs