← Voltar aos guias

Handle LLM API Rate Limits: Retry Guide

Why rate limits happen with LLM APIs

Rate limits are not a bug; they are a control mechanism. LLM providers cap request volume, tokens per minute, or concurrent traffic so their systems stay stable and fair for everyone. If your app sends bursts of chat requests, long prompts, or parallel jobs, you can hit limits even when your code is correct.

The most common symptoms are HTTP 429 responses, slowed throughput, or a request that works in staging but fails under real traffic. The fix is not to “retry harder.” You need a clear retry strategy, predictable backoff, and a way to reduce unnecessary calls.

Step 1: Identify the exact limit you are hitting

Before changing code, inspect the response. Many LLM APIs return useful headers or error payloads that tell you whether the limit is based on requests, tokens, or concurrency. Log the status code, response body, and any retry-after hint.

• 429 Too Many Requests: you are sending requests faster than allowed.
• Rate limit by tokens: large prompts or long completions are the issue.
• Concurrency cap: too many in-flight requests at once.

If you use OpenAI-compatible SDKs, make sure your error handler reads the provider’s standard error shape rather than assuming every failure is the same. This matters with relays and alternative endpoints as well.

Step 2: Use exponential backoff with jitter

The safest retry pattern for LLM APIs is exponential backoff with jitter. That means waiting a little longer after each retry, and adding randomness so many clients do not retry at the same time.

• First retry: wait 500ms to 1s.
• Second retry: wait 1s to 2s.
• Third retry: wait 2s to 4s.
• Cap retries: usually 3 to 5 attempts is enough.

Do not retry instantly. If the service is overloaded, immediate retries create a traffic spike that makes recovery worse. Also avoid unlimited retries, because they can burn tokens and increase latency without improving success rate.

Step 3: Retry only the right failures

Not every error should be retried. Retry transient failures, not permanent ones.

• Retry: 429, 500, 502, 503, 504, network timeouts.
• Do not retry: invalid API key, malformed request, bad model name, quota exhaustion, or permission errors.

This distinction saves cost and prevents hidden loops. If your prompt is too large for the model context window, retrying the same request will never succeed. Trim the prompt, summarize history, or split the job.

Step 4: Make retries safe with idempotency

In production, a request may succeed on the server but time out on the client. If you retry blindly, you can duplicate side effects such as logging, billing actions, or workflow steps. To avoid that, use idempotency keys or your own request IDs where supported.

If the API does not expose native idempotency, store a unique job identifier in your application and deduplicate responses on your side. For streaming chat, persist the last known state so a retry resumes cleanly instead of starting from scratch.

Step 5: Control concurrency and queue bursts

Many rate limit problems are really queueing problems. If your app launches 50 parallel requests at once, one user action can overwhelm the quota.

• Limit concurrent calls: use a worker pool or semaphore.
• Queue background jobs: smooth traffic instead of spiking it.
• Batch when possible: combine small tasks into one request.
• Cache repeated prompts: avoid paying twice for the same answer.

For long-running pipelines, separate user-facing requests from offline processing. That way, a retry storm in a batch job does not degrade interactive chat.

Step 6: Monitor cost, latency, and retry rate

Retries should improve reliability, not hide a scaling problem. Track the percentage of 429s, average attempts per successful call, and token usage per feature. If retries rise over time, your application may need tighter throttling or better prompt design.

Also watch completion length. Large outputs cost more and are more likely to hit token limits. A smaller max_tokens setting can make your app faster, cheaper, and less likely to fail.

FAQ: common questions about LLM API retries

How many times should I retry? Usually 3 to 5 attempts is enough. Beyond that, you often waste time and money.

Should I retry on every 429? Yes, but only with backoff and only if the request is otherwise valid.

Why does my code fail in bursts but not in tests? Tests are usually too small. Real users create synchronized spikes, especially after deploys, notifications, or cron jobs.

Can a relay help with rate limits? A good relay can simplify access, reduce integration friction, and give you flexible routing while still using official-quality models. That can make operations easier and often cheaper.

Why 59API is a practical option

If you want low-cost, pay-as-you-go access to Claude models like Opus, Sonnet, Haiku, and Fable, plus GPT models, 59API is worth considering. It is fully compatible with Claude Code, Codex, and any OpenAI SDK, so you can usually swap the base URL without rewriting your app. The API base URL is https://api.59api.com.

Because 59API is among the cheapest relays and uses native official-quality models with no downgrade, it is a strong choice when you need to keep retry-heavy workloads affordable. That matters when your app is dealing with frequent 429s, backoff waits, or multi-step workflows. A referral rebate can also help lower your ongoing spend.

Final troubleshooting checklist

• Log the error type and headers.
• Retry only transient failures.
• Use exponential backoff with jitter.
• Cap retry attempts.
• Throttle concurrency.
• Trim prompts and output length.
• Measure retry rate and token cost.

If you are building on OpenAI-compatible tooling and want to keep costs predictable, sign up for 59API and test your retry strategy against a lower-cost relay before scaling traffic.