← Back to all guides

LLM API Rate Limits: Retries That Actually Work

Why LLM Retry Logic Needs More Than a Loop

Rate limiting is normal when production workloads share finite model capacity. A naive retry loop can turn a temporary 429 response into a prolonged outage: every worker retries simultaneously, request volume spikes, and the next attempt is even more likely to fail. Reliable LLM integrations distinguish between retryable capacity errors, permanent request errors, and failures whose outcome is unknown.

Start by categorizing responses. Retry HTTP 429 rate-limit responses, most 5xx server errors, connection resets, and request timeouts. Do not retry malformed requests, authentication failures, invalid model names, or context-window violations until the application has changed the request. For a timeout, treat the result as potentially successful unless your request design gives you a way to verify it safely.

Use Exponential Backoff With Full Jitter

Exponential backoff increases the delay after each failed attempt. A practical schedule starts around 500 milliseconds and doubles per retry, with a maximum delay such as 20 or 30 seconds. The crucial detail is jitter: randomize the actual wait so a fleet of clients does not retry at the same instant.

• Base delay: 0.5 to 1 second is suitable for interactive API calls.
• Growth factor: Double the upper delay limit after each attempt.
• Full jitter: Choose a random delay between zero and the current upper limit.
• Maximum attempts: Usually 3 to 5 attempts for synchronous user requests.
• Retry budget: Stop when the cumulative wait would exceed the user-facing deadline.

When the provider returns a Retry-After header, honor it. It represents server knowledge that your local algorithm lacks. If a reset timestamp or remaining-request header is available, feed it into your scheduler rather than treating every endpoint and model as identical.

Control Concurrency Before the Provider Has To

Retries solve transient failures; concurrency control prevents many of them. Put a semaphore, token bucket, or queue in front of each model route. Track both requests per minute and tokens per minute, because a few long prompts can exhaust token capacity even when request count looks healthy.

Estimate token demand before sending work, including the requested completion budget. Reserve that capacity in your limiter, then release unused completion tokens after the response. For batch jobs, a worker queue with adaptive concurrency is more stable than firing every task at once. Reduce worker count when 429s increase; raise it slowly only after a sustained healthy period.

Make Retries Idempotent and Cost-Aware

Text generation can be expensive and nondeterministic. If a network timeout occurs after the server accepted the request, sending it again may produce a second billable completion or duplicate a downstream action. Generate a unique request ID at your application boundary and persist its status. For workflows that write records, send emails, or trigger tools, store an idempotency key before calling the model and make downstream operations deduplicate on that key.

Cache deterministic or low-temperature requests by normalized prompt, model, system instruction, and relevant generation settings. For non-cacheable calls, log attempt number, response status, model, estimated tokens, latency, and the final outcome. These fields reveal whether a limit is caused by traffic bursts, oversized prompts, a single tenant, or an overly aggressive retry policy.

Design Fallbacks Deliberately

A fallback model should preserve the user task, not merely return any text. Define which requests can move to a faster or less expensive model, which require the primary model, and how output quality is checked. For example, route classification, extraction, and summarization workloads to a lower-cost model during contention, while holding complex code review jobs in a queue.

59API is useful here because it provides pay-as-you-go access to native official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, plus GPT models through a compatible relay. Its OpenAI SDK compatibility and Claude Code and Codex support let teams keep one retry and routing layer while selecting an appropriate model per workload. Lower relay costs also make controlled fallback and bounded retries easier to operate without silently degrading output quality.

Operational Checklist

• Classify errors and retry only transient failures.
• Use exponential backoff with full jitter and respect Retry-After.
• Enforce request and token concurrency limits locally.
• Cap attempts by a real latency and cost budget.
• Use idempotency keys for actions with side effects.
• Measure 429 rate, retry success rate, queue depth, token use, and p95 latency.

Build these controls before traffic grows, then test them with simulated 429s and timeouts. Developers looking for an economical compatible endpoint can sign up for 59API, use https://api.59api.com as the base URL, and evaluate its referral rebate alongside their normal reliability and cost metrics.