← Back to all guides

LLM API Rate Limits: A Retry Strategy That Works

Why LLM Rate Limits Need a Deliberate Strategy

Rate limits protect LLM infrastructure from sudden traffic spikes. They may limit requests per minute, tokens per minute, concurrent requests, or spending volume. When your application exceeds a limit, the API commonly returns HTTP 429. Temporary provider-side issues may also produce 500, 502, 503, or 504 responses. Treating every failure the same creates unnecessary cost, duplicate work, and poor user experiences.

A reliable client should identify retryable failures, wait intelligently, reduce pressure on the API, and record enough telemetry to diagnose recurring issues. This approach is especially important for agent workflows, batch jobs, streaming chat, and code-generation tools, where one user action can trigger several large model calls.

Step 1: Classify Errors Before Retrying

Only retry failures that are likely to succeed later. HTTP 429 responses are usually retryable, as are transient server errors such as 500, 502, 503, and 504. Connection resets and short network timeouts can also be retried carefully.

• Retry: 429 rate-limit responses, temporary 5xx errors, and clearly transient network failures.
• Do not retry unchanged: 400 malformed requests, 401 invalid credentials, 403 permission failures, and 404 invalid endpoints or model names.
• Inspect first: Context-window errors, content-policy errors, insufficient balance errors, and invalid tool schemas require a request or account change rather than a blind retry.

Log the HTTP status, provider error code, request ID, model, estimated token count, and attempt number. These details let you distinguish a real capacity problem from a coding bug.

Step 2: Honor Retry-After and Use Exponential Backoff

If the response includes a Retry-After header, use that delay first. It is the server's direct guidance on when capacity may be available. If no header is present, use exponential backoff: wait a small initial interval, then increase it after each failure. For example, start at 1 second and roughly double the wait on each attempt: 1, 2, 4, 8, then 16 seconds.

Add jitter, which is a small random variation in the delay. Without jitter, many workers that fail together can retry together and immediately trigger another rate limit. A practical formula is a random delay between zero and the calculated backoff ceiling. Cap delays at 30 or 60 seconds and cap attempts, usually at three to five retries.

For example, after a 429, calculate a maximum delay of 2 raised to the attempt number, multiply by one second, cap it at 30 seconds, and sleep for a random value up to that cap. Always stop after the configured retry budget and return a useful error to the caller.

Step 3: Control Concurrency and Token Throughput

Retries alone do not solve a sustained overload. Use a concurrency semaphore or worker queue to limit in-flight LLM calls. Start conservatively, such as two to five simultaneous requests per model, then increase only after observing stable latency and low 429 rates.

Also budget tokens, not just request count. A single long prompt with a large completion allowance can consume more capacity than dozens of short classification calls. Estimate input tokens before sending a request and reserve output tokens using your maximum token setting. If your job queue is growing, slow the producer, split large batches, or route low-priority work to off-peak processing.

Step 4: Make Retries Safe and Idempotent

A timeout does not always mean the model request failed. The server may have completed the work while your client lost the response. For operations that create records, charge customers, send emails, or trigger tools, attach an idempotency key generated per logical task. Store the task state and reuse the same key if the request must be retried.

For chat and agent applications, keep a stable conversation or task identifier and persist completed tool results. Do not rerun an external tool merely because an LLM completion was retried. This prevents duplicate database writes, duplicate tickets, and repeated side effects.

Step 5: Handle Streaming Failures Differently

Streaming responses need special treatment because partial text may already be visible to the user when a connection fails. Save received chunks, mark the generation as incomplete, and offer a continue action rather than silently replaying the entire request. Replaying can produce duplicated text and doubles token spend.

For structured JSON output, validate the final assembled response before using it. If parsing fails after a stream interruption, retry with a clear instruction to return the missing structured result, ideally using a new request that includes only the necessary context.

Step 6: Monitor, Tune, and Choose a Flexible API Layer

Track 429 rate, retry count, retry success rate, queue depth, latency, token usage, and cost by model. Alert on sustained 429 spikes rather than individual events. If retries are frequently exhausted, lower concurrency, shorten prompts, reduce maximum output tokens, or spread non-urgent workloads over time.

59API is a practical low-cost option for teams building these controls. Its pay-as-you-go relay provides access to native official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, plus GPT models, without a downgrade. With the API base URL https://api.59api.com and compatibility with OpenAI SDKs, Claude Code, and Codex, you can often keep your existing client architecture while applying the retry policy described above. When you are ready to test a cost-conscious multi-model setup, consider signing up for 59API and use its referral rebate to reduce ongoing development costs.

Pricing · Docs