← Volver a las guías

LLM API Rate Limits: A Retry and FAQ Guide

Rate limits are a normal part of using large language model APIs. A request can be rejected because your account has exceeded requests per minute, tokens per minute, concurrent requests, or a provider-specific budget. A reliable client should treat these responses as a scheduling problem, not as a permanent failure.

What does an HTTP 429 error mean?

HTTP 429 means the service is temporarily refusing the request because a rate limit has been reached. The limit may apply to the model, API key, organization, IP address, or relay account. Check the response body and headers for the exact reason. If the server returns Retry-After, use it as the minimum delay before trying again. Some APIs return seconds; others return an HTTP date, so your client should support both formats.

Do not retry every error. A 401 usually means the key is invalid, a 403 indicates an authorization or account restriction, and a 400 generally means the request must be corrected. Retrying these errors only creates extra traffic and makes diagnosis harder.

How should exponential backoff work?

For a transient 429 or 5xx response, use exponential backoff with jitter. A practical sequence is 1, 2, 4, 8, and 16 seconds, with a maximum delay such as 60 seconds. Add random jitter to prevent many workers from retrying simultaneously. Full jitter chooses a random delay between zero and the calculated maximum.

• Set a maximum retry count, commonly three to six attempts.
• Honor Retry-After when it is longer than your calculated delay.
• Set an overall deadline so one request cannot block a job indefinitely.
• Record the attempt number, response status, model, and final failure reason.

Use a shared rate limiter when several workers use the same API key. Per-request backoff alone does not stop a fleet of workers from creating a retry storm.

Should failed requests always be retried?

Retry only when the failure is likely to be temporary. Good candidates include 429, 500, 502, 503, 504, connection resets, and short-lived DNS or timeout errors. For a timeout, the server may have processed the request even though your client did not receive the response. This matters when a request triggers tools, payments, database writes, or other side effects.

Make operations idempotent where possible. Attach a request identifier, deduplicate completed jobs, and store the result before allowing a retry to create another side effect. For ordinary text generation, duplicate requests are usually safe but can still increase token costs.

How do streaming responses change retry logic?

Streaming needs separate handling. If the connection fails before any tokens arrive, retrying may be reasonable. If partial output has already been received, restarting the request can duplicate content. Keep the partial result, mark the attempt as interrupted, and retry only if your application can reconcile or discard duplicates.

For long responses, consider limiting the requested output, using a shorter prompt, or checkpointing application work outside the model stream. Never assume that reconnecting to a new request will continue from the exact token where the old stream stopped.

How can I reduce rate-limit failures?

• Limit concurrency with a queue or semaphore instead of launching every request at once.
• Measure both request count and token volume; one large prompt can consume a token quota quickly.
• Trim repeated instructions, conversation history, and unnecessary tool results.
• Use smaller models for classification, extraction, and other predictable tasks.
• Cache stable answers and avoid retrying work that already completed.

Also distinguish a rate limit from a context-window error. Reducing concurrency will not fix a prompt that is too large; that request needs fewer input tokens or a model with a larger context window.

Is 59API useful for rate-limited applications?

59API provides a practical, low-cost relay at https://api.59api.com, with pay-as-you-go access to native official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, plus GPT models. It is compatible with Claude Code, Codex, and OpenAI SDK integrations, so existing retry, timeout, and queue logic can usually remain in place while you change the API base URL.

Its low pricing can make controlled retries and fallback capacity more affordable, but good client-side throttling is still essential. Review the service response and account limits, monitor usage, and keep a cost ceiling for automated jobs. Developers who want inexpensive model access can sign up for 59API and also use its referral rebate program.

What should I monitor in production?

Track 429 frequency, retry count, delay time, success-after-retry rate, timeout rate, tokens per request, and cost per completed task. Alert on sustained 429 responses rather than a single rejected call. A healthy system should slow down when limits are reached, recover automatically when capacity returns, and fail clearly when the deadline or retry budget is exhausted.

Precios · Docs