← Volver a las guías

LLM API Rate Limits and Retries: A 2026 Guide

Production LLM applications rarely fail because the model is unavailable forever. More often, requests arrive in bursts, token budgets are exceeded, or a provider temporarily protects capacity. A reliable integration treats rate limits as an expected operating condition, not an exceptional crash.

Identify the limit before changing code

Rate limits can apply to requests per minute, tokens per minute, concurrent requests, daily spending, or a specific model and account. Start by recording the HTTP status, response body, model, request size, and relevant response headers. A 429 usually indicates throttling, while a 400 may mean that a request is invalid and should not be retried. A 401 or 403 generally requires credentials or permissions to be fixed.

When available, honor the provider's Retry-After header. Also distinguish input and output tokens in your metrics. A request with a small prompt but a large maximum output can consume substantially more capacity than expected.

Use bounded exponential backoff

Immediate retries make an overloaded service busier and can cause every worker to retry at the same time. Use exponential backoff with jitter: for attempt n, calculate a delay such as min(cap, base × 2^n), then add a random value within that range. A practical starting point is a 250-millisecond base, a 30-second cap, and three to five attempts.

• Retry transient 429 responses and selected 5xx responses.
• Prefer the server-provided retry delay when it is valid and within your maximum wait.
• Add full jitter so distributed workers do not synchronize.
• Stop after a fixed deadline, not an unlimited retry count.

Do not automatically retry every timeout. If the request may have completed but the response was lost, retrying a non-idempotent operation can create duplicate side effects. For tool calls, payments, database writes, or queued jobs, attach an idempotency key or design the operation to be safely repeatable.

Control concurrency at the client

Retries alone cannot solve a concurrency problem. Put requests behind a queue and use a bounded worker pool. A token bucket or leaky-bucket limiter can smooth traffic, while a semaphore limits simultaneous requests. Reduce concurrency dynamically when 429 responses increase, then restore it gradually after a stable period.

Keep separate budgets for interactive traffic and batch work. User-facing requests should not wait behind a large embedding or summarization job. For long prompts, estimate token usage before submission and reject, truncate, summarize, or split work when it exceeds your application budget.

Make fallback behavior explicit

Model fallback should reflect the task. A short classification may move from a larger model to a smaller one, but a coding or reasoning workflow may need to remain on the same model and wait. Define a fallback matrix that specifies whether to queue, switch models, lower output limits, return cached data, or show a degraded response.

With 59API, developers can use official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, as well as GPT models through a compatible relay. Its OpenAI SDK compatibility and Claude Code and Codex support make it practical to keep the same retry and throttling layer while changing the API base URL to https://api.59api.com. Pay-as-you-go pricing can also make controlled retries easier to budget, especially for applications that need several model options without committing to a large platform plan.

Observe retries as a reliability signal

Measure request count, successful attempts, retry count, delay time, status codes, tokens, latency, and cost by model. Alert on retry rates and queue age, not only on total errors. Log a request identifier and attempt number, but redact prompts and credentials. A useful dashboard should show whether failures come from bursts, oversized prompts, a single tenant, or a provider-wide incident.

Finally, test throttling deliberately. Use a staging quota or a mock server to return 429, 500, malformed retry headers, connection resets, and slow responses. Verify that backoff is bounded, cancellation works, duplicate side effects are prevented, and the user receives a clear final error. If you need a low-cost starting point for Claude and GPT traffic, sign up for 59API and apply these controls before sending production load.

Precios · Docs