← Volver a las guías

LLM API Rate Limits: A Retry Guide That Cuts Failures

Why LLM API requests hit rate limits

An HTTP 429 usually means your application exceeded an allowed request rate, token rate, or concurrency limit. But some APIs also use 429 for exhausted credits or quota. Retrying every 429 blindly can waste time and amplify an outage.

The reliable approach is to identify the limiting resource, reduce demand, and retry only recoverable failures. This matters whether you call a provider directly or use an AI API relay such as 59API.

Troubleshooting: identify the actual limit

For each failed request, record the HTTP status, error type, model, timestamp, request ID if available, and rate-limit headers. Never log API keys or sensitive prompts.

• Requests per minute: Many small calls can exhaust this limit even when token usage is low.
• Tokens per minute: Long prompts and large output budgets can consume capacity quickly. Some providers account for requested output capacity before generation finishes.
• Concurrency: Too many simultaneous generations can trigger throttling despite acceptable average traffic.
• Quota or billing: An insufficient-credit error needs a billing or quota fix, not exponential backoff.

Check whether limits apply per model, project, organization, or account. Multiple workers sharing one key may compete for the same capacity. Do not assume switching models provides an independent quota.

Build a bounded retry policy

Typically retry transient 429 responses, connection failures, timeouts, and temporary server errors such as 502, 503, or 504. Check the API's error documentation before treating other failures as retryable. Authentication, permission, and invalid-request errors generally require configuration changes.

• Honor server guidance: If Retry-After is present, interpret it as seconds or an HTTP date. Wait at least that long; if the delay exceeds your request deadline, stop or defer the job.
• Use exponential backoff with full jitter: Without server guidance, choose a random delay between zero and the smaller of 30 seconds or 1 second multiplied by 2 to the retry number, starting at zero.
• Set hard boundaries: For example, allow four retries after the initial attempt and enforce a 60-second overall deadline, including request execution time.
• Check existing retries: Your SDK may already retry automatically. Avoid stacking an application retry loop over SDK retries without calculating the total attempts.

Jitter prevents many clients from retrying together. After the final attempt, return a clear failure or move background work to a delayed queue rather than retrying indefinitely.

Prevent throttling before it happens

Use a shared queue and concurrency semaphore. Start with a small number of in-flight requests, then increase cautiously while monitoring 429 frequency and latency. When throttling rises, lower concurrency and slow admissions.

For token-heavy workloads, estimate prompt tokens and reserve an output allowance before dispatch. Keep headroom below the published limit; minute-based quotas may use rolling windows or shorter enforcement intervals. Shorten repetitive context, set realistic output limits, and cache reusable results where privacy and freshness requirements permit.

FAQ: Is retrying a timeout safe?

Not always. A timeout does not prove the server stopped processing. A second attempt may generate another response and incur additional charges. Use idempotency keys only where the endpoint explicitly supports them. For tool-using applications, deduplicate external actions separately using an application operation ID.

FAQ: What if a stream fails halfway through?

Treat received text as a partial result. Reissuing the request generally starts a new generation; it does not resume the old stream. Do not silently append a replacement response to partial output. Tell the user the stream failed, then offer a clearly separated retry.

FAQ: How does this work with 59API?

59API offers low-cost, pay-as-you-go access to native, official-quality Claude and GPT models without downgrading them, plus compatibility with Claude Code, Codex, and OpenAI SDKs. Its API base URL is https://api.59api.com; follow its integration instructions for the exact endpoint configuration.

Affordable pricing does not eliminate upstream throttling: inspect returned errors and headers rather than assuming relay-specific limits. If you want a budget-friendly integration, consider signing up for 59API and testing your retry policy with a small workload first. Its referral rebate is an additional benefit, separate from rate-limit handling.

Precios · Docs