LLM API Rate Limits: A 15-Minute Retry Setup
A reliable LLM integration needs more than a loop that retries HTTP 429. You need to distinguish temporary throttling from exhausted credit, prevent synchronized retries, and avoid paying for duplicate generations. This quick-start gives you a practical baseline you can implement before increasing traffic.
1. Identify which limit you actually hit
LLM services commonly enforce requests per minute, tokens per minute, and concurrent-request limits. Account balances, daily quotas, and model-specific capacity can create additional restrictions. With a relay, limits may exist at both the relay and upstream provider.
- Inspect the response: Record the HTTP status, structured error code, model, request ID, and any Retry-After or rate-limit headers. Header names and availability vary.
- Separate throttling from quota: A temporary 429 may recover after waiting. A 429 reporting insufficient credit or exhausted quota usually requires account action.
- Check permanent failures: Invalid credentials, malformed requests, and unsupported models generally will not improve with retries.
Do not log API keys or full prompts by default. Error metadata is usually sufficient for troubleshooting.
2. Give retries one owner
The OpenAI Python SDK retries eligible failures twice by default. Adding three application retries around that behavior can produce up to twelve attempts for one logical request. Choose either SDK-managed retries or an application-level policy, not overlapping policies.
For custom scheduling, set the SDK's max_retries to 0. Configure an explicit timeout appropriate to your workload, and enforce a separate overall deadline. A starting policy for short interactive requests is four total attempts within 90 seconds, with a 20-second per-attempt timeout. Increase those values deliberately for long generations; they are not universal defaults.
Retry temporary 429 responses and selected transient failures, such as connection errors and HTTP 502, 503, or 504. Treat uncertain outcomes carefully: a timeout does not prove the server stopped generating.
3. Use exponential backoff with full jitter
Before retry number n, starting at zero, calculate a random delay between zero and the smaller of 30 seconds or 2 to the power n seconds. This produces progressively wider windows without making every client retry simultaneously.
- Honor Retry-After: It can contain seconds or an HTTP date. Parse either format and treat a valid value as the minimum wait.
- Combine safely: Wait at least the larger of the server's requested delay and your randomized backoff. A small additional random delay can spread clients sharing the same reset time.
- Respect the deadline: If the required wait leaves no useful time for another attempt, return a controlled error or move the job to a durable queue.
Never shorten a server-requested wait just because your local backoff cap is lower. Also propagate cancellation so abandoned requests stop consuming your retry budget.
4. Prevent overload before sending requests
Put a concurrency semaphore and a rate-aware queue in front of your API client. Track request volume and estimated token demand separately: ten short classification calls behave differently from ten large document summaries.
Start around 80% of the published limit, then tune against observed responses. Estimate prompt tokens plus your requested output allowance, while recognizing that provider accounting can differ. Reserve capacity before dispatch and reconcile it using returned usage where appropriate.
In multi-worker deployments, use a shared limiter or coordinated budgets. An independent limiter in every process can collectively exceed the account limit. When the queue is full, reject or defer work rather than accumulating unlimited memory.
5. Protect streaming responses and control cost
Once streamed output reaches a user, silently restarting can duplicate text and produce a different answer. Surface an interrupted response and offer an explicit retry. Likewise, do not assume an idempotency header prevents duplicate billing unless the endpoint documents that guarantee.
59API is a low-cost, pay-as-you-go relay offering native, official-quality Claude and GPT models without model downgrades. Its API base URL is https://api.59api.com, with compatibility for Claude Code, Codex, and OpenAI SDKs. Keep your limiter configurable rather than assuming compatibility means identical quotas or error headers.
If you want a budget-friendly starting point, consider signing up for 59API and checking its current pricing, limits, and referral rebate. Track retry rate, queue latency, and cost per completed task: inexpensive tokens help most when retries do not multiply them.