LLM API Rate Limits: A 15-Minute Retry Playbook
A reliable LLM integration needs more than a retry loop. It needs to distinguish temporary throttling from exhausted quota, avoid duplicate work, and stop before retries consume your latency and spending budgets. Here is a practical starting policy you can implement today.
1. Identify which limit you actually hit
LLM APIs commonly enforce requests per minute, tokens per minute, and concurrent-request limits. Limits may apply to a model, project, account, or upstream provider. A relay can also have limits separate from those upstream.
When a request fails, capture the HTTP status, structured error code, request ID, and any rate-limit headers. Never log API keys or full prompts by default.
- 429 with temporary throttling: Queue the request and retry within a bounded budget.
- 429 indicating exhausted credits or quota: Stop automatic retries. Check billing, account limits, or the documented reset period.
- 400, 401, or 403: Fix the request, authentication, or permissions rather than retrying unchanged.
- Transient 5xx errors or connection failures: Consider a bounded retry, accounting for the possibility that the original request already ran.
2. Give retries one owner and a deadline
Start with three total attempts: the original request plus two retries. For interactive requests, a reasonable initial configuration is a 60-second overall deadline and a maximum 20-second timeout per attempt. Adjust these values for model latency, output length, and your product's response-time target.
Before each attempt, calculate the remaining deadline and shorten the attempt timeout accordingly. Include queueing, sleeps, and connection time in the overall budget. If no useful time remains, fail clearly instead of launching another request.
Check your SDK's defaults. The OpenAI Python SDK supports automatic retries; if your application owns the retry loop, configure max_retries=0. Otherwise, three application attempts can become many more network calls when layered over SDK retries.
3. Honor Retry-After and add jitter
If the response includes Retry-After, parse it as either seconds or an HTTP date. Wait at least that long before trying again. If the required wait exceeds your remaining deadline, return a temporary-failure response or defer the job rather than retrying early.
Without a server-provided delay, use exponential backoff with full jitter. For this three-attempt policy, choose a random delay between zero and one second before the first retry, then zero and two seconds before the second. Randomization prevents many workers from retrying simultaneously.
When extending this policy for background jobs, cap the exponential delay and retain a maximum attempt count. Where both server guidance and local backoff apply, use the longer wait; optional positive jitter can spread the retry burst further.
4. Reduce traffic before adding retries
Put a bounded queue and concurrency semaphore in front of API calls. Start conservatively, measure throttling, and reduce concurrency when sustained 429 responses appear. A semaphore limits simultaneous requests; it does not enforce requests per minute, so add a rate limiter too.
- Budget tokens: Estimate input tokens and reserve an output allowance before dispatch. Reconcile against returned usage when available; follow the provider's documented accounting rules.
- Keep outputs bounded: Set a sensible output-token limit instead of requesting the maximum for every task.
- Separate workloads: Give interactive requests priority over batch jobs, and avoid an unlimited retry backlog.
5. Treat ambiguous failures carefully
A timeout does not prove the provider rejected your request. Retrying can duplicate generation and charges. Use idempotency keys only where the endpoint explicitly documents support; otherwise, deduplicate jobs in your application without assuming that prevents duplicate upstream billing.
For streaming, do not silently restart after displaying partial output. Tell the user generation was interrupted or offer an explicit regeneration action. Make downstream tool calls independently idempotent.
6. Apply the policy with 59API
59API provides low-cost, pay-as-you-go access through its API base URL, https://api.59api.com, with OpenAI SDK compatibility and native, official-quality models without downgrades. Keep retry logic provider-neutral, but track limits and usage separately for routes such as DeepSeek V4 Flash and GPT-6 Astra.
Measure retry rate, queue time, final success rate, latency, and cost per successful task. If you want to test this setup affordably, consider signing up for 59API and starting with a small spending budget.