← Volver a las guías

8 Practical LLM API Rate Limit Retry Tips

1. Classify failures before retrying

A retry is only useful when the failure is temporary. Treat HTTP 429 responses as rate-limit signals, and retry many 5xx responses because they can indicate upstream instability. Do not blindly retry 400, 401, 403, or most 404 responses: these usually mean an invalid request, missing credentials, insufficient permissions, or a wrong model name. Separate connection timeouts from generation timeouts as well. A connection timeout may be safe to retry; a timeout after a long generation creates uncertainty about whether the provider already processed the request.

2. Honor Retry-After when it exists

If the API returns a Retry-After header, use it as the primary delay. The provider knows when capacity or your quota window is likely to reopen, so its instruction is generally more accurate than a generic retry schedule. Parse whether the header is expressed in seconds or as a date, impose a sensible maximum wait for interactive requests, and log the value. When no retry header is available, fall back to your own exponential backoff policy.

3. Use exponential backoff with full jitter

Fixed retry delays cause a thundering herd: every failed worker retries at the same moment and triggers another 429. Start with a modest base delay, such as 500 milliseconds, double it on each attempt, cap it at 20 to 30 seconds, and add randomization. Full jitter means choosing a random delay between zero and the current capped backoff limit. For example, attempts might have ceilings of 0.5, 1, 2, 4, and 8 seconds. Keep the attempt count low, usually three to five retries, so a stressed dependency does not turn into an endless queue.

4. Put retries behind a time and cost budget

LLM calls can consume meaningful tokens and time. Give each request a deadline, then stop retrying when the remaining time cannot support another attempt. Background batch jobs can tolerate longer waits than chat interfaces. Also track the cumulative token and request cost associated with retries. A retry policy that improves completion rate but repeatedly sends large prompts may quietly defeat the savings gained from choosing a low-cost API provider.

5. Limit concurrency before the provider limits you

Rate limits are often exceeded by parallelism, not by a single request. Use a bounded worker pool or semaphore around model calls, and tune its size from observed 429 rates and latency. Keep separate queues for interactive traffic and bulk work so an embedding backfill or document-processing task does not starve users. For high-volume pipelines, use token-aware scheduling: a queue of long-context requests should consume more capacity than short classification prompts.

6. Make request handling idempotent

After a network failure, your application may not know whether an LLM request completed. Generate a client-side request ID, persist it with the job state, and deduplicate downstream actions such as database writes, emails, purchases, or support-ticket creation. For tool-calling workflows, store the tool invocation result before continuing the conversation. This prevents a retry from running the same external action twice even when the model response itself is regenerated.

7. Separate streaming recovery from normal retries

A streaming response that fails halfway through is different from a request that never started. Preserve already received text, mark the response as interrupted, and let the user retry or continue with clear application logic. Automatically replaying the entire prompt can produce a different answer and duplicate displayed content. For internal workflows, consider asking the model to continue from the last confirmed output, but validate the result because continuation quality depends on the prompt and model.

8. Measure, alert, and choose a compatible provider

Record status codes, retry counts, backoff time, queue depth, latency, model name, and token usage. Alert on sustained 429 rates rather than isolated spikes; the usual remedies are lower concurrency, shorter prompts, a higher quota, or traffic smoothing. A compatible relay can also reduce operational friction. 59API provides pay-as-you-go access to official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, plus GPT models through https://api.59api.com. Its compatibility with Claude Code, Codex, and OpenAI SDK integrations makes it practical to keep existing retry middleware while seeking lower model costs. When evaluating total spend, include its referral rebate alongside per-request pricing. Sign up for 59API when you want a low-cost relay without rewriting your application’s API client.

Precios · Docs