← Retour aux guides

Rate Limit Retries for LLM APIs: A Practical Guide

Rate Limit Retries for LLM APIs: A Practical Guide

LLM APIs are fast and powerful, but production apps must handle rate limits, temporary overload, network failures, and duplicate requests correctly. A good retry strategy prevents user-facing errors, protects your budget, and keeps your app stable during traffic spikes. This step-by-step tutorial shows how to build reliable retry handling for Claude and GPT-compatible APIs, including when using 59API, a low-cost AI API relay with official-quality native models and an OpenAI-compatible base URL at https://api.59api.com.

Step 1: Understand the failures you should retry

Not every error deserves another request. Retrying the wrong failures wastes money and can make outages worse. In most LLM integrations, you should retry temporary failures and avoid retrying permanent ones.

• Retry: HTTP 429 rate limit errors, HTTP 408 request timeout, HTTP 500, 502, 503, and 504 server-side errors, and short network interruptions.
• Usually do not retry: HTTP 400 invalid request, 401 authentication failure, 403 permission error, 404 wrong endpoint, or context length errors.
• Inspect provider messages: Some APIs return helpful details such as quota exceeded, requests per minute exceeded, or tokens per minute exceeded. Treat quota exhaustion differently from short-term throttling.

With 59API, you can access Claude models such as Opus, Sonnet, Haiku, and Fable, plus GPT models, through a relay compatible with Claude Code, Codex, and OpenAI SDKs. That compatibility makes it easier to keep your retry logic provider-neutral.

Step 2: Respect Retry-After when it exists

When an API returns HTTP 429 or 503, it may include a Retry-After header. This is the server telling your client how long to wait before trying again. Always prefer this value over your own delay calculation when it is present and reasonable.

For example, if Retry-After is 6, wait about six seconds before the next attempt. If it is an HTTP date, convert it into a delay. Add a small random jitter so many workers do not retry at the exact same moment. If the value is extremely large, cap it and surface a graceful message to the user instead of blocking forever.

Step 3: Use exponential backoff with jitter

For retryable errors without Retry-After, use exponential backoff. A practical pattern is: first retry after 500 ms, then 1 second, 2 seconds, 4 seconds, and 8 seconds, with random jitter added to each delay. Jitter is important because it spreads traffic and prevents a thundering herd.

A safe default is to allow three to five total attempts, including the first request. For interactive chat, keep retries short so users are not waiting too long. For background jobs such as summarizing documents, you can tolerate longer delays and more attempts.

• Interactive request: 3 attempts, maximum delay of 4 seconds, then show a friendly retry button.
• Batch processing: 5 attempts, maximum delay of 30 seconds, then move the job to a failed queue.
• High-value workflow: retry in the background and notify the user when complete.

Step 4: Make requests safe to retry

Retries can create duplicate work. For read-like operations such as generating a response, duplicates may be acceptable but still cost money. For actions that update your database, send emails, charge users, or trigger tools, you need idempotency.

Store a request ID before calling the model. If the same user action is retried, reuse the same ID and do not repeat side effects that already completed. Save the final model response with that ID. If a timeout occurs after the model finished but before your app received the response, your system can detect whether the operation already succeeded.

For tool-calling agents, separate model generation from tool execution. Retry the model call if needed, but execute external tools only after validating that the action has not already run.

Step 5: Add client-side throttling

Retries are not enough if your app constantly exceeds limits. Add a simple queue or token bucket so your application controls concurrency before the API rejects requests. Track requests per minute and tokens per minute separately because LLM limits are often based on both.

• Limit concurrency: Start with a small number of simultaneous requests per model, then increase gradually.
• Estimate tokens: Long prompts and large completions consume more capacity, so route heavy tasks through a queue.
• Prioritize users: Give interactive traffic priority over background enrichment jobs.
• Degrade gracefully: Use a smaller or faster model for low-priority work during spikes.

This is where a cost-conscious relay helps. 59API is designed for cheap, pay-as-you-go access, so developers can experiment with Claude and GPT models without committing to expensive fixed plans. Because it uses official-quality native models with no downgrade, cost savings do not require sacrificing output quality.

Step 6: Configure SDKs with a portable base URL

If your app already uses an OpenAI SDK, keep your retry middleware around the SDK call and point the client to 59API’s base URL: https://api.59api.com. Store the API key in an environment variable, not in frontend code. Keep model names configurable so you can switch between Claude Sonnet for strong general performance, Haiku for speed and lower cost, Opus for harder reasoning, or GPT models when they fit your workload.

Do not hard-code provider-specific assumptions inside business logic. Instead, create one LLM client wrapper that handles headers, timeouts, retries, logging, and model selection. Your application should call that wrapper, not the raw SDK everywhere.

Step 7: Monitor retries, latency, and spend

Reliable retry logic needs observability. Log the model, status code, error type, attempt number, delay, prompt size, completion size, latency, and final outcome. Watch for retry storms, rising 429 rates, and sudden token growth. If retries increase after a prompt change, your prompt may be too long or your concurrency too aggressive.

Also measure cost per successful task, not just cost per request. A cheap request that fails repeatedly can become expensive. 59API’s low pricing and referral rebate can reduce operating costs, but you still need metrics to identify waste and tune model choice.

Step 8: Test failure scenarios before launch

Before production, simulate 429, 500, timeout, slow streaming, and network reset failures. Confirm that your app waits correctly, caps retries, avoids duplicate side effects, and gives users a clear message when the system is busy. For streaming responses, decide whether you will retry only before any tokens are shown, or display a partial response with a retry option.

A strong LLM API integration is not just about sending prompts. It is about controlling traffic, retrying intelligently, and keeping costs predictable. If you want affordable access to Claude and GPT models through familiar SDKs, 59API is a practical option to try, especially for teams that need low-cost, pay-as-you-go scaling with referral rebates available.

Tarifs · Docs