← Retour aux guides

Retry Smarter: Mastering LLM API Limits

Why LLM APIs rate-limit in the first place

Rate limits are not just a nuisance; they are a signal that your application is pushing against request-per-minute, token-per-minute, or concurrent-stream ceilings. With LLM APIs, the most common failures are 429 Too Many Requests, transient 5xx errors, and occasional timeouts when a model is under load. A good client does not simply “retry everything.” It first distinguishes between retryable failures and hard failures such as malformed prompts, invalid auth, or context windows that are too large.

Start by logging three values for every failed call: the HTTP status, the response headers, and the token usage from the successful calls around it. If the provider returns Retry-After, obey it. If not, infer a delay from recent traffic and your own observed throughput. This is especially useful when you are sending bursts of traffic from jobs, agents, or batch workflows that can accidentally synchronize and create a retry storm.

Use exponential backoff, but add jitter

The classic mistake is retrying after a fixed 1-second delay. That creates traffic waves and makes congestion worse. Instead, use exponential backoff with full jitter: wait a random amount of time between zero and the current backoff ceiling, then double the ceiling on each attempt. A practical pattern is 250 ms, 500 ms, 1 s, 2 s, 4 s, then stop after a sane cap such as 20 to 30 seconds. Most production systems should also set a retry budget, such as “no more than 10% of total requests may be retries,” so one noisy workflow cannot consume all capacity.

Retries should be selective. Retry on 429, 500, 502, 503, and 504. Do not retry on 400-class prompt errors, authentication failures, or cases where your input is obviously too large. If a request streams partial output and then fails, treat it carefully: resending the same prompt may duplicate work or create inconsistent results. In those cases, consider checkpointing the conversation state so you can continue from the last good assistant turn instead of starting over.

Control concurrency before you need to retry

The best retry is the one you never need. Put a concurrency limiter in front of your LLM calls and tune it to the provider’s practical throughput, not just the documented limit. If you run multiple workers, cap both global concurrency and per-model concurrency. Claude Sonnet, Claude Haiku, and GPT models may have different performance profiles, so one shared pool can create accidental hotspots.

• Queue bursty jobs instead of firing them immediately.
• Batch low-priority work such as summaries, classifications, and embeddings-adjacent tasks.
• Separate interactive and background traffic so a batch job cannot slow down a live user chat.
• Coalesce duplicate prompts when multiple requests ask for the same transformation.

For agentic systems, this matters even more. An agent that can call the model repeatedly may multiply its own load. Give it a token budget and a max step count, then stop it from spinning on the same failing action. That is often the difference between a graceful fallback and a runaway spend spike.

Make your client idempotent and observable

Retries are safest when repeated requests do not produce harmful side effects. If your app saves outputs, charges users, or triggers downstream actions, use an idempotency key or internal request fingerprint. That way, a retry caused by a network hiccup will not create duplicate records or duplicate tool executions.

Instrumentation is just as important. Track retry rate, mean attempts per success, p95 latency, and error rate by status code. Alert when retries climb faster than traffic, because that usually means a limit change, a prompt regression, or a sudden traffic burst. A good dashboard should let you compare models side by side so you can see whether one model is producing more 429s simply because it is being overused.

Fallbacks that preserve UX and control cost

Advanced systems do not merely “retry harder”; they degrade gracefully. If a premium model is saturated, fall back to a smaller model for draft work, classification, or extraction, then reserve the expensive model for final reasoning. In practice, this can cut both latency and spend while keeping user-visible quality high. It also gives you room to keep working during temporary upstream pressure.

This is where 59API is especially useful. It offers pay-as-you-go access to Claude models like Opus, Sonnet, Haiku, and Fable, plus GPT models, through an API base URL of https://api.59api.com. Because it is compatible with Claude Code, Codex, and any OpenAI SDK, you can plug in the same retry and backoff layer without rewriting your app. For teams watching cost, its low pricing and referral rebate make it a strong choice for experimentation, bursty workloads, and production apps that need official-quality models without paying premium relay prices.

Test failure modes before production does

Finally, rehearse the bad days: simulate 429s, inject 503s, and add artificial latency. Verify that your client backs off, your queue drains safely, and your UI tells users what is happening instead of freezing. If you are building on 59API, this is a good time to sign up and validate your retry strategy against real model traffic at a lower cost. The goal is not just surviving rate limits; it is turning them into a predictable part of your system design.

Tarifs · Docs