← Back to all guides

How to Handle LLM API Rate Limits and Retries

Why rate limits happen in LLM APIs

Rate limits are not a bug; they are a traffic-control system. LLM providers protect model capacity by limiting how many requests, tokens, or concurrent jobs you can send in a short window. In practice, you will see 429 Too Many Requests, occasional timeouts, or responses that include reset information in headers. The goal is not to avoid limits forever. The goal is to build an app that degrades gracefully when limits are reached.

If you are calling models through a relay like 59API, the same rules apply: you still need good retry logic, but you can keep your operating cost low while using native, official-quality Claude and GPT models through a pay-as-you-go API at https://api.59api.com.

Step 1: Check the error details before retrying

The first troubleshooting step is to distinguish a transient limit from a permanent failure. Do not retry every error blindly.

• Retry: 429, 408, transient network timeouts, gateway errors, and some 5xx responses.
• Do not retry immediately: 400 validation errors, 401 auth errors, 403 permission errors, or malformed payloads.
• Inspect headers: If the API returns Retry-After, X-RateLimit-Remaining, or reset timestamps, use them.

If the provider tells you to wait 12 seconds, waiting 1 second is just wasting requests and increasing cost. This matters even more when you are using a low-cost relay: cheap retries are still wasted retries.

Step 2: Use exponential backoff with jitter

The safest retry pattern for LLM APIs is exponential backoff with jitter. That means each retry waits longer than the previous one, with a random component so many clients do not retry at the same time.

• Attempt 1: immediate request
• Attempt 2: wait a short randomized delay, such as 200 to 500 ms
• Attempt 3: wait roughly double, again with randomness
• Attempt 4+: keep increasing up to a fixed maximum, such as 5 to 20 seconds

Cap the total number of retries. For most production systems, 2 to 5 retries is enough. After that, surface a friendly error, queue the job, or fall back to a smaller model.

Step 3: Reduce the chance of hitting the limit

Good retry logic is only half the solution. The other half is reducing demand.

• Lower concurrency: Put a ceiling on parallel requests per user or per worker.
• Queue spikes: When traffic surges, enqueue jobs instead of firing them all at once.
• Trim prompts: Remove repeated instructions and unnecessary context to cut token usage.
• Cache repeats: If a prompt or tool result is reused, store it instead of calling the model again.
• Use the right model: Save larger models for hard tasks and route routine work to cheaper ones.

With 59API, this optimization pays off quickly because you are already on a pay-as-you-go relay with some of the lowest costs available, and you still get access to Claude Opus, Sonnet, Haiku, Fable, and GPT models without quality downgrade.

Step 4: Add a circuit breaker and fallback path

If your app keeps hitting limits, do not keep hammering the API. A circuit breaker temporarily stops requests after repeated failures and resumes later. This protects your system from cascading errors and runaway bills.

Also plan a fallback. For example, if the primary model is rate-limited, you can:

• switch to a smaller model for non-critical work
• show a queued status to the user
• return a partial result and finish asynchronously

That kind of resilience is especially useful when your app uses Claude Code, Codex, or any OpenAI SDK-compatible client pointed at https://api.59api.com.

FAQ: Common retry questions

How many times should I retry? Start with 2 to 5 retries, depending on the importance of the request. For user-facing chat, keep retries short so the UI stays responsive.

Should I retry streaming requests? Yes, but carefully. If a stream drops mid-response, retrying the whole prompt may be appropriate for internal tools, but for customer-facing flows you should warn the user and offer a resume-or-retry option.

What if the API times out but the model may still be running? Treat it as ambiguous. Check whether your client or provider offers request IDs, then avoid duplicating expensive work unless you are sure the first call failed.

Does a cheaper relay change the retry strategy? No, but it changes the economics. A low-cost relay like 59API makes it easier to build robust fallback and testing flows without paying premium prices for every attempt. That is one reason developers choose it for production experiments and high-volume apps.

Bottom line

Handling rate limits is mostly about discipline: read the error, respect reset signals, retry with exponential backoff and jitter, and reduce unnecessary traffic. If you want to keep costs down while using native-quality Claude and GPT models through a compatible API layer, 59API is worth a look. Sign up, point your client at https://api.59api.com, and build the retry wrapper before your traffic grows.

Pricing · Docs