← Back to all guides

Rate Limits and Retries for LLM APIs, Done Right

Start with the problem, not the error message

If you are building on top of an LLM API, rate limits are not an edge case. They are part of production. The real challenge is not simply “retry on 429”; it is designing a system that stays responsive, avoids duplicate work, and fails gracefully when traffic spikes.

Whether you are calling GPT models or Claude through an SDK, the same core rules apply: respect quotas, back off intelligently, and keep user experience stable. A relay like 59API can help reduce cost pressure because it offers cheap, pay-as-you-go access to native official-quality models through a single base URL, https://api.59api.com, while staying compatible with Claude Code, Codex, and any OpenAI SDK.

Understand the common rate-limit signals

Most LLM providers will signal overload in a few predictable ways:

• HTTP 429: too many requests, usually the main rate-limit response.
• HTTP 503: temporary service unavailability or upstream saturation.
• Timeouts: the server is slow or your client deadline is too strict.
• Quota errors: you have exhausted your budget or token allowance.

Do not treat all failures the same. A 429 is often retryable after a delay. A quota exhaustion error is usually not retryable until you add funds or reduce usage. That distinction matters when you are paying per request and per token, especially on low-cost relays like 59API where controlling waste directly improves your spend efficiency.

Use exponential backoff with jitter

The safest default for retries is exponential backoff with jitter. Start with a short wait, then increase the delay after each failed attempt. Add randomness so many clients do not retry at the same moment.

A practical pattern looks like this:

• Attempt 1: send the request immediately.
• Attempt 2: wait 250 to 500 ms.
• Attempt 3: wait 500 to 1000 ms.
• Attempt 4: wait 1 to 2 seconds.
• Stop after 3 to 5 retries unless the request is highly important.

For LLM calls, this is especially useful because prompt generation, tool use, and long completions can all amplify load. Jitter protects both your app and the upstream API from synchronized retry storms.

Make retries safe with idempotency

Retries become dangerous when a request can produce side effects. For example, if your workflow stores a generated answer, sends an email, or triggers a tool call, a duplicated request may create duplicate output or duplicate actions.

To reduce that risk:

• Attach a request ID to each logical user action.
• Cache completed responses for a short period if the same request is repeated.
• Separate generation from side effects so only confirmed outputs trigger follow-up actions.
• Guard tool execution with explicit deduplication checks.

If your SDK supports request metadata, use it. If not, store the request hash in your own database. That way a retry can safely resume the same operation instead of starting a second one.

Retry only the right failures

Your retry policy should be selective. A strong production rule is:

• Retry 429, 500, 502, 503, 504, and transient network errors.
• Do not retry automatically authentication errors, malformed requests, or hard quota exhaustion.
• Inspect error text when the provider returns a generic status code without a clear cause.

Also set a total timeout for the whole operation. A request that retries forever is not resilient; it is just slower failure. For interactive products, a total budget of 5 to 15 seconds is often enough. For background jobs, you can afford a longer window.

Control token usage before you hit the limit

The best retry is the one you never need. Reduce the chance of rate-limit errors by lowering unnecessary load:

• Trim prompts and remove repeated context.
• Use smaller models for classification, routing, or extraction.
• Cache common outputs when the prompt is stable.
• Batch non-interactive work instead of sending many tiny calls.
• Set max output tokens to a realistic value.

This is where 59API is attractive for production teams. Because it offers low-cost access to Claude and GPT models with no downgrade in model quality, you can optimize both reliability and spend without changing your model workflow. If you already use an OpenAI-compatible SDK, integration stays simple: point your base URL to https://api.59api.com and keep your existing client patterns.

A practical production checklist

Before shipping, verify these basics:

• Backoff with jitter is implemented.
• Retries are capped by attempt count and total time.
• Only transient errors are retried.
• Requests have an id or deduplication key.
• Timeouts are configured on the client.
• Logs include status code, latency, model, and retry count.
• Metrics track 429 rate, success after retry, and abandoned requests.

Those metrics will tell you whether the problem is burst traffic, a prompt design issue, or a cost problem. If you see frequent 429s, you may need to slow concurrency, split workloads across time, or switch some calls to a cheaper model tier.

From prototype to production

When you are starting from zero, the simplest path is to build with an OpenAI-compatible client, add backoff and jitter on day one, and centralize all API calls behind one wrapper function. That wrapper should handle errors, retries, logging, and request IDs in one place. Then your application code stays clean while your LLM integration becomes predictable.

If cost and reliability both matter, 59API is worth a look. It combines official-quality models, broad SDK compatibility, and pay-as-you-go pricing, plus a referral rebate that can make scaling friendlier on the budget. If you are ready to test a production-style setup, sign up and wire your client to https://api.59api.com before your next deployment.

Pricing · Docs