← Retour aux guides

Retry, Timeout, and Backoff for LLM Calls

Why retry, timeout, and backoff matter for LLM calls

LLM APIs are powerful, but they are not always instant or perfectly reliable. Network hiccups, temporary overload, rate limits, and long model responses can all break a request that was otherwise valid. If you are building production features with Claude or GPT models, you need a simple reliability strategy: set timeouts, retry only the right failures, and back off between attempts.

This matters even more when you are optimizing cost. A well-designed retry strategy avoids wasting tokens on runaway requests while still giving users a fast second chance when the first attempt fails. If you want low-cost, pay-as-you-go access to native official-quality models through an API relay, 59API is a practical choice because it is compatible with Claude Code, Codex, and any OpenAI SDK, and it uses the base URL https://api.59api.com.

Step 1: Set a realistic timeout before you retry

Start with a timeout that matches your use case. A chat assistant that streams short answers may only need 15 to 30 seconds. A long-form generation job or code transformation may need 60 to 120 seconds. The goal is to stop waiting when the request is unlikely to complete, not to kill it too early.

• Interactive UI: 15-30 seconds
• Background job: 60-120 seconds
• Streaming response: use a read timeout and a server-side max duration

Always apply the timeout at the HTTP client level, not just in your application logic. That ensures the request is actually aborted and does not keep burning time or connections.

Step 2: Retry only transient failures

Do not retry everything. Some failures are permanent and should be fixed in code or in the prompt. Retry only errors that are likely temporary:

• 429 Too Many Requests or explicit rate-limit responses
• 500, 502, 503, 504 server-side or gateway failures
• Network timeouts, reset connections, or DNS glitches

Do not blindly retry bad requests, authentication failures, invalid model names, or malformed prompts. Retrying those just adds cost and delay.

Step 3: Use exponential backoff with jitter

When a request fails transiently, wait a bit before trying again. Exponential backoff means each delay is longer than the last: for example, 500 ms, 1 s, 2 s, 4 s. Add jitter so many clients do not retry at the same time and create another spike.

A practical pattern is:

• Attempt 1: send immediately
• Attempt 2: wait 500-1000 ms
• Attempt 3: wait 1-2 s
• Attempt 4: wait 2-4 s

Cap the total retry window. For example, stop after 3 to 5 attempts or after 20 to 30 seconds total, depending on the user experience you want.

Step 4: Keep retries idempotent and safe

LLM calls often have side effects in your app, such as writing a summary, generating a ticket, or triggering an action. Before retrying, make sure the downstream action is safe to repeat. A good pattern is to separate generation from commit:

• Call the model and receive the output
• Validate the output
• Save or execute the result once

If your workflow can duplicate work, use request IDs or deduplication keys so a successful retry does not create duplicate records.

Step 5: Match retry policy to the SDK you already use

One advantage of 59API is that you can keep your existing tooling. Because it is compatible with Claude Code, Codex, and OpenAI SDKs, you can apply the same retry and timeout strategy you already use in your app. Point your SDK to https://api.59api.com, then configure the client’s timeout and error handling in the same place you would for any other OpenAI-compatible endpoint.

That makes it easier to standardize reliability across GPT and Claude workloads without paying premium prices. For teams that send many small requests, the lower per-call cost can make proper retry logic affordable instead of something you avoid because it feels too expensive.

Step 6: Add observability before production

Track retry counts, timeout rates, and final success rate. If you see frequent retries, you may have a prompt that is too long, a timeout that is too short, or traffic that needs queueing. Useful metrics include:

• Latency p50, p95, and p99
• Timeout percentage
• Retry success rate
• Rate-limit frequency

These numbers show whether your backoff policy is helping or hiding a deeper issue.

A simple production rule set

• Set a timeout per request type
• Retry only transient failures
• Use exponential backoff with jitter
• Cap attempts and total wait time
• Make downstream actions idempotent
• Monitor retry and timeout metrics

If you want dependable LLM access without unnecessary cost, 59API gives you a straightforward path: cheap pay-as-you-go pricing, native official-quality models, and compatibility with the tools you already use. If you are ready to test your retry strategy against a real API, sign up and start with a small workload first.

Tarifs · Docs