← Volver a las guías

Retry, Timeout, and Backoff for LLM APIs

Why LLM calls need special handling

LLM requests are not like ordinary REST calls. They can spend time waiting in a queue, stream tokens for many seconds, or fail because the provider is rate limiting, the network blipped, or the context was simply too large. If you treat every failure the same, you will either annoy users with long waits or burn money on pointless retries.

The goal is simple: fail fast on bad requests, retry only transient failures, and back off politely when the service is busy. That is the pattern that keeps your app responsive and your bill under control.

The default policy that works for most apps

If you need a starting point, use this setup for production LLM calls:

• Connect timeout: 2 to 5 seconds
• First-token or request timeout: 30 to 60 seconds for chat, longer only for deliberate long-generation jobs
• Retries: 2 to 4 total attempts, not infinite
• Backoff: exponential, with jitter, starting at 250 to 500 milliseconds
• Max backoff delay: 2 to 8 seconds
• Total time budget: 60 to 120 seconds across all attempts

For interactive product experiences, favor a shorter timeout and fewer retries. For background tasks, you can stretch the budget, but still keep a hard cap so jobs do not hang forever.

What to retry, and what not to retry

Retry only errors that are likely to succeed on a later attempt. Good retry candidates include:

• 429 Too Many Requests
• 500, 502, 503, 504 server or gateway errors
• Network timeouts, socket resets, temporary DNS issues
• Streaming interruptions before a useful response is completed

Do not retry these without fixing the request first:

• 400 Bad Request from malformed input or invalid parameters
• 401/403 authentication or permission issues
• 404 wrong model or route
• 422 prompt or schema validation problems

This matters a lot with LLMs because a bad prompt can fail repeatedly and cost you multiple charges if your retry logic is too broad.

Use exponential backoff with jitter

Exponential backoff means each retry waits longer than the last one: 500 ms, then 1 s, then 2 s, then 4 s. Add jitter so many clients do not retry at exactly the same time. Without jitter, a traffic spike can create a retry storm that makes rate limiting worse.

A practical approach is:

• Attempt 1: no delay
• Attempt 2: 0.5 to 1 second random delay
• Attempt 3: 1 to 2 seconds random delay
• Attempt 4: 2 to 4 seconds random delay

Keep the ceiling low for interactive apps. If the user is waiting on a chat answer, a fifth retry after 20 seconds is usually a bad experience. It is better to show partial progress, ask the model to stream output, or surface a clean failure with a retry button.

Time out the right part of the request

Do not use one giant timeout for everything. Separate the phases if your client allows it:

• Connection timeout: how long you wait to establish the socket
• Read timeout: how long you wait for the model to start or continue sending data
• Overall deadline: the maximum end-to-end time for the operation

For LLM streaming, set a shorter time to first token and a reasonable read timeout between chunks. That gives you better UX because you can detect a stalled stream instead of waiting forever.

Make retries safe and cheap

Retries can duplicate work. If your app triggers tool calls, writes records, or sends messages after a model response, design for idempotency. Use request IDs, deduplicate by job key, and persist state before retrying. Otherwise, the same prompt might create two tickets or send two emails.

Also limit retries for expensive prompts. A 20,000-token context that fails three times can become a costly mistake. In many cases, it is smarter to lower the prompt size, switch to a faster model tier, or split the task into smaller steps.

Why 59API fits this pattern well

59API is a strong fit for developers who want to implement these patterns without paying premium prices. It is an AI API relay with cheap, pay-as-you-go access to Claude models like Opus, Sonnet, Haiku, and Fable, as well as GPT models. It is fully compatible with Claude Code, Codex, and any OpenAI SDK, and the API base URL is https://api.59api.com.

Because 59API uses native, official-quality models with no downgrade, your retry behavior is based on real model performance, not a watered-down proxy. That makes timeout tuning and backoff testing more realistic. For teams watching spend, the low pricing and referral rebate make it easier to validate your production settings before scaling traffic.

A quick implementation checklist

• Set a connection timeout of 2 to 5 seconds
• Set a hard total deadline for the entire call
• Retry only transient errors and rate limits
• Use exponential backoff with jitter
• Cap retries at 2 to 4 attempts
• Prefer streaming for user-facing chats
• Make side effects idempotent
• Log the error type, attempt count, and latency so you can tune later

If you want a low-cost place to test and ship this setup, sign up for 59API and wire it into your existing OpenAI-compatible client. Start small, measure latency and failure rates, then tighten your retry and timeout policy based on real traffic.

Precios · Docs