← सभी गाइड पर लौटें

LLM Retry and Backoff Patterns That Work

LLM Retry, Timeout and Backoff Patterns That Work

Large language model calls are network calls to busy, rate-limited services. Even when the model is excellent, your application still needs to handle slow responses, temporary overload, rate limits, and occasional connection failures. This tutorial shows a practical pattern you can add to production LLM apps: request timeouts, safe retries, exponential backoff, jitter, and clear failure handling.

The examples apply whether you call GPT models, Claude Sonnet, Claude Opus, Haiku, or Fable. If you use 59API, the same approach works with OpenAI-compatible SDKs and Claude Code or Codex integrations by pointing your base URL to https://api.59api.com. Because 59API is a low-cost, pay-as-you-go relay using official-quality native models without downgrade, it is a good place to build reliable LLM workflows without letting retry traffic destroy your budget.

Step 1: Decide Which Failures Are Retryable

Do not retry every error. Some failures are permanent and retries only waste money. Before writing code, classify errors into retryable and non-retryable categories.

• Retry: HTTP 408 request timeout, 429 rate limit, 500 internal server error, 502 bad gateway, 503 unavailable, 504 gateway timeout, temporary connection reset, DNS hiccup.
• Do not retry: 400 bad request, 401 invalid API key, 403 forbidden, 404 wrong endpoint, context length exceeded, invalid model name, malformed JSON schema.
• Maybe retry once: streaming interrupted after partial output, tool-call parsing failure, or empty response from an otherwise valid request.

This classification prevents retry storms and makes debugging easier. Log the status code, request ID if available, model, attempt number, and final error reason.

Step 2: Add a Hard Timeout Per Attempt

A timeout is not the same as a retry. The timeout limits how long one attempt can run. Without it, a worker can hang, queues can fill, and users can wait forever. Choose timeouts based on the task size.

• Short chat or classification: 10 to 20 seconds.
• Code generation or long reasoning: 45 to 120 seconds.
• Background batch jobs: longer timeouts are acceptable, but still set a maximum.

For user-facing apps, prefer a shorter timeout and show a friendly retry option. For background jobs, you can afford more attempts, but always cap the total runtime.

Step 3: Use Exponential Backoff With Jitter

Backoff means waiting before the next retry. Exponential backoff increases the wait after each failure: for example 500 ms, 1 second, 2 seconds, 4 seconds. Jitter adds randomness, such as waiting anywhere between 0 and the calculated delay. This stops many clients from retrying at the exact same moment after a shared outage.

A good default pattern is three attempts total, starting with a 500 ms delay, doubling each time, with a maximum delay of 8 seconds. For 429 responses, honor a Retry-After header if the API provides one. If no header exists, use your normal jittered backoff.

Step 4: Keep Requests Idempotent When Possible

Retries are safest when repeating the request does not create duplicate side effects. For plain chat completions, retrying is usually fine. For tool calls, database writes, payments, emails, ticket creation, or workflow actions, add an idempotency key or store state before executing side effects.

A common pattern is to separate the LLM planning step from the execution step. First ask the model for structured output. Then validate it. Only after validation should your application perform external actions. If the LLM call is retried, it only regenerates the plan rather than duplicating the action.

Step 5: Implement a Simple Retry Wrapper

Your production code should centralize retry logic instead of scattering it across the app. The wrapper should accept a function that performs the LLM call, apply a per-attempt timeout, inspect errors, wait with backoff, and return either the successful result or a clean final error.

• maxAttempts: usually 3 for interactive apps, 5 for background jobs.
• attemptTimeoutMs: task-specific timeout such as 30000.
• baseDelayMs: start around 500.
• maxDelayMs: cap at 8000 or 15000.
• retryableStatusCodes: 408, 429, 500, 502, 503, 504.

If you use an OpenAI-compatible SDK with 59API, configure the client with your API key and base URL https://api.59api.com, then call your wrapper around the normal chat completion request. The benefit is that your application code stays portable while you access Claude and GPT models through one inexpensive relay.

Step 6: Budget for Retries

Retries improve reliability but can multiply token usage. If one failed request is retried twice, that single user action may consume three prompts. Reduce cost by setting a smaller maximum output token count for retries, caching deterministic prompts, and using cheaper models for fallback tasks.

For example, if Claude Opus or a top GPT model times out during a non-critical summarization task, your fallback can try Claude Sonnet, Haiku, or another lower-cost model. With 59API, this kind of model routing is especially practical because it gives pay-as-you-go access to multiple Claude and GPT options, plus referral rebates that can offset experimentation and scaling costs.

Step 7: Add Fallbacks and User-Friendly Errors

After all retries fail, do not expose raw stack traces. Return a clear message such as: The AI service is temporarily busy. Please try again in a moment. For business-critical workflows, enqueue the request for later processing or switch to a fallback model.

• Interactive chat: retry quickly, then ask the user to resubmit.
• Batch processing: retry with longer delays and store failed jobs for replay.
• Agents and tools: stop safely, persist state, and resume from the last completed step.

Step 8: Monitor the Right Metrics

You cannot tune retries without data. Track first-attempt success rate, final success rate after retries, average latency, p95 latency, retry count by model, timeout count, 429 count, and cost per successful request. If retries are frequent, the issue may be too-short timeouts, overloaded models, prompt sizes that are too large, or insufficient rate limits.

Start with conservative settings, measure for a week, then adjust. Reliable LLM engineering is not about retrying forever; it is about failing gracefully, controlling cost, and recovering from temporary problems. If you want a low-cost way to test these patterns across Claude and GPT models with familiar SDKs, consider signing up for 59API and using https://api.59api.com as your API base URL.

मूल्य · दस्तावेज़