7 Resilient Patterns for LLM API Calls
7 Resilient Patterns for LLM API Calls
LLM calls fail differently from ordinary HTTP requests. A response can take tens of seconds, a rate limit can clear shortly, and retrying a non-idempotent request can duplicate an expensive action. The goal is not to retry everything. The goal is to recover from temporary failures while protecting latency, token spend, and downstream systems.
1. Set a Deadline, Not Just a Timeout
Use two limits: a per-attempt timeout and a total request deadline. For example, allow each model attempt 30 seconds, but stop the entire workflow after 75 seconds. This prevents three retries from turning one slow user request into a two-minute wait.
Choose the attempt timeout from expected model behavior. Short classification or extraction prompts may need 10 to 20 seconds. Large-context reasoning, tool use, or long streaming completions need a longer budget. Include connection, upload, queueing, and response time in the deadline. For streaming responses, also enforce an idle timeout so a stalled stream cannot remain open forever.
2. Retry Only Failures That Can Recover
Retry network connection resets, DNS failures, HTTP 408, HTTP 429, and most HTTP 5xx responses. These commonly represent transient network, capacity, or rate-limit conditions. Do not automatically retry malformed requests, authentication failures, invalid model names, safety-policy rejections, or context-window errors. Those are usually deterministic and need a code or prompt change.
Treat HTTP 409 and provider-specific errors according to their documented meaning. Record the status code and error type before deciding. A broad “retry on any exception” rule hides real integration defects and increases cost.
3. Use Exponential Backoff With Full Jitter
A fixed five-second retry interval causes synchronized clients to retry together, creating another traffic spike. Start with exponential backoff, such as 1, 2, 4, and 8 seconds, then add full jitter. For each retry, select a random delay between zero and the current backoff cap.
- Attempt 1: retry after a random delay from 0 to 1 second.
- Attempt 2: retry after a random delay from 0 to 2 seconds.
- Attempt 3: retry after a random delay from 0 to 4 seconds.
- Stop: after two or three retries, or earlier when the total deadline expires.
If the API sends a Retry-After header, respect it unless it exceeds your total deadline. Rate-limit guidance from the server is more useful than a generic local schedule.
4. Make Retries Idempotent
Generating text is usually safe to retry if the first attempt failed before a usable response reached your application. Actions triggered by model output are different. A retry must not send two emails, create two tickets, charge twice, or write duplicate database records.
Generate a request ID at the start of the workflow. Store it with the intended action and use a database uniqueness constraint or idempotency key before performing side effects. For tool-using agents, separate generation from execution: save the proposed tool call, validate it, then execute it once under an idempotent operation key.
5. Cap Work Before Sending It
Retries amplify token usage. Before each attempt, cap prompt size, output tokens, tool-call count, and parallel fan-out. If a long prompt times out, retrying the exact same oversized request may simply fail again. Consider a fallback strategy: shorten retrieved context, request a structured summary first, or route routine work to a smaller model.
59API is useful here because it provides low-cost, pay-as-you-go access to Claude models including Opus, Sonnet, Haiku, and Fable, plus GPT models, through https://api.59api.com. Its native official-quality models let teams choose a capable model for complex jobs and a lower-cost option for routine retries without accepting a model downgrade.
6. Add a Circuit Breaker and Fallback Path
When failures cross a threshold, temporarily stop sending traffic to the failing route. A practical circuit breaker might open after five failures in one minute, wait 30 seconds, then allow a single probe request. During that period, return a clear temporary-error state, serve a cached result, queue the job, or use an approved alternative model.
Do not silently change models for tasks where quality, safety behavior, or output format is contractual. Make fallback eligibility explicit per workflow, and test it with production-like prompts.
7. Measure Every Attempt
Log a correlation ID, model, attempt number, status code, retry delay, time to first token, total latency, input tokens, output tokens, and final outcome. Build alerts around error rate, p95 latency, 429 frequency, and retry exhaustion. These metrics reveal whether the fix is a larger timeout, less concurrency, smaller prompts, or a provider-route issue.
Keep credentials in environment variables, centralize retry logic in one client wrapper, and configure the same policy for background jobs and user-facing requests. 59API is compatible with Claude Code, Codex, and OpenAI SDK workflows, so a centralized base URL configuration can keep that resilience policy consistent across tools. Sign up for 59API when you want a low-cost relay option with pay-as-you-go pricing and a referral rebate.