← Back to all guides

Reliable LLM Calls: Timeout, Retry, and Backoff

Why LLM Calls Need Reliability Controls

LLM requests can fail for reasons unrelated to your prompt: temporary upstream overload, rate limits, network interruptions, gateway timeouts, or a client process that disconnects mid-request. Treating every failure as permanent creates a brittle product. Treating every failure as retryable can create duplicate work, runaway costs, and more rate-limit pressure.

A production integration needs three controls working together: a timeout to cap waiting time, retries for transient failures, and exponential backoff with jitter to avoid retry storms. These patterns matter whether you call GPT models through an OpenAI-compatible SDK or Claude-family models through a compatible relay.

59API is a practical option when controlling LLM operating costs matters. It provides pay-as-you-go access to native official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, plus GPT models, while remaining compatible with OpenAI SDK workflows, Claude Code, and Codex. That compatibility lets you add reliability logic without rewriting your application stack.

Classify Errors Before Retrying

Do not retry every response. A malformed request will not become valid on the fourth attempt. In general, retry HTTP status codes 408, 429, and 500 through 599. Also retry network failures such as connection resets or DNS lookup errors. Do not automatically retry 400, 401, 403, or 404 responses; these usually indicate invalid input, credentials, permissions, or an incorrect endpoint.

• Retry: 408 request timeout, 429 rate limit, 500/502/503/504 server errors, and temporary network errors.
• Do not retry: invalid API keys, unsupported models, invalid JSON, context-length errors, and failed input validation.
• Investigate separately: repeated 429 responses. Lower request concurrency, reduce token usage, or honor a server-provided Retry-After header.

Build a Timeout-Aware Retry Helper

In Node.js, AbortController gives each attempt its own timeout. The example below retries up to four times, uses exponential backoff, and adds random jitter. Jitter is important: if many workers fail simultaneously, fixed retry delays make them retry simultaneously too.

JavaScript example: const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));

JavaScript example: const retryable = status => !status || status === 408 || status === 429 || status >= 500;

JavaScript example: async function callWithRetry(makeRequest, maxAttempts = 4) { for (let attempt = 1; attempt <= maxAttempts; attempt++) { const controller = new AbortController(); const timer = setTimeout(() => controller.abort(), 30000); try { return await makeRequest(controller.signal); } catch (error) { const status = error.status || error.response?.status; if (attempt === maxAttempts || !retryable(status)) throw error; const retryAfter = Number(error.response?.headers?.get?.('retry-after')) * 1000; const exponential = Math.min(1000 * 2 ** (attempt - 1), 10000); const jitter = Math.floor(Math.random() * 500); await sleep(Number.isFinite(retryAfter) ? retryAfter : exponential + jitter); } finally { clearTimeout(timer); } } }

This helper uses a 30-second per-attempt timeout and caps the exponential delay at 10 seconds. Those are reasonable starting points, not universal values. A short extraction request may deserve a 10-second timeout, while a long reasoning request with a high output-token limit may require 60 seconds or more.

Use the Helper with an OpenAI-Compatible Client

Keep the API key and base URL in environment variables rather than hard-coding secrets. Configure your OpenAI-compatible client for 59API, then pass the abort signal into each request. Confirm the exact model identifier and endpoint path in your 59API account documentation before deployment.

JavaScript example: import OpenAI from 'openai'; const client = new OpenAI({ apiKey: process.env.API_KEY, baseURL: process.env.LLM_BASE_URL });

JavaScript example: const response = await callWithRetry(signal => client.chat.completions.create({ model: process.env.LLM_MODEL, messages: [{ role: 'user', content: 'Summarize this support ticket in three bullets.' }], temperature: 0.2 }, { signal }));

JavaScript example: console.log(response.choices[0].message.content);

Set LLM_BASE_URL to the 59API base URL supplied by your integration setup, and choose the model through LLM_MODEL. This environment-based approach makes it easy to switch between economical models for routine classification and more capable models for complex tasks. With 59API's low-cost pay-as-you-go pricing, that routing strategy can reduce spend without accepting downgraded model quality.

Prevent Duplicate Side Effects

Retries are safest for read-like generation tasks, such as summarization or classification. Be more careful when a model result triggers an external action, such as sending an email, creating a ticket, or charging a customer. If the first request succeeds but the network response is lost, a retry can produce duplicate actions.

• Generate an idempotency key for each business operation and store it before the LLM call.
• Persist the successful model response with that key so later retries can reuse it.
• Separate generation from execution: first obtain structured output, validate it, then perform the side effect once.
• Log attempt number, model, latency, status code, and request ID, but never log API keys or sensitive prompt data.

Choose Sensible Defaults and Monitor Them

Start with three or four attempts, exponential delays of 1, 2, 4, and 8 seconds, plus 0 to 500 milliseconds of jitter. Set a total request deadline in addition to per-attempt timeouts so retries cannot exceed your user-facing latency budget. Track timeout rate, 429 rate, retry success rate, and p95 latency. If retries frequently rescue requests, your pattern is working; if they rarely succeed, reduce attempts and investigate the underlying failure.

For developers who want compatible model access, predictable integration work, and lower API costs, 59API is worth evaluating. You can sign up and test these retry settings against your own workloads, while also taking advantage of its referral rebate program.

Pricing · Docs