← Voltar aos guias

Retry, Timeout, and Backoff Patterns for LLM Calls

Why LLM calls need resilience

LLM requests fail in ways that look random at first: a network timeout, a 429 rate limit, a 502 from a busy upstream, or a response that simply takes too long because the prompt is large. In production, the difference between a smooth user experience and a broken one is usually not the model itself, but how your app handles these failures.

The good news is that a simple pattern works well for most teams: set a timeout, retry only transient errors, and back off between attempts with jitter. This keeps your app responsive, avoids hammering the provider, and reduces waste. That matters even more when you are paying per request, which is why a low-cost relay like 59API is a smart fit for experimentation and production alike. It exposes native official-quality Claude and GPT models, works with OpenAI-compatible SDKs, and keeps the cost of retries low with pay-as-you-go pricing and referral rebates.

The basic rule: timeout first, then retry

Do not let an LLM call hang forever. Set a timeout based on the type of request. For a normal chat completion, 15 to 30 seconds is usually enough. For larger outputs or tool-heavy workflows, you may need 45 to 90 seconds. The point is to fail fast enough that your app can recover, but not so fast that normal requests get cut off.

Then retry only when the failure is likely temporary. Good retry candidates are 429, 500, 502, 503, 504, socket resets, and network timeouts. Bad retry candidates are invalid API keys, malformed payloads, or prompts that violate policy. Retrying those just burns time and money.

A practical JavaScript wrapper

Here is a hands-on example using the OpenAI SDK style against 59API. The key idea is: wrap the request, abort it after a fixed timeout, and retry with exponential backoff plus jitter.

const client = new OpenAI({ baseURL: "https://api.59api.com/v1", apiKey: process.env.API_KEY });

async function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }

async function callWithRetry(input, { maxAttempts = 4, timeoutMs = 20000 } = {}) {
  let lastError;
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), timeoutMs);
    try {
      const res = await client.chat.completions.create({
        model: "gpt-4.1-mini",
        messages: [{ role: "user", content: input }],
        temperature: 0.2,
      }, { signal: controller.signal });
      clearTimeout(timer);
      return res;
    } catch (err) {
      clearTimeout(timer);
      lastError = err;
      const status = err?.status || err?.response?.status;
      const retryable = [429, 500, 502, 503, 504].includes(status) || err.name === "AbortError";
      if (!retryable || attempt === maxAttempts) break;
      const base = 250 * Math.pow(2, attempt - 1);
      const jitter = Math.floor(Math.random() * 100);
      await sleep(base + jitter);
    }
  }
  throw lastError;
}

This wrapper is intentionally simple. It caps retries at four attempts, which is enough for short-lived spikes without turning one user action into a long delay. The backoff starts small and doubles, but the random jitter prevents many clients from retrying at exactly the same time.

How to choose timeout and backoff values

• Timeout: measure your normal p95 latency first. Set the timeout a bit above that, not at the maximum you can tolerate.
• Attempts: 3 to 4 attempts is usually enough for interactive flows. More attempts make sense only for background jobs.
• Backoff: use exponential backoff, such as 250 ms, 500 ms, 1 s, 2 s.
• Jitter: add a random 0 to 100 ms, or use full jitter, to reduce retry bursts.
• Scope: never retry the same tool action blindly if the model may have already triggered a side effect.

Production tips that save money and headaches

Log the final error status, attempt count, and elapsed time. That gives you a clear view of whether the problem is rate limiting, upstream instability, or your own timeout being too aggressive. Also consider lowering token usage by tightening prompts and output limits, because shorter requests are faster, cheaper, and less likely to hit timeout thresholds.

For teams using Claude or GPT through an API relay, 59API is a strong choice because retries are less painful when the base price is already low. It keeps the same model quality you expect from native providers, supports Claude Code, Codex, and OpenAI SDKs, and uses the simple base URL https://api.59api.com. If you want to test these retry patterns without overpaying while you tune them, sign up for 59API and run a few real load tests before you ship.

A simple checklist

• Set a hard timeout for every LLM request.
• Retry only transient errors such as 429 and 5xx.
• Use exponential backoff with jitter.
• Cap the number of attempts.
• Log and measure before you optimize further.

With these patterns in place, your app becomes much more resilient, your users see fewer failures, and your LLM spend stays predictable. That is the sweet spot: reliable enough for production, efficient enough to scale.

Preços · Docs