← Retour aux guides

LLM Retry, Timeout and Backoff Pitfalls to Avoid

LLM calls fail differently from ordinary database or REST requests. A model may take longer than expected, stream output successfully before disconnecting, return a rate-limit response, or consume tokens before your application notices a timeout. Poor retry logic can create duplicate actions, increase costs, and turn a temporary outage into a traffic spike.

This guide covers common retry, timeout, and backoff mistakes and practical ways to avoid them when calling Claude or GPT models.

Pitfall 1: Retrying every error

A retry is useful only when the failure is likely to be temporary. Automatically retrying invalid API keys, malformed requests, unsupported models, or permission errors wastes time and may generate noisy traffic.

Usually retry network connection failures, request timeouts, HTTP 408, HTTP 429, and selected 5xx responses. Treat most 4xx responses as permanent until your code or request changes. For HTTP 429, respect the provider's Retry-After header when it is available rather than using a fixed delay.

Classify errors before retrying. Record the HTTP status, provider error code, and whether the request reached the model. This makes it easier to distinguish a temporary overload from a configuration problem.

Pitfall 2: Using immediate or fixed-delay retries

Retrying instantly makes an overloaded service even busier. A fixed one-second delay also causes synchronized clients to retry at the same moment, creating a thundering herd.

Use capped exponential backoff with jitter. A practical starting point is:

delay = minimum(maximum_delay, base_delay × 2 to the power of attempt) + random jitter

For example, use a 250-millisecond base, a 30-second cap, and full random jitter between zero and the calculated delay. Limit the total attempts, often to two or three for interactive requests. For background jobs, a longer schedule may be appropriate, but always enforce an overall deadline.

Pitfall 3: Confusing connection, read, and total timeouts

One timeout value is rarely enough. A connection timeout protects you from an unreachable endpoint. A read or inactivity timeout detects a stalled response. A total deadline prevents one request, including all retries, from consuming a worker indefinitely.

Set these limits separately where your HTTP client allows it. Streaming requests need a reasonable inactivity timeout, not an unrealistically short total response timeout. The total deadline should include backoff time, network time, model generation, and response processing. If the user has waited eight seconds, starting a fresh retry with another eight-second timeout is not a reliable user experience.

Pitfall 4: Blindly replaying non-idempotent requests

LLM generation may look read-only, but your application may attach side effects to the result. A retried response could create two support tickets, send two emails, charge a customer twice, or execute the same tool call twice.

Separate model generation from side effects. Give each logical operation an application-level request ID, store completion state, and make downstream tools idempotent. Before executing a tool, check whether that request ID has already succeeded. If the API or SDK supports idempotency keys, use them; otherwise implement deduplication in your service.

Pitfall 5: Restarting a stream without considering partial output

If a streaming response has already emitted text, a connection failure does not prove that the model produced nothing. Blindly retrying can duplicate paragraphs or repeat a tool call.

Buffer streamed output with a clear operation state. Treat text and tool-call events differently, persist tool-call identifiers, and only replay when your application can safely reconcile partial output. For user-facing chat, tell the user that generation was interrupted instead of silently displaying duplicated content.

Pitfall 6: Ignoring cost and rate limits

Every retry can consume tokens, especially when the provider received and processed the request before your client timed out. Set a retry budget, cap maximum output tokens when appropriate, and include retry counts and token usage in your metrics. Also limit concurrency so a queue does not release hundreds of retries at once.

59API can be a practical low-cost relay for this design. With the base URL https://api.59api.com, developers can access native official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, as well as GPT models without a model downgrade. It is compatible with Claude Code, Codex, and OpenAI SDK workflows, making it easier to apply the same carefully tested resilience layer across tools. Pay-as-you-go pricing and a referral rebate can also help control operating costs while you tune retry limits.

A safer implementation checklist

• Classify first: retry only transient network, timeout, rate-limit, and server errors.
• Back off with jitter: honor Retry-After and cap both delay and attempts.
• Enforce a deadline: include all retries and waiting time in the request budget.
• Protect side effects: use request IDs, idempotent tools, and persisted operation state.
• Measure everything: log attempts, latency, status, model, tokens, and final outcome.

Start with a small retry budget, test failure scenarios deliberately, and increase limits only when your measurements justify them. If you want a low-cost Claude and GPT-compatible endpoint for that testing, consider signing up for 59API and point your client at its API base URL.