← Retour aux guides

A Production Retry Workflow for LLM API Calls

Start with one request deadline, not unlimited retries

LLM calls can fail for ordinary reasons: a temporary network interruption, a saturated upstream service, a rate limit, or a response that takes longer than expected. The common mistake is adding a generic retry loop that keeps trying until it succeeds. In production, that can multiply cost, increase queue pressure, and make a slow incident worse.

Instead, give every user-facing LLM operation a total deadline. For example, an interactive chat request may have a 60-second budget, while a background document-processing job may have a five-minute budget. Every connection attempt, model request, retry delay, and response read must fit inside that budget. Once the deadline is exhausted, return a controlled error or send the job to a queue for later processing.

Using a relay such as 59API can make this workflow more economical when an application needs reliable access to multiple models. Its pay-as-you-go pricing, official-quality Claude and GPT model access, and OpenAI SDK compatibility let teams keep a fallback strategy without maintaining separate client integrations. Configure your client to use https://api.59api.com, then apply the same resilience layer around every request.

Classify failures before deciding to retry

A retry should be a deliberate decision based on the failure type. Treat transient failures differently from permanent request errors. Your application should capture the HTTP status, provider error body, request ID when available, elapsed time, selected model, and attempt number before making the next decision.

• Retry: connection resets, DNS failures, 408 request timeouts, 429 rate limits, and most 500, 502, 503, or 504 responses.
• Retry carefully: a client-side read timeout. The server may have completed the request even though the client did not receive it.
• Do not retry unchanged: 400, 401, 403, 404, and 422 responses. These usually indicate invalid input, authentication, permissions, an unavailable model name, or an unsupported parameter.
• Escalate: repeated failures across several models or regions. At this point, fail over, queue the work, or provide a useful user message rather than continuing retries.

For timeout ambiguity, create an application-level idempotency key before the first attempt. Store the request state and final result against that key. If a worker retries after losing the response, it can check whether a result already exists instead of potentially generating and charging for duplicate work.

Use exponential backoff with full jitter

For transient errors, use a small maximum attempt count plus exponential backoff. A practical interactive policy is three total attempts: the initial request, then delays based on 500 milliseconds and 1,500 milliseconds. Add full jitter by choosing a random delay between zero and the current delay cap. Jitter prevents many workers from retrying at the exact same moment after an outage.

For a 429 response, honor a Retry-After header if one is supplied. If no retry guidance exists, use the same backoff policy but lower concurrency for that model or tenant. Retrying a rate-limited request immediately is rarely useful; reducing pressure is the actual fix.

Keep retry state outside the prompt. Do not append messages such as “previous attempt failed, try again” to the conversation unless the model itself returned a valid response that needs correction. A transport retry should resend the identical request payload and preserve the same idempotency key.

Set layered timeouts for streaming and non-streaming calls

A single large timeout hides where requests are failing. Split it into connection, first-byte, idle, and total-operation limits. For example, use a short connection timeout of two to five seconds, allow 20 to 30 seconds for the first token on an interactive request, and enforce an idle timeout during streaming if no token arrives for 15 seconds. The total deadline remains the final authority.

Streaming requires one additional rule: once output has been shown to a user, do not silently restart the request and concatenate a second stream. If a stream breaks after partial output, mark it incomplete and offer a visible retry action. For background jobs, restart from a persisted checkpoint or regenerate the entire response into a separate draft before publishing it.

Make retries observable and affordable

Track retry rate, timeout rate, 429 rate, median latency, p95 latency, and cost per successful task by model. Also record how often fallbacks are used. A rising retry rate may signal a bad timeout value, oversized prompts, a concurrency spike, or an upstream incident long before customers report a problem.

Set a circuit breaker around repeated failures. If a model fails five times in a short window, temporarily stop sending fresh traffic to it and route eligible requests to a fallback model or a queue. Choose fallbacks by task requirements: a fast, lower-cost model for classification or extraction, and a stronger model only for complex reasoning or high-value generation.

59API is especially useful for this model-routing approach because developers can access Claude and GPT options through a compatible API setup while keeping spend predictable. If you are building a resilient multi-model workflow, consider signing up for 59API and testing your retry policy with real traffic before launch.