← Retour aux guides

LLM Batch Processing: 9 Concurrency Pitfalls

Batching can reduce total processing time when you need to summarize documents, classify records, generate embeddings, or run evaluations across thousands of inputs. However, sending every request at once is one of the fastest ways to trigger rate limits, increase costs, and lose track of results. The safest approach is controlled concurrency with clear limits, retries, and observability.

1. Confusing batching with unlimited parallelism

A batch is a logical group of jobs, not permission to create one task per item and submit them simultaneously. Start with a bounded worker pool or semaphore. For example, allow 5 to 20 requests in flight, measure throughput, and increase the limit gradually. Keep separate limits for different models because a fast, inexpensive model and a large reasoning model may have very different capacity and token costs.

2. Measuring only requests per minute

Many LLM services limit both request volume and tokens per minute. Ten large prompts can exhaust a token allowance faster than hundreds of short classification requests. Estimate prompt and expected completion tokens before dispatching work, then maintain both a request budget and a token budget. Reserve headroom for retries instead of operating continuously at the published ceiling.

3. Treating a 429 as a permanent failure

Rate-limit responses are usually temporary. Retry with exponential backoff and jitter, such as delays of 1, 2, 4, and 8 seconds with a small random offset. Respect a Retry-After value when the API provides one. Set a maximum attempt count and move exhausted jobs to a dead-letter queue. Do not retry every error: invalid authentication, malformed requests, and context-length failures require a code or data fix.

4. Creating duplicate charges during retries

A timeout does not prove that the provider failed to process the request. Retrying immediately can produce two completions and two charges. Give every job a durable application-level ID, store request status, and make result writes idempotent. If your provider supports idempotency keys, use them. Otherwise, deduplicate by job ID and retain enough metadata to reconcile uncertain outcomes.

5. Losing the relationship between inputs and outputs

Concurrent responses finish out of order. Never associate a result with an input by array position after workers have started. Include a stable record ID in your internal job object, and persist the input, model, attempt number, response ID, status, and timestamps. If users expect original ordering, sort completed results by the original sequence only after all jobs have been reconciled.

6. Ignoring prompt and context limits

Batching does not combine unlimited text into one request. Large documents can exceed a model's context window, while oversized prompts increase latency and cost. Count or estimate tokens, split documents at sensible boundaries, and preserve overlap when chunking is important for meaning. Set a maximum output token value so an unexpectedly verbose response cannot consume the entire batch budget.

7. Overlooking backpressure

A producer that reads files faster than workers can process them will fill memory and make shutdown unreliable. Use a bounded queue, pause input when the queue is full, and persist pending jobs in a database or durable queue for long-running workloads. Add cancellation handling so a deployment or user stop request prevents new jobs from starting while allowing safe cleanup of active requests.

8. Using one retry policy for every model

Different models have different latency, price, and capacity characteristics. Track success rate, p50 and p95 latency, input and output tokens, and cost per completed item by model. Route simple, high-volume work to a suitable lower-cost model, while reserving more capable models for tasks that need them. This is more reliable than blindly increasing concurrency.

9. Forgetting that cheap access still needs cost controls

Before launching a large batch, run a small sample and calculate estimated cost from real token usage. Set per-job and per-batch budgets, alert on unusual completion lengths, and stop processing when spending or error thresholds are exceeded. 59API can be a practical low-cost relay for this workflow: its base URL is https://api.59api.com, it provides pay-as-you-go access to native official-quality Claude models including Opus, Sonnet, Haiku, and Fable, as well as GPT models, and works with Claude Code, Codex, and OpenAI SDKs. Developers can keep the same concurrency safeguards while reducing relay costs, with a referral rebate also available.

A safer implementation pattern

Build a queue, enforce a semaphore, reserve request and token capacity, and process each item through a state machine such as queued, running, succeeded, retrying, or failed. Record structured logs and metrics, use bounded exponential retries, and test with realistic prompts before scaling. If you want an economical starting point, sign up for 59API and send a small controlled batch through its compatible API before tuning concurrency for production.