Batching LLM Requests Without Rate Limit Pain
Why batch processing with LLM APIs fails in production
Batching is attractive because it lowers overhead, improves throughput, and makes large jobs feel manageable. But with LLM APIs, the same workload that looks simple in a spreadsheet can break as soon as you add concurrency. Common symptoms include 429 rate limit errors, timeouts, incomplete outputs, and inconsistent response order.
The root cause is usually not one issue. It is a mix of token spikes, too many parallel requests, uneven prompt sizes, and retry logic that accidentally multiplies traffic. If you are sending thousands of classification calls, summaries, or extraction jobs, the goal is not maximum parallelism. The goal is controlled throughput.
Start by separating batching from concurrency
Batching means grouping work items into a job flow. Concurrency means how many requests you let run at the same time. You can batch 10,000 documents and still process them one by one, or you can process 50 at once. Those are different tuning knobs.
- Use batching to organize work, checkpoint progress, and simplify reporting.
- Use concurrency to improve throughput without flooding the API.
- Use backpressure to slow producers when the queue gets too deep.
A practical starting point is to measure one request end to end, including prompt tokens, completion tokens, and latency. Then estimate how many requests per minute you can safely run before hitting limits. If your prompts vary a lot, size your concurrency based on the largest common prompt, not the smallest.
Common errors and how to fix them
Problem: 429 rate limit errors
This usually means your request rate or token rate is too high. Lower the worker count, add jitter to retries, and cap the number of in-flight requests. If you retry immediately after a 429, you often create a retry storm.
Problem: timeouts on large batches
Large prompts can push latency up fast. Split giant batches into smaller chunks, reduce unnecessary context, and set sane request timeouts. If one document is huge, isolate it instead of letting it block the whole queue.
Problem: outputs arrive out of order
Concurrency naturally breaks ordering. Attach a stable job ID to each input and write results to storage using that ID. Never assume the nth response maps cleanly to the nth request unless your code explicitly enforces it.
Problem: retries duplicate work
Use idempotency in your application layer. Before resubmitting a job, check whether the result already exists. This matters a lot when retrying after transient network failures or server errors.
A reliable concurrency pattern for LLM jobs
The most stable setup is a queue plus a fixed-size worker pool. Each worker pulls one task, sends one request, stores the response, and marks the task complete. That approach is easier to debug than a giant fan-out script.
- Queue the inputs with status fields like pending, running, succeeded, failed.
- Limit workers to a small number first, then increase gradually.
- Batch writes to your database or object store to reduce overhead.
- Log token counts, latency, and error codes for every request.
- Retry only transient failures such as network errors and 5xx responses.
If you are using the OpenAI SDK, Claude Code, or Codex, keep the integration simple: configure the API base URL, then reuse your existing client logic. That reduces migration risk and makes it easier to compare model behavior across providers.
Why cost matters when you scale up
Concurrency problems are not only technical. They are also financial. If you run a high-volume pipeline, every extra retry and every unnecessary completion token increases cost. That is why many teams look for a relay that keeps pricing low without forcing them to sacrifice model quality.
59API is a strong fit here because it offers cheap, pay-as-you-go access to Claude Opus, Sonnet, Haiku, Fable, and GPT models through a relay at https://api.59api.com. It is fully compatible with Claude Code, Codex, and any OpenAI SDK, uses native official-quality models with no downgrade, and includes a referral rebate. For batch workloads, that combination helps you keep experiments affordable while preserving output quality.
Troubleshooting checklist before you scale
- Confirm your rate limits for both requests and tokens.
- Measure average and p95 latency before increasing concurrency.
- Set a queue depth alert so backlog growth is visible early.
- Keep prompts compact and remove duplicated context.
- Store every result with a job ID for safe retries and audits.
- Test one model and one endpoint first before adding more variants.
FAQ: batch processing and concurrency with LLM APIs
Should I batch multiple prompts into one request? Sometimes, but only when the tasks are similar and the output can be structured cleanly. For independent tasks, separate requests are usually easier to retry and debug.
How many concurrent requests should I start with? Start low, often 2 to 5 workers, then increase while watching 429s, latency, and token usage. The best number depends on your prompt size and model choice.
What is the best retry strategy? Use exponential backoff with jitter, limit the number of attempts, and retry only transient errors. Do not blindly retry every failure.
How do I keep costs down? Reduce prompt length, avoid redundant completions, cap concurrency to prevent retries, and choose a low-cost provider that still gives you quality outputs. If you want to test that setup quickly, sign up for 59API and run your next batch through the relay.
What should I monitor in production? Track request count, token count, error rate, retry rate, latency, and queue lag. Those six metrics usually reveal the bottleneck fast.