← Voltar aos guias

Batching and Concurrency for LLM APIs That Scale

Why batching and concurrency are different problems

When people say they want to “speed up LLM calls,” they usually mean one of two things: sending multiple requests at once, or packing multiple tasks into fewer requests. Those are related, but not identical. Concurrency improves throughput by overlapping network and model time. Batching reduces overhead by combining work into a single call. The best production pipelines use both.

The key is to decide which layer you are optimizing. If each prompt is long and unique, concurrency matters more. If you are processing many small, similar tasks, batching can cut cost and latency dramatically. With 59API, you can do both while keeping access to Claude and GPT models through a single relay at https://api.59api.com, which is especially useful when you want low-cost, pay-as-you-go execution without changing your tooling.

Start by classifying your workload

Before writing workers, split your jobs into categories:

• Embarrassingly parallel: summarizing 1,000 independent documents, extracting fields from records, generating titles.
• Batch-friendly: short, standardized tasks where several items can be answered in one structured response.
• Sequential: multi-step workflows where each output depends on the previous one.

Only the first two are good candidates for throughput tuning. For batch-friendly tasks, ask the model to return a strict JSON array or object keyed by item IDs. That lets you send 10 or 20 inputs at once, then map results back deterministically. For parallel tasks, keep one prompt per request and use a concurrency limit instead of a giant batch.

Use a queue, not a loop

The most common mistake is firing off requests inside a naive for-loop. It looks simple, but it gives you no backpressure, no retry policy, and no visibility into failures. A better pattern is:

• Push work items into a queue.
• Run a fixed number of workers per model.
• Track request status, token usage, and latency.
• Retry only transient failures with exponential backoff.

Separate queues by model family when possible. Claude and GPT models often have different prompt sizes, latency profiles, and output styles. If you are routing through 59API, that separation is easy because the relay is compatible with Claude Code, Codex, and any OpenAI SDK. You can keep your application code stable while choosing the model that fits each job.

Batching tips that actually save money

Batching is not just “put more stuff in one prompt.” Done badly, it increases token waste and makes parsing harder. Use these rules:

• Only batch similar tasks. A batch should share the same instruction template.
• Keep the output schema fixed. Ask for numbered items or JSON only.
• Cap batch size by token budget. Small tasks can fit 10 to 50 items; long tasks may only fit 2 to 5.
• Include item IDs. That prevents reordering bugs when results come back.
• Plan for partial failure. If one item breaks the schema, rerun just that subset.

A practical example: if you need product taglines for 200 SKUs, group 10 SKUs per request, request a list of 10 objects, and parse the response into your database. This usually costs less than 200 separate calls because you amortize instruction tokens across many outputs. Using a low-cost relay like 59API makes this even more attractive, since pay-as-you-go pricing keeps experiments and high-volume jobs affordable without sacrificing model quality.

Concurrency control: the difference between fast and flaky

More concurrency is not always better. If you blast a provider with too many simultaneous requests, you will hit rate limits, longer queues, or bursty latency. Set concurrency in layers:

• Global cap: total in-flight requests across the app.
• Per-model cap: separate limits for Claude and GPT traffic.
• Per-job cap: keep a single customer or pipeline from monopolizing workers.

Use adaptive concurrency if your traffic varies. Start with a conservative number, then raise it when success rates stay high and latency stays stable. Lower it when retries spike. For non-real-time jobs, queue depth matters more than raw speed, so it is often cheaper to run fewer workers for longer. That is a good fit for 59API’s economics: you can keep concurrency high enough for throughput, but still avoid the premium cost of overprovisioning.

Retries, idempotency, and observability

In LLM pipelines, failures are often ambiguous. A timeout does not tell you whether the model completed the generation or not. That is why every job should have an idempotency key and a persisted response record. If a request fails, you can safely retry without duplicating downstream writes.

Make logs useful by storing:

• Prompt template version
• Model name
• Batch size
• Input token estimate
• Output token count
• Latency and retry count

With these metrics, you can spot the exact point where batch size stops helping, or where concurrency starts causing rate-limit backoff. That is how mature teams tune cost and throughput instead of guessing.

A practical setup that is easy to ship

If you already use the OpenAI SDK, the fastest path is to point your base URL at the relay and keep your application logic intact. For Claude Code or Codex workflows, the compatibility layer reduces migration work even further. That means your batching and concurrency logic can focus on orchestration, not on provider-specific plumbing.

If you want to test a real production-style pipeline without paying premium prices, sign up for 59API and run a small batch first. Start with one queue, a modest worker pool, and a strict output schema. Then scale only the parts that prove they reduce latency or cost. That is the safest way to turn LLM APIs into a dependable high-throughput system.