← सभी गाइड पर लौटें

From Queue to Production: LLM Batch Concurrency

Processing one LLM request at a time is simple, but it becomes expensive and slow when you need to summarize thousands of documents, classify support tickets, or enrich a product catalog. The production solution is a batch worker with bounded concurrency: group independent jobs, run a controlled number in parallel, and record every result.

Understand batching and concurrency

A batch is a collection of independent tasks. Concurrency is the number of requests allowed to run at the same time. These are different controls. A batch may contain 10,000 items, while your worker may process only 8 requests concurrently.

Unbounded concurrency is a common mistake. Calling Promise.all on a large array can exhaust sockets, trigger provider rate limits, create expensive retry storms, and make failures difficult to resume. Start with a small limit, measure throughput, and increase it only when latency, error rates, and quotas remain healthy.

Choose a compatible API endpoint

59API provides pay-as-you-go access to native Claude models, including Opus, Sonnet, Haiku, and Fable, as well as GPT models. It is designed as a low-cost relay without downgrading the underlying official-quality models. Its API base URL is https://api.59api.com, and it works with the OpenAI SDK, Claude Code, and Codex.

For an OpenAI-compatible client, configure the SDK with your 59API key and set its base URL to https://api.59api.com. Keep the key in an environment variable such as LLM_API_KEY; never commit it to source control. Select a lower-cost model for straightforward classification and reserve stronger models for tasks that genuinely need deeper reasoning.

Build a bounded worker

Represent each input with a stable job ID, the source data, the selected model, and an attempt counter. In JavaScript, a library such as p-limit can enforce a fixed number of active calls. The core pattern is: create a limiter with a value such as 8, map each item to a limited function that calls the model, and await the resulting promises.

Keep the model request small and explicit. Set a system instruction, include only the fields needed for the task, and request a predictable output format. If you need structured data, validate the response against a schema before marking the job complete. A successful HTTP response is not enough if the model returned malformed or incomplete content.

Handle rate limits and transient errors

Use exponential backoff with jitter for timeouts, connection failures, and rate-limit responses. A practical schedule is 1 second, 2 seconds, 4 seconds, then 8 seconds, with a small random delay added to each wait. Cap retries at three or five attempts. Do not retry authentication errors, invalid requests, or schema failures without changing the input.

Honor provider hints such as Retry-After when available. Also apply a per-request timeout so one stalled call cannot occupy a concurrency slot forever. When the retry budget is exhausted, write the failure and reason to durable storage instead of silently dropping the item.

Make jobs restartable

Production batches need idempotency. Before sending a request, check whether the job ID already has a verified result. After validation, store the result, model, prompt version, token usage if available, timestamps, and final status. If the process crashes, the next run should resume pending and retryable jobs rather than duplicate completed work.

Preserve input order only when the downstream system requires it. Otherwise, write results as they complete and associate them with their job IDs. This improves throughput and prevents a slow request from blocking the entire batch.

Control cost and observe throughput

Measure requests per minute, average latency, tokens per item, retry rate, and cost per successful result. Set a maximum batch size and a spending guard before starting large jobs. Prompt caching, smaller inputs, and model routing often reduce cost more effectively than simply increasing concurrency.

59API is a practical option when you want low-cost, usage-based access across Claude and GPT models, with a referral rebate available as well. You can sign up, configure the compatible endpoint, and begin with a small test batch before moving to production volume.

मूल्य · दस्तावेज़