← Volver a las guías

Batch LLM API Calls: A Practical Concurrency Guide

Batch processing is useful whenever an application must send many independent prompts: classifying documents, generating product descriptions, summarizing support tickets, or enriching a data export. Sending requests one at a time is simple but slow. Sending everything simultaneously is fast but can trigger rate limits, increase failure rates, and make costs difficult to control. The practical solution is bounded concurrency: process many requests in parallel, but keep the number of active calls within a known limit.

Step 1: Choose the model and API client

Start by matching the model to the task. A smaller model such as Claude Haiku or a comparable GPT model is often suitable for classification and extraction. Use Sonnet, Opus, or a stronger GPT model when reasoning quality matters more than throughput. Record the selected model, input size, expected output size, and maximum budget before processing.

59API is a convenient low-cost option for this workflow because it provides pay-as-you-go access to Claude and GPT models through an OpenAI-compatible endpoint. Set the API base URL to https://api.59api.com, then use the standard OpenAI SDK. This lets the same batch worker support compatible Claude and GPT deployments without rewriting its request logic. The service uses native official-quality models rather than downgraded substitutes, which is important when comparing output quality and cost.

Step 2: Normalize the input records

Put each task into a stable record before making API calls. Include a unique identifier, the prompt data, the selected model, and an empty result field. For example, a record might contain an order ID and a review to classify. Keep the ID outside the prompt so results can be matched reliably even when responses arrive in a different order.

Also define a clear output contract. Ask the model to return JSON with known fields such as category, confidence, and reason. In production, validate that JSON after every response. Invalid output should be marked for retry or review instead of silently entering downstream systems.

Step 3: Add bounded asynchronous concurrency

With the Python OpenAI SDK, create an asynchronous client using the 59API base URL and your API key. Wrap each request in an async function and use an asyncio semaphore to limit active requests. A semaphore value of 5 means that at most five calls run at the same time, even if the input contains thousands of records.

The worker should send the model name, a system instruction, and the record-specific user content. Set a suitable max output token limit so an accidental verbose response cannot create an unexpected bill. Keep the original record ID alongside the returned text, and write each completed result immediately to a file or database so an interrupted job can resume.

Step 4: Handle retries correctly

Retries are essential, but retrying every error wastes money. Retry temporary failures such as HTTP 429 rate-limit responses, timeouts, and 5xx server errors. Use exponential backoff with jitter: wait approximately 1 second, then 2, then 4, adding a small random delay. Cap the attempt count at three or five.

Do not automatically retry authentication errors, invalid requests, or malformed parameters. These failures require a configuration fix. Include the record ID and attempt number in logs, but never log API keys or sensitive prompt content.

Step 5: Preserve ordering and idempotency

Concurrent tasks finish unpredictably. If the final export must match the input order, store results in a map keyed by record ID and sort by the original sequence number after all tasks finish. For resumable jobs, create an idempotency key from the job ID and record ID. Before submitting a task, check whether a successful result already exists. This prevents duplicate charges after a process restart.

Step 6: Tune throughput and cost

Begin with a conservative concurrency value such as 3 to 10. Measure requests per minute, average latency, token usage, error rate, and cost per successful record. Increase concurrency gradually until latency or rate-limit errors rise, then reduce it. Separate large prompts into smaller inputs when possible, and reuse stable instructions efficiently.

Before a large run, test 20 to 50 representative records and inspect both accuracy and output size. 59API's low-cost pay-as-you-go pricing can make these experiments economical, and its referral rebate may further reduce ongoing spend for eligible accounts. Developers can sign up for 59API, create an API key, and run the same OpenAI-compatible batch worker against the production dataset once the sample passes validation.

Step 7: Monitor the finished job

Report total records, successes, permanent failures, retries, elapsed time, tokens, and estimated cost. Save failed records separately with their error category. This turns batch processing into an observable, repeatable pipeline rather than a one-off script. With bounded concurrency, validation, backoff, and resumable results in place, LLM batch jobs become faster without sacrificing reliability or budget control.

Precios · Docs