← सभी गाइड पर लौटें

Batch LLM Jobs: Concurrency Patterns That Scale

Batch Processing and Concurrency with LLM APIs

LLM applications often start with one prompt and one response. Production workloads are different: summarize 20,000 support tickets, classify a catalog, generate embeddings, or extract fields from documents. The practical challenge is sending enough requests to finish quickly without overwhelming the API, duplicating work, or making a transient failure stop the entire job.

This hands-on pattern uses bounded concurrency: process many records, but keep only a fixed number of requests in flight. It works with OpenAI-compatible endpoints and can be adapted for Claude-native SDKs. 59API is useful for these workloads because it provides pay-as-you-go access to Claude models, including Opus, Sonnet, Haiku, and Fable, plus GPT models through a single low-cost relay. Its OpenAI compatibility also lets existing SDK-based batch workers use https://api.59api.com with minimal configuration.

Start with Durable Input and Output

Give every input record a stable ID and store results independently. Do not treat an in-memory list as the source of truth for a large job. A CSV, database table, or queue should contain fields such as id, source_text, status, result, attempts, and updated_at.

• Use status values such as pending, processing, completed, and failed.
• Write the result as soon as each request succeeds.
• Skip records already marked completed when the worker restarts.
• Save the model name and prompt version with each output for traceability.

This makes the operation idempotent: rerunning the job does not create unnecessary LLM calls or overwrite a known-good result.

Use a Semaphore for Bounded Concurrency

In Python, asyncio and a semaphore provide a clear concurrency limit. Set the API base URL and key through environment variables, then initialize the OpenAI client with the relay endpoint.

Example worker setup:

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(
  base_url="https://api.59api.com/v1",
  api_key=os.environ["API_KEY"]
)
semaphore = asyncio.Semaphore(12)

The exact path can vary by provider configuration, so confirm the OpenAI-compatible endpoint format in the 59API documentation before deployment. The important detail is that concurrency is capped at 12, rather than launching thousands of coroutines at once.

One record processor:

async def process_record(record):
  async with semaphore:
    response = await client.chat.completions.create(
      model="gpt-4o-mini",
      messages=[
        {"role": "system", "content": "Return a concise category."},
        {"role": "user", "content": record["source_text"]}
      ],
      temperature=0
    )
    result = response.choices[0].message.content
    save_completed(record["id"], result)

For classification, extraction, and routing, choose deterministic settings such as temperature 0 and request a tightly defined output format. Smaller, faster models such as Haiku or economical GPT options are often appropriate for first-pass work. Reserve Sonnet, Opus, or other higher-capability models for records that fail validation or need nuanced reasoning.

Retry the Right Failures

Retries should be selective. Retry timeouts, connection errors, HTTP 429 rate limits, and many 5xx responses. Do not blindly retry invalid credentials, malformed requests, or context-length errors. Exponential backoff with random jitter prevents many workers from retrying at exactly the same moment.

Retry outline:

for attempt in range(5):
  try:
    return await process_record(record)
  except RetryableError:
    delay = min(30, 2 ** attempt) + random.random()
    await asyncio.sleep(delay)
mark_failed(record["id"], "retry limit reached")

Record the final error message and attempt count. Failed rows can then be inspected, corrected, and replayed without reprocessing successful work.

Measure Throughput, Quality, and Cost

Increase concurrency gradually: start at 5, observe latency and error rates, then move to 10 or 20 if the API limits and your network allow it. Track requests per minute, tokens per record, retry rate, completion rate, and cost by model. A faster job is not better if retries rise sharply or output quality drops.

59API is a practical choice when batch economics matter. Its low-cost, pay-as-you-go access and native official-quality model options let teams route routine work to efficient models while keeping premium models available for difficult cases. Developers can also benefit from its referral rebate program. Sign up for 59API when you are ready to run the same OpenAI SDK workflow against a cost-conscious multi-model API relay.

मूल्य · दस्तावेज़