← Voltar aos guias

Production LLM Batches: Safe Concurrency with 59API

Why Batch Processing Needs More Than a Loop

Batch processing lets an application send many independent LLM tasks, such as summarizing support tickets, extracting invoice fields, classifying product reviews, or generating catalog descriptions. The common mistake is to loop through records and fire every request at once. That can create rate-limit errors, duplicate work, unpredictable spend, and failures that are difficult to replay.

A production pipeline should treat every LLM call as a job with a stable ID, input payload, selected model, status, attempt count, result, and error details. Store these fields in a database or durable queue before calling the API. This gives you a reliable audit trail and allows incomplete work to resume after a deployment, timeout, or worker crash.

Step 1: Choose the Right Model for Each Job Class

Do not use one expensive model for every task. Split work into classes based on quality, reasoning depth, latency, and output risk. For example, use a fast, lower-cost model such as Claude Haiku for straightforward tagging, cleanup, and extraction; use Sonnet for higher-quality structured analysis and writing; and reserve Opus or GPT reasoning models for the small percentage of tasks that genuinely need deeper reasoning.

59API is a practical option for this setup because it provides pay-as-you-go access to Claude models including Opus, Sonnet, Haiku, and Fable, plus GPT models, without downgrading model quality. Its low relay pricing makes it easier to run large backfills while keeping per-job costs visible. Configure your OpenAI-compatible SDK to use the 59API base URL, https://api.59api.com, then choose the model name and compatible endpoint documented for your account.

Step 2: Build a Durable Job Queue

Create a jobs table or queue with fields such as job_id, source_id, prompt_version, model, state, attempts, locked_at, output_json, input_tokens, output_tokens, and last_error. Use states such as queued, processing, succeeded, retryable_failed, and permanently_failed. Include a unique constraint on source_id plus prompt_version so rerunning a batch does not silently generate duplicate work.

• Write jobs first; do not generate requests directly from an untracked spreadsheet or browser process.
• Assign a prompt version to every job so output changes can be traced to a specific instruction set.
• Set a lock or lease when a worker claims a job. Expire stale locks so abandoned jobs can be recovered.
• Save raw model output when permitted, but also save validated structured fields separately for downstream systems.

Step 3: Set Concurrency by Tokens, Not Just Requests

Concurrency is the number of requests your workers run simultaneously. Start conservatively, such as five workers, and increase gradually while monitoring latency, failures, and token usage. Request count alone is misleading: ten short classification calls may be safe, while ten requests containing large documents can consume far more capacity.

Use two controls. First, cap in-flight requests with a worker pool or semaphore. Second, apply a token budget per minute based on estimated input tokens plus a maximum output allowance. If a job contains a large document, route it to a lower-concurrency queue. This prevents a few oversized prompts from starving smaller jobs.

Also separate interactive and background traffic. Customer-facing requests should have their own reserved concurrency, while nightly imports, embedding generation, or content backfills use a background queue. That separation protects user experience when a large batch starts.

Step 4: Retry Safely and Validate Every Response

Retry transient failures such as timeouts, connection resets, and rate-limit responses with exponential backoff and random jitter. A reasonable pattern is to retry after roughly 2, 4, 8, and 16 seconds, while respecting any retry guidance returned by the API. Do not blindly retry authentication failures, invalid requests, unsupported model names, or malformed prompts; send those jobs to a review queue instead.

Make retries idempotent. Use the same job ID and do not publish a result until it has been validated. For JSON tasks, instruct the model to return the required schema, parse it, validate required fields and types, then mark the job successful only after validation passes. If the response is incomplete, retry with a repair prompt or flag it for human review.

Step 5: Monitor Cost, Quality, and Throughput

Track jobs completed per minute, queue age, success rate, retry rate, average latency, token totals, and cost by model and prompt version. Alert when queue age rises, failures exceed a threshold, or a new prompt version increases output tokens unexpectedly. Sample outputs regularly for quality; a pipeline can be technically successful while producing subtly incorrect classifications.

Before processing millions of records, run a representative pilot of 100 to 500 jobs. Measure quality, token consumption, and the highest stable concurrency. Then scale in stages. With native official-quality model access, OpenAI SDK compatibility, and economical pay-as-you-go pricing, 59API can keep this production workflow flexible across Claude and GPT models. If you are ready to test your own workload, sign up for 59API and begin with a small, measurable batch.

Preços · Docs