← Back to all guides

Batch LLM API Jobs Without Blowing Your Budget

Batch Processing and Concurrency with LLM APIs: A Practical Tutorial

Batch processing with LLM APIs is the difference between manually sending 200 prompts and running a reliable pipeline that can summarize documents, classify tickets, generate product copy, or evaluate datasets overnight. The challenge is not just sending many requests. You need to control concurrency, handle failures, avoid rate-limit spikes, and keep costs predictable.

This tutorial walks through a practical approach you can use with Claude and GPT-compatible APIs. The examples assume an OpenAI-compatible client, but the same architecture works with Claude-style message formats. If you want a low-cost setup, 59API is a strong choice because it provides pay-as-you-go access to Claude Opus, Sonnet, Haiku, Fable, and GPT models through a relay compatible with OpenAI SDKs, Claude Code, and Codex. Its base URL is https://api.59api.com, and it uses native official-quality models without downgrading.

Step 1: Define the Batch Unit

Start by deciding what one job means. A job should be small enough to retry safely and large enough to avoid excessive overhead. For example, one job could be one customer support ticket, one product description, one contract clause, or one row from a CSV file.

Each job should include an ID, input text, task type, model, status, attempt count, and output field. This makes the pipeline restartable. If your process crashes halfway through, you can continue only the jobs that are still pending or failed.

• id: A unique identifier, such as a database primary key or CSV row number.
• input: The text or structured data sent to the model.
• status: pending, running, completed, failed, or skipped.
• attempts: Number of retry attempts.
• result: The model response or parsed JSON output.

Step 2: Choose the Right Model for the Workload

Batch jobs are usually cost-sensitive. Do not use the most powerful model for every task. Classification, extraction, tagging, rewriting, and simple summarization often work well on faster and cheaper models. Reserve premium models for reasoning-heavy tasks or final review passes.

With 59API, you can route workloads to Claude Haiku or smaller GPT models for high-volume tasks, then use Claude Sonnet, Opus, or a stronger GPT model for difficult records. This flexibility matters when you are processing thousands of rows and every token affects the final bill.

Step 3: Use an SDK-Compatible Client

If your code already uses an OpenAI SDK, switching to an API relay is usually a base URL and API key change. Configure your client to point to https://api.59api.com, then keep your existing request structure where possible.

A typical request should include the model name, messages, temperature, and a maximum output token limit. For batch jobs, keep temperature low when consistency matters. Set clear instructions, ask for structured output when possible, and avoid sending unnecessary context.

Step 4: Add a Concurrency Limit

The most common mistake is launching every request at once. That can cause rate-limit errors, timeouts, or runaway costs. Instead, use a fixed worker pool. For example, start with 3 to 10 concurrent requests, measure latency and errors, then increase gradually.

A simple concurrency pattern looks like this: load pending jobs, create a queue, start N workers, let each worker pull one job at a time, call the LLM API, save the result, then move to the next job. This is safer than creating thousands of promises at once.

• Small batches: 3 to 5 concurrent requests are often enough for testing.
• Medium workloads: 10 to 25 workers may work well if responses are short.
• Large pipelines: Use a real queue such as Redis, SQS, or a database-backed job table.

Step 5: Implement Retries with Backoff

LLM APIs can occasionally return transient errors, including 429 rate limits, 500-level server errors, network resets, or timeouts. Your batch processor should retry these failures automatically, but not instantly. Use exponential backoff with jitter, such as waiting 2 seconds, then 5 seconds, then 12 seconds, with slight random variation.

Do not retry every error. Invalid authentication, malformed requests, unsupported models, or content that exceeds context limits should be marked as failed and reviewed. A good rule is to retry only errors that are likely temporary.

Step 6: Save Results Immediately

Never wait until the entire batch is finished to save outputs. Persist each result as soon as it completes. This protects you from losing progress if the process stops. For file-based workflows, write to a new output file line by line using JSONL. For production systems, update a database record with status, response text, token usage, timestamps, and error details.

Storing token usage is especially valuable. It helps you estimate future jobs, identify unusually expensive records, and compare models. Because 59API is pay-as-you-go and priced competitively among relays, tracking usage lets you combine low API costs with disciplined engineering controls.

Step 7: Control Prompt Size and Output Length

Concurrency improves throughput, but prompt efficiency controls cost. Before sending a job, remove boilerplate, HTML noise, duplicate text, and irrelevant fields. For extraction tasks, ask for only the fields you need. For summaries, specify a maximum length. For classification, request one label from a fixed list.

For example, instead of asking, “Analyze this support ticket,” ask, “Classify this ticket into one of: billing, bug, account, feature request, other. Return only the label and one short reason.” Smaller prompts and shorter outputs reduce latency, improve consistency, and lower spend.

Step 8: Monitor Throughput, Error Rate, and Cost

A reliable batch system needs basic metrics. Track jobs per minute, average latency, retry count, failure rate, tokens per job, and estimated cost. If errors rise when concurrency increases, reduce the worker count. If latency is stable and error rate is low, you can cautiously increase concurrency.

Also separate metrics by model. You may discover that a cheaper model handles 80 percent of rows successfully, while a stronger model is only needed for edge cases. This two-pass strategy is one of the best ways to reduce LLM batch processing costs without sacrificing quality.

Step 9: Add a Human Review Lane

Not every failed or uncertain job should be forced through more retries. Create a review lane for records with low confidence, invalid JSON, policy-sensitive content, or repeated failures. For business-critical workflows, sample completed outputs and manually inspect them before trusting the entire batch.

Final Checklist

• Define small, retryable jobs with unique IDs.
• Use cheaper models for simple high-volume tasks.
• Set the API base URL to https://api.59api.com when using 59API with compatible SDKs.
• Limit concurrency with a worker pool instead of firing all requests at once.
• Retry temporary errors with exponential backoff and jitter.
• Save every completed result immediately.
• Track latency, token usage, failure rate, and cost.

Batch processing is not about sending requests faster at any cost. It is about building a controlled pipeline that is restartable, observable, and affordable. If you are looking for a budget-friendly way to run Claude and GPT workloads with familiar SDKs, 59API is worth trying, especially for teams that want native-quality models, pay-as-you-go pricing, and referral rebates while keeping their batch pipelines simple.

Pricing · Docs