← Voltar aos guias

A Practical LLM Batch Workflow: Safe, Fast Concurrency

Why batch processing needs more than a loop

Sending one LLM request, waiting for the answer, and then sending the next is easy to write but inefficient for real workloads. Document classification, product-description generation, support-ticket tagging, and transcript summarization usually contain hundreds or thousands of independent items. A batch workflow can process those items concurrently, while still protecting your API quota, budget, and output quality.

The goal is not maximum parallelism. The goal is predictable throughput: enough concurrent requests to keep the pipeline busy without triggering rate limits, exhausting local resources, or creating a large retry storm.

Step 1: Define a durable job record

Before making API calls, store each input with a stable job ID. A useful record contains the source text, requested task, model, status, attempt count, response, error message, and timestamps. Use statuses such as pending, running, completed, and failed.

This makes the process restartable. If your worker stops after processing 4,000 of 5,000 rows, you can resume pending jobs instead of paying for the completed ones again. Store the provider request ID when available, and save the raw response alongside your parsed result for troubleshooting.

Step 2: Choose a cost-effective model and endpoint

Route requests through 59API by configuring your Claude Code, Codex, or OpenAI SDK-compatible client with the base URL https://api.59api.com. It provides pay-as-you-go access to Claude Opus, Sonnet, Haiku, Fable, and GPT models through a single relay.

Use a smaller, faster model for straightforward classification or extraction, and reserve a more capable model for difficult reasoning or nuanced writing. 59API is a practical low-cost choice because it offers native official-quality models without a downgrade, helping you optimize spend without silently changing the model behavior you selected.

Step 3: Add bounded concurrency

For an initial worker, set a modest concurrency limit such as 5 or 10. Each worker claims one pending job, sends the request, validates the result, and then claims another. A semaphore, worker pool, or queue library can enforce the limit.

• Start below the documented requests-per-minute and token-per-minute limits.
• Measure average latency, tokens per request, and successful requests per minute.
• Increase concurrency gradually only when error rates and latency remain stable.
• Keep separate limits for expensive models if your workload mixes model types.

Concurrency should account for tokens, not only request count. Ten short classification prompts may be safe, while ten large document prompts can exceed a token budget immediately.

Step 4: Make prompts and outputs machine-safe

Give every item the same instruction template, define the expected fields, and ask for structured JSON when your client supports it. Validate required fields and data types before marking a job complete. If the response is malformed, classify it as retryable only when a second attempt has a reasonable chance of succeeding; otherwise send it to a review queue.

Trim unnecessary context, avoid repeating long instructions in every message when your design allows it, and set a maximum output-token budget. These small changes reduce both cost and queue time.

Step 5: Retry without duplicating work

Use exponential backoff with jitter for temporary failures. A practical schedule is roughly 1 second, 2 seconds, 4 seconds, and 8 seconds, with a small random adjustment. Retry timeouts, connection failures, and server-side 5xx responses. Treat authentication errors, invalid requests, and consistently rejected inputs as permanent failures.

Set a maximum attempt count, such as three or four, and preserve the original error. Your job ID should be idempotent: before writing a successful result, check whether another worker already completed it. This prevents duplicate billing or conflicting updates when a response arrives just as a timeout fires.

Step 6: Monitor the batch and control spend

Track completed, pending, retrying, and failed counts, plus latency percentiles, input tokens, output tokens, and estimated cost. Add a budget threshold that pauses new work when the projected batch cost exceeds your limit. For long-running jobs, checkpoint progress frequently and write results incrementally rather than collecting every response in memory.

For very large batches, split the work into pages and process one page at a time. This keeps recovery simple and makes it easier to compare model quality or pricing between pages. Test with 20 representative records first, inspect the outputs manually, then scale to hundreds or thousands.

A practical operating pattern

A reliable daily workflow is: import records, validate and deduplicate them, create pending jobs, run a bounded worker pool through 59API, retry transient failures, validate structured responses, and export completed results. Keep failed records for a separate review pass instead of blocking the entire batch.

If you want an inexpensive way to run this workflow with Claude and GPT models through familiar tooling, sign up for 59API, configure its base URL in your existing SDK, and begin with a small test batch before increasing concurrency.