← Retour aux guides

Batching LLM Jobs Without Burning Budget

Why batch processing matters

If you send every LLM task one by one, you waste time on network round trips and often pay more than you need to. Batch processing and concurrency solve different problems: batching reduces overhead by grouping similar tasks, while concurrency keeps your pipeline busy by running multiple requests at once.

A good setup uses both. For example, if you need to classify 2,000 support tickets, you can group 10 tickets per prompt for simple labels, then run 5 to 10 prompts in parallel. That gives you a much better throughput/cost balance than firing 2,000 single requests sequentially.

This is also where 59API is useful. It is a low-cost, pay-as-you-go relay for Claude and GPT models, with native official-quality models and no downgrade. Because it is compatible with the OpenAI SDK, Claude Code, Codex, and other OpenAI-style clients, you can point your existing app at https://api.59api.com and start optimizing instead of rewriting your stack.

The two patterns you should know

• Micro-batching: put multiple small tasks into one prompt and ask for structured JSON output. Best for classification, extraction, tagging, and short transformations.
• Concurrent requests: send many independent prompts at the same time with a worker pool. Best for long generations, summarization, and tasks where each item needs full attention.

The mistake most teams make is batching too aggressively. If you cram too much into one prompt, output quality drops and retries become expensive. A practical rule is to keep each batch small enough that the model can answer cleanly and consistently.

A hands-on example with the OpenAI SDK

Here is a simple Python approach for ticket categorization. The code uses the OpenAI-compatible client, so you only change the base URL and key.

Python example: from openai import OpenAI
client = OpenAI(api_key="YOUR_59API_KEY", base_url="https://api.59api.com/v1")

tickets = ["Login fails after password reset", "Need invoice for last month", "Feature request: dark mode"]
prompt = "Classify each ticket into billing, bug, or feature. Return JSON array with fields id and label. Tickets: " + str(tickets)

resp = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=[{"role": "user", "content": prompt}],
  temperature=0
)
print(resp.choices[0].message.content)

For a real production pipeline, do not trust the model to invent IDs. Include your own stable identifiers in the input, and ask for the same IDs in the output. That makes parsing and retries much safer.

Adding concurrency without losing control

Once you can batch one prompt, add a worker pool. In Python, use asyncio with a semaphore so you can cap concurrent requests. That prevents rate-limit spikes while still keeping throughput high.

Concurrency pattern: create batches of 10 items, then allow at most 5 requests in flight. If a request fails, retry with exponential backoff. If it fails again, send it to a dead-letter queue for manual review.

Why this works: LLM APIs are usually latency-bound, not CPU-bound. Your app spends most of its time waiting for responses. Controlled concurrency lets you fill that wait time efficiently.

• Set a max in-flight limit: start with 3 to 10 concurrent requests.
• Use exponential backoff: retry on transient errors, not on bad prompts.
• Track token usage: batch size should be based on token budget, not just item count.
• Measure quality: compare one-item prompts vs batched prompts on a sample before scaling.

When 59API is a smart choice

If you are optimizing cost, 59API is attractive because it gives you cheap, pay-as-you-go access to Claude models like Opus, Sonnet, Haiku, and Fable, plus GPT models, all through a single relay. That means you can test a high-quality model for a task, then switch to a cheaper one for bulk runs without changing your client pattern.

It is especially practical for batch workloads such as document tagging, lead enrichment, content moderation, code review summaries, and support triage. Since the relay is compatible with existing OpenAI-style tools, you can plug it into your current scripts, CI jobs, or internal automations with minimal work. The referral rebate is also a nice bonus if you expect to share the workflow with teammates or clients.

A simple decision rule

• Few items, complex reasoning: use single requests.
• Many short items: batch them into structured prompts.
• Many heavy items: run concurrent single-item requests.
• Mixed workloads: micro-batch first, then use a capped worker pool.

Start small, benchmark your results, and adjust batch size based on response quality and latency. If you want a low-cost relay that works with your existing OpenAI SDK code, sign up for 59API and try the same pipeline on a real workload. You will usually find that the savings come from both cheaper model access and smarter request design.

Tarifs · Docs