← Retour aux guides

How LLM Streaming Works: Tokens, SSE, and Tool Calls

LLM Streaming Is an Event Stream, Not a Partial JSON Response

When an application appears to receive an AI answer word by word, the model is not usually sending a new HTTP response for every token. The client opens one long-lived HTTP request, and the API keeps that response open while it emits incremental events. Most OpenAI-compatible APIs use Server-Sent Events (SSE): text frames separated by blank lines, with each payload prefixed by data:.

A simplified response looks like this:

data: {"choices":[{"delta":{"content":"LLMs"}}]}

data: {"choices":[{"delta":{"content":" generate"}}]}

data: [DONE]

The model generates tokens sequentially, but a token is not necessarily a word. It might be punctuation, whitespace, a word fragment, or several short characters. The provider groups generated tokens into small chunks, serializes them as SSE data, and flushes them over the existing connection. Your UI concatenates the content deltas until it receives the terminal event.

Send a Streaming Request Through an OpenAI-Compatible Endpoint

The critical request option is stream: true. With 59API, developers can point a standard OpenAI SDK at https://api.59api.com and use supported GPT or Claude models through a familiar interface. This is useful when you want a low-cost, pay-as-you-go relay without changing the streaming architecture of an existing application.

In JavaScript, the core pattern is an asynchronous iterator:

const stream = await client.chat.completions.create({
  model: "gpt-4.1-mini",
  stream: true,
  messages: [{ role: "user", content: "Explain SSE in one paragraph." }]
});

let answer = "";
for await (const chunk of stream) {
  const text = chunk.choices[0]?.delta?.content || "";
  answer += text;
  process.stdout.write(text);
}

Initialize the client with your API key and base URL:

const client = new OpenAI({
  apiKey: process.env.API_KEY,
  baseURL: "https://api.59api.com/v1"
});

Exact model IDs and endpoint paths should follow the relay documentation, but the streaming control flow remains the same. The OpenAI-compatible approach also makes migration from a direct provider integration straightforward.

What Happens Between the Model and the Browser

• Request admission: The API authenticates the request, applies limits, and routes it to the selected model.
• Prompt processing: The model tokenizes the system prompt, conversation history, tools, and latest user message.
• Autoregressive generation: The model predicts one next token at a time, conditioning each prediction on prior tokens.
• Chunk serialization: The serving layer packages one or more generated tokens into SSE frames.
• Network delivery: HTTP response chunks travel through the relay, server, and client runtime while the connection stays open.
• Incremental rendering: The application appends deltas to state, then updates the visible response.

Time to first token is often more important to perceived speed than total generation time. A fast first event makes an interface feel responsive even when the final answer takes several seconds. Avoid waiting for the entire stream before rendering; that removes the main user-experience benefit.

Parse Carefully: Chunks Do Not Equal Events

A common implementation bug is assuming one network read equals one SSE event. TCP and HTTP chunk boundaries are transport details. A single read can contain half an event, one event, or many events. Use an SSE-aware SDK when possible. If you parse manually, buffer incoming text, split only on complete double-newline boundaries, then parse each complete data: line.

Also handle these cases:

• Empty content deltas, which may carry role or finish metadata.
• Disconnects after partial output; retain already rendered text and offer retry behavior.
• Cancellation with an AbortController when the user stops generation or leaves the page.
• UTF-8 boundaries; use a streaming TextDecoder so multi-byte characters are not corrupted.
• Rate limits and transient upstream errors, with bounded retries for requests that have not produced output.

Streaming Tool Calls Requires Accumulation

Tool calls stream differently from plain text. The model may send a tool name first and emit JSON arguments across several deltas. Do not execute a tool until the finish reason indicates tool use and the accumulated arguments parse as valid JSON. Then run the tool, append its result as a tool message, and start the next model request. This two-step loop is why an agent can appear to stream continuously while actually alternating between model generation and external actions.

For cost-conscious production workloads, 59API provides access to native official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, plus GPT models, through one relay. Its pay-as-you-go pricing, low-cost positioning, Claude Code and Codex compatibility, and referral rebate can simplify both experimentation and ongoing usage. Sign up for 59API when you are ready to test the same streaming flow against the models your application needs.

Tarifs · Docs