← Voltar aos guias

LLM Streaming Under the Hood: From Tokens to SSE

LLM streaming does not make a model produce its entire answer instantly. It lets your application receive output while generation is still happening. The practical payoff is lower perceived latency: users can start reading before the response finishes. Here is the shortest useful path from model inference to a reliable streaming implementation.

1. Understand what happens before the first chunk

A typical autoregressive language model processes a request in two broad phases:

• Prefill: The model processes the prompt and builds attention state, commonly stored in a key-value cache. Long inputs can increase this work, although prompt caching may reduce it.
• Decode: The model generates output incrementally, usually selecting one token at a time and extending its cached state. Some serving systems accelerate this with techniques such as speculative decoding.

Streaming exposes incremental results from that generation process. It does not eliminate queueing, network latency, prompt processing, or internal reasoning that a model may perform before producing visible text.

Measure time to first token separately from total completion time. At the application layer, time to first visible text is often the most useful metric: the first streamed event may contain metadata rather than readable content.

2. Know what actually travels over HTTP

Many OpenAI-compatible streaming endpoints use Server-Sent Events, or SSE. The server keeps an HTTP response open and sends events, typically with the response content type text/event-stream.

An event can look like: data: {"choices":[{"delta":{"content":"Hello"}}]}. A blank line terminates the event. OpenAI-compatible Chat Completions streams commonly end with data: [DONE]. Other APIs, including Responses-style and native Claude interfaces, use different event schemas.

A streamed chunk is not necessarily one token. Providers may combine several tokens, emit empty deltas, or send role information and tool-call fragments. Network reads also do not align with SSE events: one read might contain half an event or several complete events.

3. Make a minimal streaming request

For an OpenAI-compatible Chat Completions route, set stream to true. With 59API, use the API host https://api.59api.com and the compatible /v1/chat/completions path. Export your API key as FIFTYNINE_API_KEY, then replace YOUR_ENABLED_MODEL with an exact model identifier available to your account.

curl -N https://api.59api.com/v1/chat/completions -H "Authorization: Bearer $FIFTYNINE_API_KEY" -H "Content-Type: application/json" -d '{"model":"YOUR_ENABLED_MODEL","messages":[{"role":"user","content":"Explain TCP in two sentences."}],"stream":true}'

The -N option disables curl's output buffering. Without it, your terminal can make a working stream appear delayed. Keep API keys on your server rather than embedding them in browser code.

Using an OpenAI SDK instead? Configure its base URL as https://api.59api.com/v1, enable streaming, and iterate over the returned stream. Read content deltas defensively because not every chunk contains text.

4. Parse events, not arbitrary network chunks

• Use the SDK parser when possible. A manual implementation needs incremental UTF-8 decoding, a persistent buffer, and proper SSE handling, including multiline data fields.
• Append text deltas. Do not treat each delta as a complete message or replace the existing answer with it.
• Assemble tool calls separately. Accumulate fragments by their call identifier or index. Parse arguments only after the call is complete; partial JSON is often invalid.
• Handle termination explicitly. Distinguish a normal finish, an output-limit finish, an error, and an unexpectedly disconnected stream.

5. Fix buffering and budget for real usage

If output arrives in bursts, inspect your own application server, reverse proxy, compression middleware, and CDN. Any layer can buffer the response. Flush promptly, avoid unnecessary batching, and propagate client cancellation upstream where supported.

Streaming itself generally does not make tokens cheaper. Canceling unnecessary generation may reduce usage, depending on provider behavior and billing. Retrying after partial output can generate duplicate text and additional charges, so avoid silently restarting an interrupted answer.

59API is a practical low-cost option for this setup: it offers cheap, pay-as-you-go Claude and GPT access using native, official-quality models without downgrades. Its compatibility with OpenAI SDKs, Claude Code, and Codex reduces integration friction, while its referral rebate offers another potential saving. Compare model-specific rates alongside measured latency and reliability.

If that fits your workload, sign up for 59API and test one short streaming request before connecting your full application.

Preços · Docs