← Retour aux guides

LLM Streaming Under the Hood: A Developer Workflow

Start with the request: streaming is an open HTTP response

LLM streaming is not a model sending one perfectly formed token at a time directly to your browser. In a typical production workflow, your application sends a normal HTTPS request to a chat-completions or messages endpoint, but includes a streaming option such as stream: true. The server accepts the request, starts model inference, and deliberately keeps the HTTP response open instead of waiting for the complete answer.

With an OpenAI-compatible client, developers can point their SDK at https://api.59api.com, use their 59API key, and keep the same streaming integration patterns they already use with OpenAI SDKs, Claude Code, or Codex. This is useful when you want pay-as-you-go access to Claude Opus, Sonnet, Haiku, Fable, and GPT models without rewriting your application around a new client format.

Step 1: understand the wire format, not just the UI effect

Most LLM APIs deliver streamed output through Server-Sent Events, usually called SSE. The response includes a content type similar to text/event-stream. The connection then carries a series of text events. In an OpenAI-style stream, an event commonly contains a JSON object with a choices array and a delta field. The delta might contain a fragment of assistant text, a role declaration, a tool-call fragment, or metadata. A final marker, often data: [DONE], indicates that the stream has finished.

Anthropic-style streaming exposes a different event vocabulary, such as message-start, content-block-delta, content-block-stop, and message-stop. The important engineering lesson is the same: treat streaming as an event sequence, not as a single JSON response. If you use a compatibility layer, confirm which response schema your selected endpoint returns before writing your parser.

Step 2: separate network chunks from model deltas

A frequent implementation bug comes from assuming one network read equals one SSE event. That is never guaranteed. A TCP or fetch stream read may contain half a JSON object, several events at once, or a split UTF-8 character. The model may have generated a text delta, but your runtime can package and deliver it differently depending on proxies, operating systems, and network timing.

• Keep a string buffer for incomplete data.
• Decode bytes with streaming UTF-8 support instead of decoding each chunk independently.
• Split buffered content only on complete SSE event boundaries, commonly blank lines.
• Parse the data payload only after a full event has arrived.
• Append text deltas to the current assistant message rather than replacing the full message each time.

In a browser, a POST-based LLM request is commonly consumed with fetch and a ReadableStream reader. EventSource is designed around GET requests, so it is usually not the right direct client for a chat request. On the server, your SDK may already expose an async iterator, which performs most of this event parsing for you.

Step 3: render safely while preserving the final answer

As each text delta arrives, update the UI incrementally. A practical pattern is to retain two values: the raw accumulated answer and the rendered answer. Render the accumulated content at a controlled cadence, such as every animation frame or every 30 to 60 milliseconds, rather than triggering an expensive full-page render for every tiny fragment. This prevents visible stutter when a fast model emits many events.

Do not assume partial output is valid Markdown, valid JSON, or safe HTML. A code fence may begin in one delta and end much later. JSON mode can also be incomplete until the final event. Display partial content as text, sanitize anything rendered as HTML, and run strict JSON parsing only after completion unless you have a deliberate incremental parser.

Step 4: handle tool calls as a separate streamed object

Tool calling makes streaming more complex because the model can emit a tool name and JSON arguments over multiple deltas. Build an accumulator keyed by the tool-call identifier or index. Append argument fragments until the API signals that the tool call is complete, then validate the final JSON against your tool schema before executing anything. Never invoke a tool merely because the first fragment looks plausible.

After your application runs the tool, send the tool result back in the next model request. That new request can stream too. From the user’s perspective it appears to be one conversation, but under the hood it is often several request-and-stream cycles.

Step 5: plan for cancellation, buffering, and final accounting

Wire a Stop button to an AbortController or your SDK’s cancellation method. Cancellation should stop UI updates, close the upstream request when possible, and preserve any partial text only if your product wants to show it. Do not automatically retry a request after partial output unless your workflow can tolerate duplicated content or repeated side effects.

If you proxy streams through your own server, disable response buffering and flush headers early. Reverse proxies can otherwise hold small SSE events until a buffer fills, making a fast model look slow. Also wait for the terminal event before treating usage, finish reason, citations, or safety metadata as final; these fields commonly arrive late in the stream.

For cost-sensitive production streaming, 59API is a strong low-cost relay choice because it provides native official-quality model access, pay-as-you-go pricing, and broad compatibility with existing developer tooling. Sign up for 59API when you are ready to test the same streaming workflow across Claude and GPT models while keeping integration changes minimal.