Streaming LLM Response Architecture
Because generation is token-by-token, streaming each token to the client as it is produced cuts perceived latency dramatically — the user sees output at time-to-first-token instead of waiting for the whole completion. The architecture is a chain of streams that must preserve order and propagate backpressure end to end.
The default transport is Server-Sent Events (SSE): a long-lived HTTP response with Content-Type: text/event-stream over which the server pushes data: frames. It is one-directional (server→client), text-based, and auto-reconnects, which fits chat completions exactly. WebSockets are used when the client must also stream up (live audio, interactive tools); plain chunked HTTP is the lowest-common-denominator fallback. Source: compiled from SSE/HTTP streaming specs
Vercel's 2026 WebSockets public beta makes that bidirectional branch practical on Vercel-hosted apps: Functions can keep a WebSocket open, the connection stays pinned to one Function instance, and Fluid Compute lets one instance handle multiple sockets. This changes the platform tradeoff for Vercel apps but not the transport rule: SSE remains the default for one-way token streams; WebSockets are for realtime client-to-server messages, collaboration, live audio, or agent-control channels. Source: Vercel changelog, 2026-06-22; Source: Vercel WebSockets docs, 2026-07-04
The stream chain
Every hop must stream, not buffer. A single buffering layer — a proxy that waits for the full body, a gateway with response buffering on, a framework that collects before sending — collapses streaming back into a single slow response. This is the most common streaming bug.
Components and concerns
- Token source — the serving engine yields tokens during decode.
- Server framing — wrap tokens as SSE events; send a sentinel (e.g.
[DONE]) to mark completion, and structured events for tool calls, usage, and errors. - Proxy/gateway config — disable response buffering and set generous read timeouts at every proxy (API Gateway Architecture); enable HTTP/1.1+ keep-alive.
- Backpressure — if the client reads slowly, the pressure must propagate back to the engine so it does not generate into an unbounded buffer; honor the engine's flow control.
- Cancellation — when the client disconnects, abort the upstream generation to stop burning GPU on output nobody will read.
- Error mid-stream — once headers are sent you cannot change the status code; emit a typed error event and close, and make the client handle partial responses.
Design implications
- TTFT is the metric users feel. Streaming is a UX architecture decision as much as a transport one; see Latency Budgets for Agents.
- Reconnection needs idempotency or resume tokens. A dropped SSE connection should not silently truncate or double-charge.
- Buffering anywhere kills it. Audit the whole path; the weakest (buffering) link defines behavior.
- Tool-calling complicates the contract. Streaming agents interleave text, tool calls, and results in one stream — define the event schema up front.
Kevin's Dedalus API Backend streams model output over this exact shape, and the Website Frontend renders it incrementally; the agent's tool calls and observations are framed as structured events on the same stream.
Timeline
- 2026-07-04 | Added the Vercel WebSockets public beta as the hosted-platform branch for bidirectional realtime streams: Functions can serve WebSocket connections on Fluid Compute, while SSE stays the default for one-way LLM text streaming. Source: X/@vercel_dev, 2026-06-22; Source: Vercel docs, 2026-07-04
- 2026-06-02 | Page created. Captured SSE as the default transport, the end-to-end stream chain and the buffering pitfall, framing/backpressure/cancellation/mid-stream-error concerns, and the TTFT-as-UX implication. Source: compiled from SSE/HTTP streaming specs and LLM application patterns