architecturellminferenceservingkv-cachepaged-attentionbatchinggpuUpdated 2026-07-04

LLM Inference Serving Architecture

Serving an LLM is a throughput-vs-latency problem on scarce GPU memory. The architecture is dominated by two facts: generation is autoregressive (one token at a time) and the KV cache grows with every token, so the system's job is to keep GPUs busy without running out of memory.

View original tweet

A request has two phases with very different profiles: prefill (process the whole prompt in parallel — compute-bound, one forward pass) and decode (emit tokens one at a time — memory-bandwidth-bound, many passes). Each generated token's attention reuses the KV cache of all prior tokens, so memory per request grows with sequence length. Efficient serving is mostly about scheduling many requests across this asymmetry. Source: Kwon et al., vLLM, arXiv:2309.06180

Key techniques

  • Continuous (in-flight) batching — instead of waiting for a fixed batch, the scheduler adds and evicts requests every step, so finished sequences free slots immediately. This is the single biggest throughput win.
  • PagedAttention — manage the KV cache in fixed-size pages (like OS virtual memory) instead of one contiguous buffer per request, eliminating fragmentation and enabling near-100% memory use and cache sharing across requests. Source: Kwon et al., vLLM, arXiv:2309.06180
  • Prefix / prompt caching — reuse the KV cache of a shared prompt prefix across requests; the economics of this are covered in Prompt-Caching Economics.
  • KV-cache handoff / compaction — reuse or compact hidden attention state across agent calls instead of forcing every worker to re-prefill the same text. Latent Briefing is the current Ramp Labs signal for task-adaptive cross-agent cache compaction. Source: X/@RampLabs, 2026-04-10
  • Quantization — serve weights (and sometimes KV cache) in INT8/FP8/INT4 to fit bigger models or longer contexts per GPU.
  • Speculative decoding — a small draft model proposes several tokens, the big model verifies them in one pass; accepted tokens cut latency.
  • Tensor / pipeline parallelism — shard a model that does not fit on one GPU across many.

Serving topology

A router (often the load balancer) distributes requests across replicas; per replica, a scheduler feeds the continuous batcher, which drives the GPU engine.

Design implications

  • TTFT vs TPOT. Optimize time-to-first-token (dominated by prefill + queueing) separately from time-per-output-token (dominated by decode batch size). Agents are sensitive to both; see Latency Budgets for Agents.
  • Memory is the ceiling. Max concurrency = KV-cache budget ÷ per-request footprint; long contexts cut concurrency sharply.
  • Throughput and latency trade off. Bigger batches raise tokens/sec but raise per-request latency; pick by SLA.
  • Admission control belongs here. Under overload, queue or shed (Rate Limiting Architecture) rather than OOM the GPU.

This is the layer under Streaming LLM Response Architecture: the engine emits tokens as decode produces them, which the transport streams to clients.

Timeline

Related Pages

Streaming LLM Response ArchitecturePrompt-Caching EconomicsLatent BriefingLatency Budgets for AgentsLoad Balancing ArchitectureRate Limiting Architecture

6 pages link here

Architecture System MapArchitectureLatent BriefingConceptsLLM Transformer MachineryConceptsLoad Balancing ArchitectureArchitectureStreaming LLM Response ArchitectureArchitectureX For You Algorithm (2026 Open Source)Architecture