Load Balancing Architecture
A load balancer spreads incoming requests across a pool of backend instances so no single one is overwhelmed, and routes around unhealthy ones. It is the front door that turns N identical servers into one horizontally-scalable, highly-available service.
The balancer sits between clients and a server pool, picks a backend per request by some algorithm, and continuously health-checks the pool so traffic only goes to live instances. This is what lets you add capacity by adding instances and survive instance failure transparently. Source: compiled from load-balancing literature
L4 vs L7
| Layer | Operates on | Can do | Cost |
|---|---|---|---|
| L4 (transport) | TCP/UDP, IP + port | Fast, protocol-agnostic forwarding | No content awareness |
| L7 (application) | HTTP — path, headers, cookies | Content routing, TLS termination, sticky sessions, header rewrite | More CPU per request |
L4 maximizes raw throughput; L7 enables smart routing (and is what an API gateway builds on).
Algorithms
- Round robin — rotate through instances; even when requests are uniform.
- Least connections — send to the instance with fewest active connections; better for variable request durations.
- Weighted — bias toward bigger instances.
- Consistent hashing — hash a key (client IP, session, cache key) to an instance so the same key keeps hitting the same backend with minimal reshuffling when the pool changes — essential for cache affinity and stateful routing (same primitive as Sharding and Partitioning Architecture).
- Latency / EWMA — route to the fastest-responding instance.
Health checks
- Active — the balancer probes a health endpoint on a schedule.
- Passive — it observes real traffic and ejects instances that error/time out (outlier ejection), overlapping with resilience.
Failing instances are removed from rotation and re-added when healthy again.
Design implications
- The balancer must not be the SPOF. Run it HA (active-passive or anycast); DNS or a virtual IP fails over.
- Sticky sessions vs statelessness. Affinity (consistent hashing / cookies) helps caches and stateful backends but undermines even spreading and complicates deploys; prefer stateless backends where possible.
- Drain on deploy. Gracefully drain connections from an instance before removing it, or in-flight requests break.
- LLM serving is a specialized case — route by KV-cache locality / prefix so requests sharing a prompt prefix hit the same replica (LLM Inference Serving Architecture).
Architecture Position
| Axis | Value |
|---|---|
| Family | Distributed systems primitives |
| Boundary owned | Distributing requests across backends with health checks and routing algorithms. |
| Read with | API Gateway Architecture, Service Mesh and Service Discovery Architecture, LLM Inference Serving Architecture |
| Use this page when | balancing traffic or model serving load |
Timeline
-
2026-07-01 | Architecture category refresh added this page to the Distributed systems primitives family, linked it to Architecture System Map, and kept it standalone because it owns this boundary: Distributing requests across backends with health checks and routing algorithms. Source: User request, 2026-07-01
-
2026-06-02 | Page created. Captured the spread-and-route-around-failure role, L4 vs L7, the algorithm catalog (round-robin/least-conn/weighted/consistent-hash/latency), active vs passive health checks, and implications (HA, stickiness, connection draining, LLM prefix routing). Source: compiled from load-balancing / consistent-hashing literature