Rate Limiting Architecture
Rate limiting caps how often a caller may hit a system over a window, protecting capacity, enforcing fairness across tenants, and pricing usage tiers. The core decisions are which algorithm, where the counter lives, and what to do on rejection.
A limiter sits in the request path, keyed by some identity (API key, user, IP, tenant, route), and on each call decides allow or deny. The output is usually an HTTP 429 Too Many Requests plus Retry-After and RateLimit-* headers so clients can back off intelligently. The hard part in production is not the single-node algorithm but making the decision correct and cheap across a fleet of stateless servers. Source: compiled from distributed-systems literature
Algorithms
| Algorithm | Behavior | Tradeoff |
|---|---|---|
| Fixed window | Count per discrete window (e.g. per minute) | Simple, but allows 2x burst at the window boundary |
| Sliding window log | Timestamp every request, count those in the trailing window | Exact, but O(N) memory per key |
| Sliding window counter | Weighted blend of current + previous fixed windows | Near-exact, O(1) memory — common default |
| Token bucket | Tokens refill at rate R, each call spends one; bucket cap = burst | Allows controlled bursts; the usual choice for APIs |
| Leaky bucket | Requests drain from a queue at a fixed rate | Smooths output to a constant rate; adds queueing latency |
Token bucket is the workhorse: it permits short bursts up to the bucket size while bounding the long-run average, which matches real client behavior better than a hard fixed cap.
Where the counter lives
- In-process — fastest, but each server only sees its own slice of traffic, so N servers permit ~N× the intended limit. Acceptable only for coarse local protection.
- Centralized store (Redis) — the standard distributed design. Counters live in Redis; an atomic Lua script does check-and-decrement in one round trip so concurrent requests cannot overspend. One extra hop per request (mitigated by co-location and pipelining). See Distributed Caching Architecture.
- Gateway-enforced — the limiter runs at the edge / API Gateway Architecture before requests reach app servers, shedding load earliest.
- Hybrid — a local token bucket as a fast first gate, reconciled against a central store, trading a little accuracy for far fewer round trips.
Design implications
- Reject cheaply and early. A limiter that does expensive work before denying defeats its purpose; enforce at the gateway and fail fast.
- Fairness vs protection. Per-tenant keys prevent one noisy neighbor from starving others (a Multi-Tenancy Architecture concern); a global key only protects total capacity.
- Degrade, don't collapse. Pair limiting with Resilience Patterns (Circuit Breaker, Bulkhead, Retry) (load shedding, queueing) so overload produces clean
429s, not cascading failure. - Make it observable. Emit per-key utilization; limits you cannot see are limits you cannot tune against Latency Budgets for Agents.
- Limits as product. Tiered limits are a pricing lever; the counters often feed the same usage signal as Billing Ledgers.
In Kevin's Dedalus work the pattern shows up as per-API-key throttling on the API Backend and as a registration/eligibility gate in the radar pipeline (Radar Registration Gate) — the same allow/deny-on-a-keyed-window shape applied to agent traffic rather than human clients.
Architecture Position
| Axis | Value |
|---|---|
| Family | Distributed systems primitives |
| Boundary owned | Capacity/fairness/spend protection over windows and callers. |
| Read with | API Gateway Architecture, Multi-Tenancy Architecture, Resilience Patterns (Circuit Breaker, Bulkhead, Retry) |
| Use this page when | protecting APIs from abuse or overload |
Timeline
-
2026-07-01 | Architecture category refresh added this page to the Distributed systems primitives family, linked it to Architecture System Map, and kept it standalone because it owns this boundary: Capacity/fairness/spend protection over windows and callers. Source: User request, 2026-07-01
-
2026-06-02 | Page created. Captured the allow/deny model, the five core algorithms with tradeoffs, counter-placement options (in-process / Redis / gateway / hybrid), and design implications around fairness, degradation, observability, and pricing. Source: compiled from distributed-systems literature and Kevin's Dedalus API work