Sharding and Partitioning Architecture
When data or load outgrows one machine, partition (shard) it: split the dataset into pieces, each owned by a node, so capacity and throughput scale horizontally. The whole game is choosing a partition scheme that spreads load evenly while keeping common queries efficient.
Partitioning answers "which node owns this key?" Replication (a separate concern, see Replication and Consistency Models) answers "how many copies and where?" — production systems do both: shard for capacity, replicate each shard for availability. The defining risk of sharding is skew: a scheme that concentrates data or traffic on one shard (a hot spot) defeats the purpose. Source: Kleppmann, Designing Data-Intensive Applications
Partitioning strategies
| Strategy | How | Pro | Con |
|---|---|---|---|
| Range | Contiguous key ranges per shard | Efficient range scans | Hot spots on sequential keys (timestamps) |
| Hash | hash(key) mod N → shard |
Even spread | Kills range queries; resharding moves most keys |
| Consistent hashing | Keys + nodes on a ring; key → next node clockwise | Adding/removing a node moves only ~1/N keys | Uneven without virtual nodes |
| Directory / lookup | A lookup table maps key → shard | Flexible, rebalanceable | The directory is a dependency + SPOF risk |
Consistent hashing (with virtual nodes for balance) is the standard for elastic clusters precisely because plain mod N reshuffles nearly everything when N changes.
Secondary indexes
- Local (document-partitioned) — each shard indexes only its own data; writes are cheap, but a query by the indexed field must scatter-gather across all shards.
- Global (term-partitioned) — the index itself is partitioned by term; reads hit one shard, but writes must update a remote index partition.
This read-vs-write tradeoff is a core sharding decision.
Design implications
- Pick the key for the dominant query. Co-locate data that is read together; cross-shard joins/transactions are expensive.
- Plan rebalancing from day one. Fixed partition counts (more partitions than nodes, reassigned in bulk) avoid the
mod Nreshuffle. - Hot keys need special handling. Split a celebrity key, add randomness, or cache it (Distributed Caching Architecture).
- Routing. A request router (or smart client) maps key → shard — the same dispatch role as a load balancer.
The same key-to-owner mapping appears in log partitions, vector shards, and per-tenant data isolation in Multi-Tenancy Architecture.
Architecture Position
| Axis | Value |
|---|---|
| Family | Distributed systems primitives |
| Boundary owned | Splitting data/load across partitions and owners. |
| Read with | Replication and Consistency Models, Load Balancing Architecture, Multi-Tenancy Architecture |
| Use this page when | scaling data beyond one owner |
Timeline
-
2026-07-01 | Architecture category refresh added this page to the Distributed systems primitives family, linked it to Architecture System Map, and kept it standalone because it owns this boundary: Splitting data/load across partitions and owners. Source: User request, 2026-07-01
-
2026-06-02 | Page created. Captured partition-vs-replication, skew/hot-spots, the range/hash/consistent-hashing/directory strategies, local vs global secondary indexes, and implications (key choice, rebalancing, hot keys, routing). Source: Kleppmann, Designing Data-Intensive Applications