Sharding & Partitioning

Sharding (horizontal partitioning) splits a dataset across multiple database instances, each owning a disjoint key range or hash bucket, enabling linear write and storage scaling beyond what a single node can sustain. The sharding key selection is critical: a poor key creates hot shards; a good key distributes load evenly and keeps related data co-located. The three primary strategies — range, hash, and directory-based sharding — each offer different trade-offs for query efficiency, rebalancing complexity, and hot-spot risk.

Key Points

• Range sharding (customer_id 0–999 → shard 1, 1000–1999 → shard 2) enables range scans but creates hot spots when IDs are sequential and writes cluster at the high end.
• Hash sharding (shard = hash(key) % N) distributes writes uniformly but destroys locality — range queries require scatter-gather across all shards.
• Directory-based sharding uses a lookup table (shard map) to route each key to its shard; flexible (keys can be moved) but the directory is a single point of failure and a bottleneck.
• Cross-shard queries (JOIN across shards, aggregations) require application-level fan-out and merge — avoid designs where common queries span many shards.
• Re-sharding (changing N) is the most painful operation: hash sharding requires rehashing ~all keys; range sharding requires splitting ranges and moving data; directory sharding updates the map.
• MongoDB automatic sharding uses a shard key + chunks (64 MB by default); the balancer migrates chunks between shards to keep counts equal — causes latency spikes during migration.
• Vitess (YouTube's MySQL sharding layer) adds a topology service (etcd), query routing, and online schema changes on top of MySQL, handling Google-scale sharding transparently.
• Tenant-based sharding (one shard per large customer) simplifies compliance and data isolation but creates variable-size shards; combine with consistent hashing for mid-tier tenants.