Replication helps when you want multiple copies of the same data for availability, latency, or read scaling. But replication alone does not solve the problem of a dataset that is too large for one machine or a write/query load that one machine cannot handle. Partitioning is the next move: instead of making more copies of the same data, you split the dataset itself across machines.
This chapter is foundational for both interviews and real-world design because it turns the vague statement "we'll shard it later" into concrete engineering choices: what is the shard key?, how do you avoid hot spots?, what happens to secondary indexes?, how do you rebalance as the cluster grows?, and who knows where the data lives?
Mission tie-in: Chapter 6 is where distributed storage stops being just a copy problem and becomes a placement problem. The central challenge is not merely splitting data, but splitting it in a way that keeps load balanced, queries efficient, and future growth manageable.
Partitioning, also called sharding, means dividing a large dataset into smaller subsets called partitions. Each record belongs to exactly one partition from the logical point of view, even if that partition is replicated onto multiple nodes for fault tolerance.
You can think of each partition as a small database of its own. Together, the partitions form the whole database. Once data is partitioned, different machines can store different pieces of the dataset and process requests for those pieces independently.
This is what lets a shared-nothing cluster scale horizontally. If a query or write touches only one partition, then the responsible node can process it independently. That independence is the whole reason partitioning works.
A very important idea early in the chapter is that partitioning and replication solve different problems:
| Technique | What it does | Main goal |
|---|---|---|
| Replication | Makes multiple copies of the same partition | Availability, latency, read scale |
| Partitioning | Splits the dataset into different pieces | Storage scale and throughput scale |
In production systems, you almost always combine them. Each partition may have replicas, and each node may lead some partitions and follow others. But the decision of how to split the data is mostly independent of the decision of how to replicate each split.
The purpose of partitioning is not just to split data somehow. It is to spread both data volume and query load evenly.
If one partition gets much more data or traffic than others, partitioning stops helping. That imbalance is called skew. If one partition becomes disproportionately busy, it becomes a hot spot.
If you assign records to nodes randomly, distribution may be decent, but lookup becomes awful: to find one item, you may need to ask every node. That destroys efficiency.
Good partitioning therefore has to solve two problems at once:
One approach is to sort the keys and assign each partition a continuous range, like pages or volumes in an encyclopedia. If you know the range boundaries, then you can route a key directly to the correct partition.
This is why range partitioning is especially appealing for workloads where ordered access matters, such as time-series or lexicographically ordered data.
A subtle but important point: partition boundaries should usually be chosen according to data distribution, not according to the apparent shape of the key domain. For example, alphabetical keys are not uniformly distributed, so equal letter ranges would not create equal partition sizes.
The right question is not "what key interval looks symmetric?" but rather "how much data and traffic will land in each interval?"
The same property that gives range scans also creates danger. If writes arrive in a monotonic order, then they all target the newest end of the range.
The timestamp example is the classic case: if keys are timestamps and partitions represent time ranges, then today's writes all hammer today's partition. Older partitions sit mostly idle.
One fix is to make the first component of the key something with higher distribution diversity, such as sensor ID, tenant ID, or user ID, and put timestamp second. Then writes spread across many first-key buckets while still preserving local order within each bucket.
This is an important design pattern:
Composite key trick: use one part of the key to distribute load and another part to preserve useful order. You give up one global range scan in exchange for many efficient per-entity range scans.
To avoid hot spots caused by clustered key values, many systems hash the key first and partition by hash range instead of raw key range.
A good hash function takes similar-looking or skewed inputs and spreads them uniformly across a large output range. Then each partition owns a slice of that hash space.
Hashing destroys natural ordering. Keys that were adjacent before hashing get scattered across partitions. That means efficient range queries over the primary key are largely lost.
| Question | Key-range partitioning | Hash partitioning |
|---|---|---|
| Point lookup by key | Good | Good |
| Range scan on key | Excellent | Poor |
| Load balancing under monotonic keys | Risky | Better |
| Preserves locality | Yes | No |
DDIA makes an important correction here. People often loosely call hash partitioning "consistent hashing," but the original meaning of consistent hashing is more specific, tied to certain random boundary assignment techniques for cache systems.
The book's advice is pragmatic: avoid overusing that term loosely. In most database discussions, it is clearer to say hash partitioning unless you really mean the specific consistent-hashing technique.
Pure key-range partitioning gives order but risks hot spots. Pure hash partitioning spreads load but destroys order. Some systems combine both ideas using a compound primary key.
Cassandra is the chapter's canonical example. It hashes only the first part of the primary key to determine the partition, while the remaining columns determine sorted order inside that partition.
This is elegant for one-to-many relationships. For example, a primary key like
(user_id, update_timestamp) means:
user_id.
This pattern appears constantly in real systems. The price is that you can no longer do a range query across all users by timestamp alone without additional indexing or scanning.
Hashing spreads different keys well. It does nothing when one single key is itself extremely hot. If millions of operations target one user ID or one content ID, the hash of that key still points to one partition.
This is the celebrity problem: when one item becomes globally popular, partition balance by key is no longer enough.
A common mitigation is to artificially split one hot key into many derived keys by appending
or prepending randomness, such as
hotKey-00 to hotKey-99. That spreads writes across many partitions.
But the read path gets worse: reads must now fan out to all subkeys and merge results. You also need metadata telling the application which keys are hot and therefore sharded specially.
Deep tradeoff: relieving a hot spot often means deliberately making reads or writes more complex. There is rarely a free fix.
So far partitioning has been relatively straightforward because the primary key tells you where a record lives. Secondary indexes break that simplicity.
A secondary index answers questions like:
The problem is that the secondary-index value usually does not align neatly with the primary-key partitioning. There are two broad ways to handle this: document-partitioned and term-partitioned indexes.
In a document-partitioned index, each partition stores the secondary indexes only for the documents physically stored in that partition. In other words, the index is local to the primary-key partition.
If red cars are distributed across many primary-key partitions, then a query for
color = red must ask all partitions, collect their partial results, and merge
them.
This is called scatter/gather. It is simple conceptually, but costly in practice:
This is why local-index designs are often good for write-heavy workloads but can become painful for rich search patterns.
Instead of storing an index locally next to each document, you can build a
global index that covers all documents across the database. Then partition
that index by the indexed term itself, such as color:red or
make:honda.
A query for a term can go directly to the partition that owns that term in the index, instead of broadcasting to all document partitions. This can make reads dramatically more efficient.
A write to one document may affect many index terms, and those terms may live on many different partitions and nodes. So one logical write can become several distributed updates.
| Property | Local index | Global index |
|---|---|---|
| Write complexity | Lower | Higher |
| Read complexity | Higher due to scatter/gather | Lower for term lookups |
| Need cross-partition coordination | Less | More |
| Likelihood of asynchronous propagation | Lower | Higher |
To update a global secondary index synchronously with every write, the system may need a distributed transaction across multiple index partitions. Many databases avoid that cost and complexity by updating global indexes asynchronously.
That means a freshly written document may not immediately appear in index results. The index becomes eventually consistent with the base data.
This is a major interview insight: query flexibility often comes with write complexity and delayed index consistency.
Once you partition data, the job is not finished. Over time:
The system therefore has to move partitions or parts of partitions around. This process is called rebalancing.
That last point is not just optimization. Moving data is expensive: it consumes network bandwidth, disk I/O, and operational attention. Bad rebalancing can make the cluster unstable while it is trying to improve it.
The seemingly obvious scheme is: compute hash(key) % N, where N is
the number of nodes. That directly maps keys to nodes.
It looks elegant until the cluster size changes. If you add or remove one node, the modulo changes for a huge fraction of keys, so most of the dataset has to move.
Why mod N is bad: the key-to-node mapping depends directly on the current node count, so a tiny topology change causes a massive data reshuffle.
This is one of the classic early mistakes in distributed-system design.
A common solution is to create many more partitions than nodes from the start and assign several partitions to each node. Then, when a node is added, it "steals" some full partitions from other nodes.
The number of partitions has to be chosen up front. Too few partitions and future growth becomes constrained. Too many partitions and metadata/management overhead grows.
Also, partition size becomes proportional to total dataset size. If the dataset grows a lot, partitions get huge, making recovery and rebalancing more expensive.
This is a strong engineering lesson: fixed upfront decisions often shift complexity into capacity planning.
Instead of fixing partitions forever, some systems split partitions when they get too big and merge them when they get too small. This is especially natural for key-range partitioning.
If the system starts with one partition and no prior knowledge, then at the beginning all writes land on one node until the first splits happen. That can make a new cluster underutilize its available hardware.
Some systems address this with pre-splitting: create an initial set of partitions before data arrives. But pre-splitting requires you to know something about future key distribution, which is often hard.
Dynamic partitioning trades some operational simplicity for better long-run adaptability. It works best when partition size needs to stay within a target band rather than being fixed forever.
A third approach is to keep a fixed number of partitions per node. When a new node joins, it splits some existing partitions and takes ownership of half of each. Cassandra's vnode design is the best-known example.
This keeps partition size relatively stable as the cluster grows, because more nodes mean more partitions overall.
More partitions mean more metadata and more movement bookkeeping. Random boundary choices can also create unfair splits if not handled carefully, which is why implementations evolve over time.
Even after choosing a rebalancing algorithm, there is a separate operational question: should the system rebalance automatically, or should a human confirm the change?
Rebalancing is expensive. If the system misinterprets a transient slowdown as a node failure and starts moving lots of data, it adds more load to an already stressed cluster. That can trigger a cascading failure.
This is exactly the kind of failure mode that separates "works in theory" from "operates safely in production."
Operational lesson: some of the most dangerous distributed-system bugs come from automation that is individually reasonable but globally destabilizing.
A human in the loop slows things down, but may prevent the cluster from making a bad situation worse.
Once data is partitioned, clients need a way to reach the correct node. That sounds simple, but partition ownership changes during rebalancing, failover, or membership changes. So the real problem is not static lookup, but keeping routing knowledge current.
| Model | How it works | Main tradeoff |
|---|---|---|
| Any-node forwarding | Client talks to any node; that node forwards if needed | Simpler clients, smarter nodes |
| Routing tier | Client talks to a partition-aware proxy/load balancer | Centralized routing logic, extra hop |
| Partition-aware client | Client knows partition-to-node map and talks directly | Fast direct access, more complex clients |
None of these removes the underlying problem. Somebody still needs authoritative cluster metadata.
All participants must agree enough on which node owns which partition. If different components believe different mappings, requests are sent to the wrong place. That turns routing into a distributed-consensus-flavored metadata problem.
Many systems use a separate coordination service such as ZooKeeper. Nodes register themselves there, and the service maintains the authoritative mapping from partitions to nodes. Routers or clients subscribe to updates.
This externalizes coordination and keeps metadata management separate from the storage engine, at the cost of an additional dependency.
Other systems, such as Cassandra and Riak, propagate cluster-state information among the database nodes themselves via gossip. Then a client can send a request to any node, and that node forwards appropriately.
This removes the external coordinator but increases the complexity inside the database nodes.
This is a recurring design tradeoff in distributed systems: centralize coordination in a dedicated system, or distribute the coordination complexity across the data nodes themselves.
Most of the chapter focuses on point lookups, writes, and simple scatter/gather patterns typical of many NoSQL systems. But for analytics and data warehouses, partitioning enables something deeper: massively parallel query execution.
A complex SQL query with joins, filters, grouping, and aggregation can be decomposed into stages, and many of those stages can run on multiple nodes in parallel. This is one of the main reasons partitioning is so powerful in analytical systems.
The key idea is that partitioning is not only about storage placement. It is also about creating opportunities for execution parallelism.
| Decision | Main upside | Main downside | Best fit |
|---|---|---|---|
| Key-range partitioning | Efficient range scans and locality | Hot spots under monotonic access | Ordered/time-series/entity-local scans |
| Hash partitioning | Evener load distribution | Loses range-query locality | Primary-key point lookups |
| Local secondary indexes | Fast/simple writes | Scatter/gather reads | Write-heavy systems |
| Global secondary indexes | Efficient indexed reads | More complex/slower writes | Read-heavy search/filter workloads |
| Fixed partitions | Simple operational model | Capacity planning burden | Predictable growth and stable workloads |
| Dynamic partitions | Adapts to data volume | Split/merge operational complexity | Variable growth and range-partitioned data |
When an interviewer says "how would you shard this?" a strong answer should go beyond naming a shard key. You should talk through these questions explicitly:
Interview-quality answer: "I would choose the partition key based on the
highest-volume access path, but I would explicitly check whether that key is monotonic or
skew-prone. If I need per-user time-ordered reads, I would use a composite key like
(user_id, timestamp) so I get distribution by user and order within user. I
would also be careful with global secondary indexes because they improve read efficiency at
the cost of write complexity and possible asynchronous propagation."
On the surface, Chapter 6 is about sharding strategies. At a deeper level, it is about controlling where work happens.
In short, partitioning is where logical data modeling and physical cluster behavior collide. A good shard key is not just a data-model choice. It is a load-distribution decision, an indexing decision, a future-operations decision, and often a product-behavior decision too.
1. What problem does partitioning solve that replication alone does not?
Splitting data volume and throughput across machines Encrypting data across regions Making all replicas strongly consistent2. What is skew?
Uneven distribution of data or load across partitions A sorted order of keys within a partition Different replicas holding different values3. What is the main strength of key-range partitioning?
Efficient ordered and range queries Perfect immunity to hot spots It is only useful for exact key lookups4. Why can key-range partitioning create hot spots?
Monotonic or clustered keys direct many writes to one range Because range partitioning only supports two partitions Because followers cannot replicate ordered keys5. What is the main cost of hash partitioning?
It loses efficient key-order range scans It makes point lookups impossible It always creates celebrity hot spots6. What is the key difference between local and global secondary indexes?
Local indexes favor writes; global indexes favor indexed reads Local indexes only work in SQL databases Global indexes cannot be replicated7. Why is hash(key) % N a bad rebalancing strategy?
8. What is the key advantage of dynamic partitioning?
Partition count adapts to dataset size It removes the need for routing metadata It guarantees no hot spots under any key pattern9. Why can automatic rebalancing be dangerous?
It can destabilize an already stressed cluster Because routing cannot be automated at all Because automatic systems cannot use replication10. What is the core routing problem in a partitioned cluster?
Knowing which node currently owns which partition Choosing whether to use IPv4 or IPv6 Making every client connect through DNS round-robin onlyReplication = more copies ; Partitioning = split dataset itself
Main goal :- spread data volume + query load across many machines
Each record logically belongs to one partition
Need even distribution
Skew = uneven data/load distribution
Hot spot = one partition gets disproportionate traffic
Random placement alone × bad because lookup would need all nodes
Each partition owns continuous key interval
Big advantage = ordered storage + efficient range scan
Boundaries should match data distribution, not just key aesthetics
Main danger = monotonic keys (like timestamp) create hot latest partition
Put distributing field first ; ordering field second
Example :- (sensor_id, timestamp) or (user_id, timestamp)
Load spreads across entities ; order preserved within one entity
Hash key first ; partition by hash range
Good for even distribution
Bad for range scans because natural key order destroyed
Say hash partitioning unless you really mean original consistent hashing technique
Hash first component of compound key ; sort by remaining components inside partition
Example :- (user_id, update_timestamp)
Good for one-to-many entity timeline pattern
Hashing many different keys helps ; one single celebrity key still hot
Mitigation :- split one hot key into many synthetic subkeys using random suffix/prefix
Cost :- reads must fan out + merge ; need bookkeeping for which keys are split
Primary key gives partition directly ; secondary index usually does not
Need either document-partitioned or term-partitioned strategy
Each partition stores secondary index only for its own documents
Writes simple because update stays local
Reads expensive because query may need scatter/gather across all partitions
Tail latency gets amplified by slowest participant
Index partitioned by indexed term itself, not by primary document location
Reads efficient because go to term owner directly
Writes more complex because one document update may touch many index partitions
Often updated asynchronously → index may lag base data
When nodes added/removed or fail, need move data/load
Good rebalancing :- fair final load ; continue reads/writes ; move minimum data
If node count changes, mapping of huge fraction of keys changes
So tiny cluster-size change causes massive data movement
Create many partitions upfront ; assign many per node
New node steals whole partitions from old nodes
Simple operationally ; but partition count must be chosen upfront
Too many partitions = overhead ; too few = growth pain
Split large partitions ; merge small partitions
Adapts to data volume
But empty DB may start with one partition only ; all early writes hit one node
Pre-splitting helps if future key distribution known
Fixed number of partitions per node (vnode style)
When node joins, split some existing partitions and take halves
Keeps partition size more stable as cluster grows
Full automation convenient but risky
Bad failure detector + auto rebalance can worsen overload and cause cascading failure
Human in loop slower but safer in many prod systems
1) send to any node ; it forwards if needed
2) send to routing tier / partition-aware load balancer
3) client itself knows partition-to-node map
Main hard part = keep partition ownership metadata correct as cluster changes
Use external coordinator like ZooKeeper / Helix, or spread metadata via gossip
External coordinator = simpler data nodes but extra dependency
Gossip = no external coordinator but more logic inside nodes
Primary source: Kleppmann, M. (2017).
Designing Data-Intensive Applications, Chapter 6: “Partitioning.”
O'Reilly Media.
Recommended supplements: Cassandra data modeling docs, HBase region-splitting
docs, Elasticsearch sharding/indexing guides, and practical material on ZooKeeper/Helix-style
metadata coordination.
Ask follow-up questions if you want to drill into shard-key design, celebrity hot keys, local vs global indexes, or how to explain rebalancing tradeoffs in a system design interview.