Lesson 6: Partitioning

Lesson 0006 — 50 min read

Replication helps when you want multiple copies of the same data for availability, latency, or read scaling. But replication alone does not solve the problem of a dataset that is too large for one machine or a write/query load that one machine cannot handle. Partitioning is the next move: instead of making more copies of the same data, you split the dataset itself across machines.

This chapter is foundational for both interviews and real-world design because it turns the vague statement "we'll shard it later" into concrete engineering choices: what is the shard key?, how do you avoid hot spots?, what happens to secondary indexes?, how do you rebalance as the cluster grows?, and who knows where the data lives?

Mission tie-in: Chapter 6 is where distributed storage stops being just a copy problem and becomes a placement problem. The central challenge is not merely splitting data, but splitting it in a way that keeps load balanced, queries efficient, and future growth manageable.

What Partitioning Really Is

Partitioning, also called sharding, means dividing a large dataset into smaller subsets called partitions. Each record belongs to exactly one partition from the logical point of view, even if that partition is replicated onto multiple nodes for fault tolerance.

You can think of each partition as a small database of its own. Together, the partitions form the whole database. Once data is partitioned, different machines can store different pieces of the dataset and process requests for those pieces independently.

Why partitioning exists

This is what lets a shared-nothing cluster scale horizontally. If a query or write touches only one partition, then the responsible node can process it independently. That independence is the whole reason partitioning works.


Partitioning and Replication Are Separate Axes

A very important idea early in the chapter is that partitioning and replication solve different problems:

Technique What it does Main goal
Replication Makes multiple copies of the same partition Availability, latency, read scale
Partitioning Splits the dataset into different pieces Storage scale and throughput scale

In production systems, you almost always combine them. Each partition may have replicas, and each node may lead some partitions and follow others. But the decision of how to split the data is mostly independent of the decision of how to replicate each split.


The Core Goal: Even Distribution

The purpose of partitioning is not just to split data somehow. It is to spread both data volume and query load evenly.

If one partition gets much more data or traffic than others, partitioning stops helping. That imbalance is called skew. If one partition becomes disproportionately busy, it becomes a hot spot.

Why naive random placement is not enough

If you assign records to nodes randomly, distribution may be decent, but lookup becomes awful: to find one item, you may need to ask every node. That destroys efficiency.

Good partitioning therefore has to solve two problems at once:

  1. Place data fairly across the cluster.
  2. Make it easy to determine where a given record or query should go.

Partitioning by Key Range

One approach is to sort the keys and assign each partition a continuous range, like pages or volumes in an encyclopedia. If you know the range boundaries, then you can route a key directly to the correct partition.

What makes key-range partitioning attractive

This is why range partitioning is especially appealing for workloads where ordered access matters, such as time-series or lexicographically ordered data.

The boundaries should not be evenly spaced in key space

A subtle but important point: partition boundaries should usually be chosen according to data distribution, not according to the apparent shape of the key domain. For example, alphabetical keys are not uniformly distributed, so equal letter ranges would not create equal partition sizes.

The right question is not "what key interval looks symmetric?" but rather "how much data and traffic will land in each interval?"

The downside: hot spots from locality

The same property that gives range scans also creates danger. If writes arrive in a monotonic order, then they all target the newest end of the range.

The timestamp example is the classic case: if keys are timestamps and partitions represent time ranges, then today's writes all hammer today's partition. Older partitions sit mostly idle.

How to reduce that hot spot

One fix is to make the first component of the key something with higher distribution diversity, such as sensor ID, tenant ID, or user ID, and put timestamp second. Then writes spread across many first-key buckets while still preserving local order within each bucket.

This is an important design pattern:

Composite key trick: use one part of the key to distribute load and another part to preserve useful order. You give up one global range scan in exchange for many efficient per-entity range scans.


Partitioning by Hash of Key

To avoid hot spots caused by clustered key values, many systems hash the key first and partition by hash range instead of raw key range.

A good hash function takes similar-looking or skewed inputs and spreads them uniformly across a large output range. Then each partition owns a slice of that hash space.

Why hash partitioning helps

Why hash partitioning hurts

Hashing destroys natural ordering. Keys that were adjacent before hashing get scattered across partitions. That means efficient range queries over the primary key are largely lost.

Question Key-range partitioning Hash partitioning
Point lookup by key Good Good
Range scan on key Excellent Poor
Load balancing under monotonic keys Risky Better
Preserves locality Yes No

Important terminology warning: consistent hashing

DDIA makes an important correction here. People often loosely call hash partitioning "consistent hashing," but the original meaning of consistent hashing is more specific, tied to certain random boundary assignment techniques for cache systems.

The book's advice is pragmatic: avoid overusing that term loosely. In most database discussions, it is clearer to say hash partitioning unless you really mean the specific consistent-hashing technique.


Cassandra's Hybrid Trick: Hash First, Sort Second

Pure key-range partitioning gives order but risks hot spots. Pure hash partitioning spreads load but destroys order. Some systems combine both ideas using a compound primary key.

Cassandra is the chapter's canonical example. It hashes only the first part of the primary key to determine the partition, while the remaining columns determine sorted order inside that partition.

This is elegant for one-to-many relationships. For example, a primary key like (user_id, update_timestamp) means:

This pattern appears constantly in real systems. The price is that you can no longer do a range query across all users by timestamp alone without additional indexing or scanning.


Hashing Does Not Eliminate Hot Spots Completely

Hashing spreads different keys well. It does nothing when one single key is itself extremely hot. If millions of operations target one user ID or one content ID, the hash of that key still points to one partition.

This is the celebrity problem: when one item becomes globally popular, partition balance by key is no longer enough.

Application-level write sharding for hot keys

A common mitigation is to artificially split one hot key into many derived keys by appending or prepending randomness, such as hotKey-00 to hotKey-99. That spreads writes across many partitions.

But the read path gets worse: reads must now fan out to all subkeys and merge results. You also need metadata telling the application which keys are hot and therefore sharded specially.

Deep tradeoff: relieving a hot spot often means deliberately making reads or writes more complex. There is rarely a free fix.


Secondary Indexes Make Partitioning Much Harder

So far partitioning has been relatively straightforward because the primary key tells you where a record lives. Secondary indexes break that simplicity.

A secondary index answers questions like:

The problem is that the secondary-index value usually does not align neatly with the primary-key partitioning. There are two broad ways to handle this: document-partitioned and term-partitioned indexes.


Document-Partitioned Secondary Indexes (Local Indexes)

In a document-partitioned index, each partition stores the secondary indexes only for the documents physically stored in that partition. In other words, the index is local to the primary-key partition.

Why local indexes are attractive

The cost: scatter/gather on reads

If red cars are distributed across many primary-key partitions, then a query for color = red must ask all partitions, collect their partial results, and merge them.

This is called scatter/gather. It is simple conceptually, but costly in practice:

This is why local-index designs are often good for write-heavy workloads but can become painful for rich search patterns.


Term-Partitioned Secondary Indexes (Global Indexes)

Instead of storing an index locally next to each document, you can build a global index that covers all documents across the database. Then partition that index by the indexed term itself, such as color:red or make:honda.

Why global indexes help reads

A query for a term can go directly to the partition that owns that term in the index, instead of broadcasting to all document partitions. This can make reads dramatically more efficient.

Why global indexes hurt writes

A write to one document may affect many index terms, and those terms may live on many different partitions and nodes. So one logical write can become several distributed updates.

Property Local index Global index
Write complexity Lower Higher
Read complexity Higher due to scatter/gather Lower for term lookups
Need cross-partition coordination Less More
Likelihood of asynchronous propagation Lower Higher

Why global indexes are often asynchronous

To update a global secondary index synchronously with every write, the system may need a distributed transaction across multiple index partitions. Many databases avoid that cost and complexity by updating global indexes asynchronously.

That means a freshly written document may not immediately appear in index results. The index becomes eventually consistent with the base data.

This is a major interview insight: query flexibility often comes with write complexity and delayed index consistency.


Rebalancing: Partitioning Does Not End on Day One

Once you partition data, the job is not finished. Over time:

The system therefore has to move partitions or parts of partitions around. This process is called rebalancing.

What good rebalancing should achieve

  1. Load should end up fairly distributed.
  2. The system should keep accepting reads and writes during the move.
  3. Only the minimum necessary data should move.

That last point is not just optimization. Moving data is expensive: it consumes network bandwidth, disk I/O, and operational attention. Bad rebalancing can make the cluster unstable while it is trying to improve it.


How Not to Rebalance: Hash Mod N

The seemingly obvious scheme is: compute hash(key) % N, where N is the number of nodes. That directly maps keys to nodes.

It looks elegant until the cluster size changes. If you add or remove one node, the modulo changes for a huge fraction of keys, so most of the dataset has to move.

Why mod N is bad: the key-to-node mapping depends directly on the current node count, so a tiny topology change causes a massive data reshuffle.

This is one of the classic early mistakes in distributed-system design.


Rebalancing Strategy 1: Fixed Number of Partitions

A common solution is to create many more partitions than nodes from the start and assign several partitions to each node. Then, when a node is added, it "steals" some full partitions from other nodes.

Why this works better

Advantages

Drawbacks

The number of partitions has to be chosen up front. Too few partitions and future growth becomes constrained. Too many partitions and metadata/management overhead grows.

Also, partition size becomes proportional to total dataset size. If the dataset grows a lot, partitions get huge, making recovery and rebalancing more expensive.

This is a strong engineering lesson: fixed upfront decisions often shift complexity into capacity planning.


Rebalancing Strategy 2: Dynamic Partitioning

Instead of fixing partitions forever, some systems split partitions when they get too big and merge them when they get too small. This is especially natural for key-range partitioning.

Why dynamic partitioning is attractive

The catch: empty clusters start poorly

If the system starts with one partition and no prior knowledge, then at the beginning all writes land on one node until the first splits happen. That can make a new cluster underutilize its available hardware.

Some systems address this with pre-splitting: create an initial set of partitions before data arrives. But pre-splitting requires you to know something about future key distribution, which is often hard.

Deep intuition

Dynamic partitioning trades some operational simplicity for better long-run adaptability. It works best when partition size needs to stay within a target band rather than being fixed forever.


Rebalancing Strategy 3: Partitions Proportional to Nodes

A third approach is to keep a fixed number of partitions per node. When a new node joins, it splits some existing partitions and takes ownership of half of each. Cassandra's vnode design is the best-known example.

This keeps partition size relatively stable as the cluster grows, because more nodes mean more partitions overall.

Why this is interesting

The tradeoff

More partitions mean more metadata and more movement bookkeeping. Random boundary choices can also create unfair splits if not handled carefully, which is why implementations evolve over time.


Automatic vs Manual Rebalancing

Even after choosing a rebalancing algorithm, there is a separate operational question: should the system rebalance automatically, or should a human confirm the change?

Why full automation is tempting

Why full automation is dangerous

Rebalancing is expensive. If the system misinterprets a transient slowdown as a node failure and starts moving lots of data, it adds more load to an already stressed cluster. That can trigger a cascading failure.

This is exactly the kind of failure mode that separates "works in theory" from "operates safely in production."

Operational lesson: some of the most dangerous distributed-system bugs come from automation that is individually reasonable but globally destabilizing.

A human in the loop slows things down, but may prevent the cluster from making a bad situation worse.


Request Routing: Who Knows Where the Data Is?

Once data is partitioned, clients need a way to reach the correct node. That sounds simple, but partition ownership changes during rebalancing, failover, or membership changes. So the real problem is not static lookup, but keeping routing knowledge current.

Three broad routing models

Model How it works Main tradeoff
Any-node forwarding Client talks to any node; that node forwards if needed Simpler clients, smarter nodes
Routing tier Client talks to a partition-aware proxy/load balancer Centralized routing logic, extra hop
Partition-aware client Client knows partition-to-node map and talks directly Fast direct access, more complex clients

None of these removes the underlying problem. Somebody still needs authoritative cluster metadata.

Why cluster metadata is hard

All participants must agree enough on which node owns which partition. If different components believe different mappings, requests are sent to the wrong place. That turns routing into a distributed-consensus-flavored metadata problem.

Coordination service approach

Many systems use a separate coordination service such as ZooKeeper. Nodes register themselves there, and the service maintains the authoritative mapping from partitions to nodes. Routers or clients subscribe to updates.

This externalizes coordination and keeps metadata management separate from the storage engine, at the cost of an additional dependency.

Gossip approach

Other systems, such as Cassandra and Riak, propagate cluster-state information among the database nodes themselves via gossip. Then a client can send a request to any node, and that node forwards appropriately.

This removes the external coordinator but increases the complexity inside the database nodes.

This is a recurring design tradeoff in distributed systems: centralize coordination in a dedicated system, or distribute the coordination complexity across the data nodes themselves.


Parallel Query Execution

Most of the chapter focuses on point lookups, writes, and simple scatter/gather patterns typical of many NoSQL systems. But for analytics and data warehouses, partitioning enables something deeper: massively parallel query execution.

A complex SQL query with joins, filters, grouping, and aggregation can be decomposed into stages, and many of those stages can run on multiple nodes in parallel. This is one of the main reasons partitioning is so powerful in analytical systems.

The key idea is that partitioning is not only about storage placement. It is also about creating opportunities for execution parallelism.


Comparing the Main Partitioning Choices

Decision Main upside Main downside Best fit
Key-range partitioning Efficient range scans and locality Hot spots under monotonic access Ordered/time-series/entity-local scans
Hash partitioning Evener load distribution Loses range-query locality Primary-key point lookups
Local secondary indexes Fast/simple writes Scatter/gather reads Write-heavy systems
Global secondary indexes Efficient indexed reads More complex/slower writes Read-heavy search/filter workloads
Fixed partitions Simple operational model Capacity planning burden Predictable growth and stable workloads
Dynamic partitions Adapts to data volume Split/merge operational complexity Variable growth and range-partitioned data

How to Reason About Partitioning in Interviews

When an interviewer says "how would you shard this?" a strong answer should go beyond naming a shard key. You should talk through these questions explicitly:

  1. What are the dominant queries: point lookups, range scans, or secondary-index searches?
  2. What key pattern would create even write and read distribution?
  3. Could monotonic keys create hot spots?
  4. Will there be celebrity keys or tenant skew?
  5. Do secondary indexes need local or global semantics?
  6. How will partitions rebalance as nodes are added?
  7. Who owns routing metadata?

Interview-quality answer: "I would choose the partition key based on the highest-volume access path, but I would explicitly check whether that key is monotonic or skew-prone. If I need per-user time-ordered reads, I would use a composite key like (user_id, timestamp) so I get distribution by user and order within user. I would also be careful with global secondary indexes because they improve read efficiency at the cost of write complexity and possible asynchronous propagation."


What This Chapter Is Really Teaching

On the surface, Chapter 6 is about sharding strategies. At a deeper level, it is about controlling where work happens.

In short, partitioning is where logical data modeling and physical cluster behavior collide. A good shard key is not just a data-model choice. It is a load-distribution decision, an indexing decision, a future-operations decision, and often a product-behavior decision too.

Retrieval Quiz

1. What problem does partitioning solve that replication alone does not?

Splitting data volume and throughput across machines Encrypting data across regions Making all replicas strongly consistent

2. What is skew?

Uneven distribution of data or load across partitions A sorted order of keys within a partition Different replicas holding different values

3. What is the main strength of key-range partitioning?

Efficient ordered and range queries Perfect immunity to hot spots It is only useful for exact key lookups

4. Why can key-range partitioning create hot spots?

Monotonic or clustered keys direct many writes to one range Because range partitioning only supports two partitions Because followers cannot replicate ordered keys

5. What is the main cost of hash partitioning?

It loses efficient key-order range scans It makes point lookups impossible It always creates celebrity hot spots

6. What is the key difference between local and global secondary indexes?

Local indexes favor writes; global indexes favor indexed reads Local indexes only work in SQL databases Global indexes cannot be replicated

7. Why is hash(key) % N a bad rebalancing strategy?

A small node-count change remaps most keys Because modulo is computationally expensive Because hashes become nondeterministic

8. What is the key advantage of dynamic partitioning?

Partition count adapts to dataset size It removes the need for routing metadata It guarantees no hot spots under any key pattern

9. Why can automatic rebalancing be dangerous?

It can destabilize an already stressed cluster Because routing cannot be automated at all Because automatic systems cannot use replication

10. What is the core routing problem in a partitioned cluster?

Knowing which node currently owns which partition Choosing whether to use IPv4 or IPv6 Making every client connect through DNS round-robin only

Notes

Partitioning why :-

Replication = more copies ; Partitioning = split dataset itself

Main goal :- spread data volume + query load across many machines

Each record logically belongs to one partition

Core success condition :-

Need even distribution

Skew = uneven data/load distribution

Hot spot = one partition gets disproportionate traffic

Random placement alone × bad because lookup would need all nodes

Key-range partitioning :-

Each partition owns continuous key interval

Big advantage = ordered storage + efficient range scan

Boundaries should match data distribution, not just key aesthetics

Main danger = monotonic keys (like timestamp) create hot latest partition

Composite-key trick :-

Put distributing field first ; ordering field second

Example :- (sensor_id, timestamp) or (user_id, timestamp)

Load spreads across entities ; order preserved within one entity

Hash partitioning :-

Hash key first ; partition by hash range

Good for even distribution

Bad for range scans because natural key order destroyed

Say hash partitioning unless you really mean original consistent hashing technique

Cassandra-style hybrid :-

Hash first component of compound key ; sort by remaining components inside partition

Example :- (user_id, update_timestamp)

Good for one-to-many entity timeline pattern

Hot key problem :-

Hashing many different keys helps ; one single celebrity key still hot

Mitigation :- split one hot key into many synthetic subkeys using random suffix/prefix

Cost :- reads must fan out + merge ; need bookkeeping for which keys are split

Secondary indexes difficulty :-

Primary key gives partition directly ; secondary index usually does not

Need either document-partitioned or term-partitioned strategy

Local / document-partitioned index :-

Each partition stores secondary index only for its own documents

Writes simple because update stays local

Reads expensive because query may need scatter/gather across all partitions

Tail latency gets amplified by slowest participant

Global / term-partitioned index :-

Index partitioned by indexed term itself, not by primary document location

Reads efficient because go to term owner directly

Writes more complex because one document update may touch many index partitions

Often updated asynchronously index may lag base data

Rebalancing goal :-

When nodes added/removed or fail, need move data/load

Good rebalancing :- fair final load ; continue reads/writes ; move minimum data

Bad idea :- hash(key) mod N

If node count changes, mapping of huge fraction of keys changes

So tiny cluster-size change causes massive data movement

Fixed partitions :-

Create many partitions upfront ; assign many per node

New node steals whole partitions from old nodes

Simple operationally ; but partition count must be chosen upfront

Too many partitions = overhead ; too few = growth pain

Dynamic partitions :-

Split large partitions ; merge small partitions

Adapts to data volume

But empty DB may start with one partition only ; all early writes hit one node

Pre-splitting helps if future key distribution known

Partitions proportional to nodes :-

Fixed number of partitions per node (vnode style)

When node joins, split some existing partitions and take halves

Keeps partition size more stable as cluster grows

Auto v/s manual rebalancing :-

Full automation convenient but risky

Bad failure detector + auto rebalance can worsen overload and cause cascading failure

Human in loop slower but safer in many prod systems

Routing models :-

1) send to any node ; it forwards if needed

2) send to routing tier / partition-aware load balancer

3) client itself knows partition-to-node map

Main hard part = keep partition ownership metadata correct as cluster changes

Coordination choices :-

Use external coordinator like ZooKeeper / Helix, or spread metadata via gossip

External coordinator = simpler data nodes but extra dependency

Gossip = no external coordinator but more logic inside nodes

Primary source: Kleppmann, M. (2017). Designing Data-Intensive Applications, Chapter 6: “Partitioning.” O'Reilly Media.
Recommended supplements: Cassandra data modeling docs, HBase region-splitting docs, Elasticsearch sharding/indexing guides, and practical material on ZooKeeper/Helix-style metadata coordination.

Ask follow-up questions if you want to drill into shard-key design, celebrity hot keys, local vs global indexes, or how to explain rebalancing tradeoffs in a system design interview.

← Lesson 5: Replication Lesson 7: Transactions →