Replication sounds simple: keep the same data on multiple machines. In reality, it is where distributed systems stop being "just software" and become a fight against network delay, partial failure, stale reads, write conflicts, and ambiguity. Chapter 5 is one of the most interview-relevant chapters in DDIA because it gives you the mental models behind questions like: why does my read return stale data?, why can failover lose acknowledged writes?, why is multi-region active/active hard?, and why does quorum not automatically mean strong consistency?
Mission tie-in: if you want to design production systems for FAANG-style interviews or build your own distributed systems, you need to understand replication not as a feature checkbox, but as a set of tradeoffs among latency, availability, durability, read scale, and correctness under failure.
Replication means storing copies of the same data on multiple networked machines. Systems replicate data for three primary reasons:
This chapter assumes the dataset is small enough that each replica can hold the full dataset. That matters: the hard problem here is not partitioning large datasets yet, but rather keeping multiple copies of changing data in sync. Partitioning comes next chapter.
If data never changed, replication would be trivial. Copy it once and you are done. The entire challenge comes from replicating writes: ordering them, delivering them, recovering missed ones, and deciding what to do when multiple writes happen at once.
| Approach | Who accepts writes? | Main strength | Main pain |
|---|---|---|---|
| Single-leader | One leader per dataset or partition | Simple mental model, no normal write conflicts | Leader bottleneck, failover pain, stale follower reads |
| Multi-leader | Several leaders | Better write locality and outage tolerance | Conflict resolution and causality problems |
| Leaderless | Any replica or coordinator | High availability and flexible quorum tuning | Weak guarantees, conflict handling, hard staleness reasoning |
Nearly every distributed database lands in one of these buckets. Everything else in the chapter is essentially about understanding when each model breaks, and what guarantees you do or do not get from it.
In leader-based replication, one replica is designated the leader. All writes go to the leader first. The leader commits the write locally, then sends a log of changes to the other replicas, called followers. Followers apply those writes in the same order and become read-only copies.
The important idea is not just "one primary, many replicas." The deeper idea is: one place decides write order. That is what avoids the kinds of conflicts that make multi-leader and leaderless systems harder.
Single-leader systems feel simple as long as the leader is healthy and replication lag is low. The moment the leader fails, a follower is stale, or the system is stretched across regions, the simplicity starts to crack. That is why the next few sections matter.
A critical design choice is whether the leader waits for followers before acknowledging a write.
| Mode | Leader waits? | Strength | Risk |
|---|---|---|---|
| Synchronous | Yes | Follower is guaranteed up to date when write succeeds | Follower slowness/failure blocks writes |
| Asynchronous | No | Leader keeps accepting writes despite lagging followers | Acknowledged writes can be lost on leader failure |
That last point is one of the chapter's most important truths: in a fully asynchronous system, a client may be told "write successful" even though the write exists on only one node. If that leader dies before followers catch up, the write is gone.
Because then any slow or failed replica becomes a write outage for the whole system. That is too fragile in practice. Many deployments use a semi-synchronous approach: one follower must acknowledge, while others replicate asynchronously.
Core tradeoff: synchronous replication strengthens durability but hurts availability and latency. Asynchronous replication strengthens write availability and performance but weakens durability during failover.
If someone asks, "Why do acknowledged writes still disappear after failover?" the correct explanation is not "buggy replica." It is: because the system acknowledged before enough replicas durably stored the write.
Adding a follower sounds easy until you realize the leader is being written to continuously. A naive file copy can capture one table before a write and another table after that write, resulting in an inconsistent snapshot.
The usual process is:
The phrase "consistent snapshot" matters a lot operationally. It means the new replica starts from a transactionally meaningful point, not from a random copy of bytes observed mid-change.
This same snapshot-plus-log idea is also why backups and point-in-time recovery work in many systems.
Replication exists partly so you can survive node outages. But surviving which node failed matters.
Followers keep a local record of the replication log they have already applied. If a follower crashes or loses its network connection temporarily, it can reconnect and ask the leader for everything since its last known log position. This is catch-up recovery.
As long as the leader still has the necessary log history, this is straightforward.
If the leader dies, the system needs failover:
| Problem | Why it is bad |
|---|---|
| Unreplicated writes on old leader | Recently acknowledged data may be discarded during promotion of a stale follower. |
| Split brain | Two leaders accept writes concurrently and corrupt state unless fenced or resolved. |
| Wrong timeout | Too long means slow recovery; too short causes unnecessary failovers under transient slowness. |
| External side effects | Database failover can desynchronize external systems such as caches or ID generators. |
The GitHub incident in the chapter is a strong real-world example: promoting a lagging MySQL follower caused reused primary keys, which in turn broke consistency with Redis and exposed private data incorrectly. That is a classic distributed-systems lesson: replication mistakes escape the database layer.
Deep point: failover is not merely a routing change. It is a correctness event. If the promoted node is behind, you are changing history.
Under the hood, leader-based replication means "stream changes to followers." But what is the format of those changes? There are several answers, and each has different operational consequences.
The leader logs each write statement, such as SQL
INSERT, UPDATE, or DELETE, and followers execute the
same statements.
This is conceptually simple but brittle. It breaks when statements are nondeterministic or depend on execution context.
NOW() or RAND() may evaluate differently on each replica.Statement-based replication is compact, but modern systems generally avoid relying on it because there are too many edge cases.
Instead of replicating SQL, the leader replicates its low-level storage log: the physical bytes written to the storage engine's append-only log or write-ahead log.
This is powerful because followers reconstruct the exact same on-disk structures. It is also restrictive because the replication format is tightly coupled to storage-engine internals.
| Benefit | Drawback |
|---|---|
| Exact replica of storage state | Tight coupling to database version and storage engine format |
| Efficient and mature | May prevent zero-downtime upgrades across database versions |
| Simple for the database to apply | Hard for external systems to consume as a logical change stream |
That upgrade point is subtle but very practical. If followers must run the exact same storage format as the leader, you cannot easily upgrade replicas first, fail over, and then upgrade the former leader. WAL shipping can therefore turn software upgrades into downtime events.
Here, the replication stream describes changes at the logical row level instead of the physical byte/block level.
This is a huge conceptual improvement because the replication stream is now decoupled from on-disk representation. That makes version skew easier to handle, and it also makes the stream useful to outside consumers such as caches, search indexes, and data warehouses.
This is one of the roots of change data capture (CDC): treat the database's logical replication stream as a feed of business events.
If built-in replication is too rigid, you can move replication logic into the application/database layer using triggers and stored procedures. A trigger records writes into some side table, and another process ships them elsewhere.
This gives flexibility, but it is usually slower, more fragile, and more bug-prone than native replication. You use it when you need custom behavior, subset replication, cross-database replication, or app-specific conflict handling.
Leader-based replication is attractive because it scales reads well: one leader handles writes, many followers answer reads. But if followers replicate asynchronously, reads can become stale.
This temporary inconsistency is called eventual consistency: if writes stop and enough time passes, replicas converge. The problem is that "enough time" is not bounded tightly in general. Under healthy conditions lag may be milliseconds; under overload or network trouble it may become seconds or minutes.
The rest of this section is extremely important because it turns vague eventual consistency into concrete user-visible anomalies.
Scenario: a user updates their profile, then refreshes the page and sees the old value because the read hit a stale follower. To the user, it looks like the write was lost.
Read-after-write consistency or read-your-writes consistency guarantees that a user will always see their own writes. It does not guarantee they see everyone else's latest writes.
The timestamp-based method is conceptually strong, but once users operate across devices, metadata must be shared centrally and requests may need careful routing.
Scenario: a user refreshes twice. The first request hits a nearly fresh follower and shows a new comment. The second request hits a more lagging follower and the comment disappears. Time appears to move backward.
Monotonic reads guarantee that once a user has seen some version of the data, later reads by that same user will not show an older version.
A common way to approximate this is to route one user's reads consistently to the same replica, perhaps via hashing on user ID. If that replica fails, the guarantee becomes harder to preserve.
Scenario: one write causally depends on another, like an answer depending on a question. Because different partitions or replicas lag differently, an observer sees the answer before the question.
Consistent prefix reads guarantee that writes are seen in an order that respects causality: if write B depends on write A, observers should not see B without A first.
This gets especially tricky in partitioned systems because different shards may progress independently. There may be no single global order of writes.
What eventual consistency really means operationally: not just "maybe stale," but possibly broken user expectations around self-visibility, time ordering, and causality.
Eventual consistency is often marketed as if it were one clean behavior. It is not. It is a family of possible anomalies. Good system design means naming the anomaly you cannot tolerate and then adding the minimum mechanism needed to prevent it.
Single-leader has one big downside: only one node can accept writes for a given dataset or partition. Multi-leader replication relaxes that by allowing several nodes to accept writes. Each leader then replicates its changes to the others.
This sounds like a natural upgrade to single-leader. In practice it is dangerous territory because the moment multiple places can accept writes, you need a story for concurrent updates.
If every write in Europe must cross the ocean to a US leader, latency becomes bad and network interruptions can block writes. Multi-leader allows each datacenter to accept local writes, then asynchronously exchange them with other datacenters.
| Dimension | Single-leader across DCs | Multi-leader across DCs |
|---|---|---|
| Write latency | Often remote and expensive | Usually local to the writer |
| Datacenter outage tolerance | Requires failover | Other DCs can keep accepting writes |
| Inter-DC network issues | Can block writes | Usually tolerated better via async replication |
| Conflict complexity | Low | High |
Mobile devices, laptops, or calendars that must accept writes while disconnected are basically mini leaders. When they reconnect, they sync asynchronously with the server and other devices. That is multi-leader replication in disguise.
If many users can edit a shared document at once without locking, you again have multiple places generating concurrent updates. The problem is the same, even if the product does not call it a database.
Many databases bolted it on later rather than designing around it from the beginning. As a result, there are often nasty interactions with autoincrement IDs, triggers, uniqueness constraints, and ordering assumptions. If you can use single-leader safely, it is usually easier.
Suppose two users edit the same field concurrently in different datacenters. Each local leader accepts the write. Only later, during replication, does the system discover that the writes conflict.
This is the core difference from single-leader systems: in single-leader, the second write would wait or fail. In multi-leader, both writes can appear successful.
In principle, you could wait for all leaders to coordinate before confirming a write. But then each leader is no longer independently accepting writes. You have effectively backed your way toward a single-leader or consensus-style design.
The safest strategy is often to ensure that all writes for the same record always go through the same leader. For example, all writes for one user's data might always route to that user's home region.
This is very practical and often recommended. But it breaks down when traffic is rerouted after outages, when users move regions, or when true shared editing is needed.
Replicas must eventually agree on one state. If each replica simply applies writes in arrival order, different replicas may keep different final values forever. Conflict resolution therefore must be convergent: every replica reaches the same result after all writes are delivered.
The first two are operationally simple but imply silent data loss. The latter two preserve information but require more logic.
| When resolved | How it works | Tradeoff |
|---|---|---|
| On write | Background conflict handler runs as soon as conflict is detected | Fast and automatic, but cannot ask the user and is easy to get wrong |
| On read | Store all versions and let application/user resolve later | Preserves information, but pushes complexity to reads and app logic |
CouchDB-style conflict resolution on read is a good example of the second approach. It exposes conflicts instead of pretending they did not happen.
CRDTs, mergeable persistent data structures, and operational transformation are attempts to turn "conflict handling" from ad hoc app code into principled data structures/algorithms. This matters a lot for collaborative editing and shared mutable structures.
But the chapter's practical stance is still sober: most production systems implement conflict handling poorly, and app developers need to understand the danger.
Not all conflicts look like "same row, same field, different value." Booking the same room for overlapping time ranges is also a conflict, even if the rows are distinct. This is why transactions and stronger concurrency control matter later in the book.
With more than two leaders, topology matters. Who forwards writes to whom?
| Topology | Shape | Strength | Weakness |
|---|---|---|---|
| All-to-all | Every leader talks to every other leader | Good fault tolerance, alternate paths | Ordering/crossing-message issues |
| Circular | Each node forwards to one next node | Simple wiring | One failed node can break propagation path |
| Star/tree | Root forwards to others | Operationally simple | Root/path failures become bottlenecks or SPOFs |
Dense connectivity improves fault tolerance, but introduces a causality problem: some links are faster than others, so updates can overtake each other.
Example: an INSERT on one leader and an UPDATE on another may arrive
in the wrong order at a third. The update depends on the insert, but the third node may see
the update first. Timestamps do not reliably fix this because clocks are not trustworthy
enough to define causal order.
This is why version vectors and causal tracking matter later in the chapter.
Leaderless systems abandon the idea that one node decides write order. Clients send writes to multiple replicas directly, or to a coordinator that forwards them, but the coordinator is not a leader in the single-leader sense.
Dynamo, Riak, Cassandra, and Voldemort are the classic family here.
But giving up a central write-ordering authority means consistency becomes statistical and configuration-dependent, not obvious.
Imagine three replicas store a value. One node is down. In a leaderless system, the client can still send the write to all replicas, get acknowledgments from the two healthy ones, and consider the write successful.
When the down node returns, it is stale. Reads therefore typically query multiple replicas and compare versions.
If a read detects that one replica returned an older version, the reader or coordinator writes the newer value back to that stale replica. This works well for hot data that is read frequently.
A background job compares replicas and copies missing data around over time. Unlike a leader's replication log, this process may not preserve original write order and may take a long time.
Without anti-entropy, rarely read data may stay missing from some replicas for a long time, which quietly weakens durability.
Leaderless systems often use three parameters:
The classic rule is:
Quorum condition: if w + r > n, the read set and write set
should overlap in at least one replica, so a read is expected to encounter a replica with
the latest successful write.
| Example | Meaning |
|---|---|
n = 3, w = 2, r = 2 |
Can tolerate one unavailable node and still maintain overlap. |
n = 5, w = 3, r = 3 |
Can tolerate two unavailable nodes with majority-style overlap. |
w = n, r = 1 |
Fast reads, expensive/fragile writes. |
w = 1, r = n |
Cheap writes, expensive reads. |
The beauty of quorums is tunability. The danger of quorums is that people often mistake the overlap argument for a blanket guarantee of strong consistency.
Even if w + r > n, stale reads can still happen in real systems.
| Edge case | Why quorum logic breaks down |
|---|---|
| Sloppy quorum | Writes may land on temporary substitute nodes, so read and write sets may not overlap on the intended replicas. |
| Concurrent writes | There may be no clear "latest" value without conflict resolution. |
| Read racing a write | Some replicas have the new value and others do not. |
| Partial write failure | A write reported as failed may still exist on some replicas. |
| Replica rollback/rebuild | A node with fresh data may fail and be restored from a stale one. |
| Timing anomalies | Implementation details and request timing can still violate the simplified overlap intuition. |
This is one of the chapter's most useful corrections to industry folklore: quorums reduce the probability of stale reads; they do not automatically provide the user-facing guarantees you may assume.
In particular, leaderless quorum systems usually do not give you read-your-writes, monotonic reads, or consistent prefix reads unless additional mechanisms exist.
In leader-based systems, lag is easier to measure because everyone advances through one ordered log. In leaderless systems there is no single log position to subtract, so staleness becomes much harder to quantify.
That means "eventual" often stays vague in operations unless the system provides explicit staleness metrics or you build them yourself.
Suppose a client cannot reach the exact n home replicas for some key due to a
network partition, but it can reach other nodes in the cluster. Should the database
reject the write, or should it store the write somewhere else temporarily?
A sloppy quorum says: accept the write on any reachable nodes, even if they are not the key's designated home replicas.
Later, once the network recovers, those temporary holders transfer the data to the correct home replicas. That transfer is called hinted handoff.
w nodes somewhere.
If the latest value is sitting on substitute nodes instead of the normal home replicas, then
even with w + r > n, a read of the home replicas may miss it. So a sloppy
quorum is not a strict quorum in the traditional overlap sense.
Precise takeaway: sloppy quorums mainly protect availability and durability, not freshness. They weaken the usual read-latest intuition behind quorums.
In leaderless systems, even strict quorums do not prevent conflicts. Two clients can write to the same key at nearly the same time, and different replicas may observe those writes in different orders.
If each replica simply overwrites whatever value it currently has, the replicas can diverge permanently. So the database needs a way to distinguish:
The simplest strategy is to assign timestamps and say the highest timestamp wins. This forces convergence, but at the cost of potentially dropping successful writes silently.
LWW is attractive because it is easy. It is dangerous because it throws away information. It can even lose non-concurrent writes when clocks skew or timestamps are misleading.
The only truly safe use case for LWW is when each key is written once or treated as effectively immutable afterward.
DDIA makes a crucial conceptual move here: concurrency is not primarily about overlapping wall-clock time. It is about causal knowledge.
Operation A happens before operation B if B knows about A, depends on A, or builds upon A. Two operations are concurrent if neither happens before the other.
This is one of the deepest ideas in distributed systems. It appears again later in vector clocks, causal consistency, transactions, and consensus.
Even if two writes are seconds apart on real clocks, they may still be concurrent if the network prevented one writer from learning about the other. Conversely, two writes may be near-simultaneous physically but causally ordered if one explicitly depends on the result of the other.
So the system needs to track dependency information, not just timestamps.
The shopping-cart example in the chapter is extremely important because it shows how a system can preserve all information instead of dropping conflicting writes.
At a high level, the algorithm is:
This way, a newer write overwrites the states it actually built upon, but does not erase concurrent states it never saw.
Concurrent versions kept side by side are often called siblings. The database is saying: "I cannot safely decide which of these is the real winner, so I will preserve them both and make you merge them."
For shopping carts that only ever add items, union is reasonable. But once deletions are allowed, naive union makes removed items reappear. That is why deletion usually needs an explicit marker called a tombstone.
This is another very important distributed-systems pattern: deleting data is often not literally "remove the bytes now." It is "record that a deletion happened, so merges and late replicas can interpret history correctly."
A single version number per key is enough when one replica serializes writes. It is not enough when multiple replicas accept writes concurrently.
The fix is a version vector: each replica maintains its own version counter per key and also tracks the versions seen from other replicas. The vector captures causal history across replicas.
When clients read, they receive this causal metadata. When they write, they send it back. The database then knows which versions are ancestors, which are concurrent, and which can be safely overwritten.
Riak calls this metadata causal context. The idea is closely related to vector clocks, though DDIA notes the terminology is sometimes used imprecisely.
| Question | Single-leader | Multi-leader | Leaderless |
|---|---|---|---|
| Where do writes go? | One leader | Any leader | Several replicas |
| Who defines order? | Leader | No single global order | No leader-defined order |
| Typical read freshness | Fresh on leader, stale on followers | Potentially stale/conflicted | Depends on quorum and timing |
| Main failure pain | Leader failover | Conflict resolution | Staleness/conflicts/quorum edge cases |
| Best use cases | General databases, read scaling | Multi-region writable systems, offline sync, collaboration | High availability with eventual consistency tolerance |
| Main mental trap | Underestimating lag and failover loss | Underestimating conflict semantics | Assuming quorum means strong consistency |
If an interviewer asks you to replicate a datastore or make a service multi-region, a strong answer usually sounds like this:
Interview-quality answer: "I would start with single-leader replication because it simplifies write ordering and conflict avoidance. I would use followers for read scaling, but I would explicitly handle replication lag by routing read-after-write traffic to the leader or to replicas known to be caught up. If I needed multi-region active/active writes, I would only do that for data models where conflict resolution semantics are well-defined."
The surface topic is replication. The deeper topic is distributed uncertainty.
Once you understand this chapter well, you stop asking "Which database is best?" in the abstract, and start asking better questions: Who orders writes? What happens on failover? How stale can reads become? What does a successful write actually mean? How are conflicts detected and resolved?
1. What is the core benefit of single-leader replication?
One node defines write order It guarantees no stale reads anywhere It removes the need for replication logs2. What is the main risk of fully asynchronous replication?
Acknowledged writes may be lost after leader failure Writes fail whenever any follower is slow Followers cannot be added later3. Why is failover a correctness event, not just a routing event?
Promoting a stale replica can change which writes survive Because clients always reconnect slowly Because replication logs become unreadable4. What anomaly does read-your-writes consistency prevent?
A user not seeing their own recent update Seeing a reply before the question Two leaders both accepting writes5. What do monotonic reads guarantee?
A user's reads do not move backward in time Every read returns the globally latest value Followers replicate synchronously6. Why is multi-leader replication hard?
Concurrent writes create conflicts and ordering problems Because all clocks must be perfectly synchronized Because writes cannot happen during network interruptions
7. What does the quorum condition
w + r > n try to ensure?
8. Why can stale reads still happen even when
w + r > n?
9. What does "happens-before" mean?
One operation causally depends on another One operation had a smaller wall-clock timestamp One operation used a faster network route10. Why are version vectors useful?
They capture causal metadata across replicas They mainly reduce network bandwidth They make all replicas identical instantly1) low latency by placing data near user
2) higher availability under node failure
3) higher read throughput via replicas
Hard part = not copying static data ; hard part = replicating changes
Single-leader ⇒ one node accepts writes ; followers replay log
Multi-leader ⇒ many nodes accept writes ; then replicate to each other
Leaderless ⇒ client/coordinator writes to several replicas directly
Main advantage = one place defines write order
That avoids normal write conflicts
Reads can go to followers ; writes only to leader
Synchronous :- leader waits for follower ack → stronger durability ; weaker availability
Asynchronous :- leader does not wait → better write availability ; acknowledged writes can be lost if leader dies early
Semi-sync :- often one follower sync, rest async
Need consistent snapshot + exact log position
Then replay backlog till follower catches up
Naive file copy × unsafe because DB changing continuously
Follower failure :- easy ; reconnect + request missed log entries
Leader failure :- hard ; detect death, choose new leader, reroute clients, demote old leader
Failover risks :- stale promotion, split brain, bad timeout choice, external system inconsistency
Statement-based :- replay SQL ; breaks with nondeterminism / side effects
WAL shipping :- ship physical storage log ; efficient but tightly coupled to storage format/version
Logical row log :- send row-level inserts/updates/deletes ; easier upgrades + CDC
Trigger-based :- flexible custom replication ; more overhead + bugs
Read-your-writes :- user must see own latest write
Monotonic reads :- user's later read should not go backward in time
Consistent prefix :- causally ordered writes must be seen in same order
Eventual consistency = these anomalies possible unless extra mechanism added
1) multiple datacenters with local writes
2) offline-capable clients / device sync
3) collaborative editing
Main downside = concurrent writes create conflicts
Can avoid conflicts by routing one record to one leader if possible
Else need convergent resolution
Options :- LWW, replica priority, merge, preserve all conflicting versions
LWW simple but dangerous because it silently drops data
No failover in normal single-leader sense
Write to several replicas ; read from several replicas ; compare versions
Repair via read repair + anti-entropy
n = replica count ; w = write acks needed ; r = read responses needed
If w + r > n ⇒ read set overlaps write set
But this is not equal to strong consistency in practice
Edge cases :- sloppy quorum, concurrent writes, read/write race, partial failure, unlucky timing
If home replicas unreachable, write to substitute reachable nodes
Later move data back to home replicas = hinted handoff
Improves availability ; weakens freshness guarantee
Concurrent × not "same wall-clock time"
Concurrent = neither operation knows about / depends on the other
Happens-before = causal dependency
Client reads version + values ; on write sends prior version back
Server overwrites only ancestors ; keeps concurrent values as siblings
Merge siblings carefully ; deletion needs tombstone or removed item may reappear
Single version number enough only for single-replica ordering
Multiple replicas need one version component per replica
Version vector captures causal context across replicas
Lets system distinguish overwrite v/s concurrent write
Primary source: Kleppmann, M. (2017).
Designing Data-Intensive Applications, Chapter 5: “Replication.” O'Reilly
Media.
Recommended supplements: PostgreSQL replication docs, MySQL replication
internals, Kafka replication design, and papers/articles on Dynamo, Cassandra quorum behavior,
CRDTs, and vector clocks.
Ask follow-up questions if you want to drill into failover safety, causal consistency, quorum math, LWW dangers, or how to answer replication tradeoff questions in a system design interview.