Lesson 12: The Future of Data Systems

Lesson 0012 — 45 min read

Chapter 12 is Kleppmann's capstone — his personal synthesis of everything in the book projected forward. Three big themes: (1) how to compose specialized data tools into a coherent whole, (2) how to get correctness guarantees without distributed transactions, and (3) the ethics of building data-intensive systems. The chapter also contains the only sustained first-person argument in the book, so opinions are explicitly marked.

Data Integration

The central problem: no single tool serves all use cases. You need an OLTP database and a search index and a cache and a data warehouse and a stream processor. So you cobble them together. The hard part is keeping them consistent.

The naive approach: dual writes — application code writes to each system. Two fatal flaws (from Chapter 11): race conditions (different order at different systems) and partial failures (one succeeds, one fails). Both cause permanent inconsistency.

The better approach: designate a system of record and derive everything else from it. Derivation uses batch + stream processing to transform the source data into indexes, caches, materialized views, ML models, summaries — whatever downstream consumers need.

Derived Data vs Distributed Transactions

Kleppmann's view: log-based derivation is the most promising integration approach. But he doesn't dismiss read-your-writes guarantees — they matter, and "eventual consistency is inevitable, suck it up" is not an acceptable answer without guidance.

The Limits of Total Ordering

A totally ordered event log (single leader) works for small systems but breaks down under:

1. Throughput exceeds single node — must partition, introducing cross-partition ambiguity.
2. Multi-datacenter — network latency makes synchronous cross-DC coordination inefficient; each DC needs its own leader, breaking total order.
3. Microservices — each service owns its state; events from different services have no defined order.
4. Offline clients — client-side state updates immediately; server sees events in different order.

Total order broadcast is equivalent to consensus (Chapter 9). Scaling consensus beyond single-node throughput in geo-distributed settings remains an open research problem.

Capturing Causality Without Total Order

Example: user unfriends ex-partner, then sends message complaining about them. If the causal dependency (unfriend → message) is lost, a notification service may incorrectly notify the ex. Solutions:

• Logical timestamps (sequence numbers, Lamport clocks) — provide total ordering without coordination, but require handling out-of-order delivery and extra metadata.
• Event reference IDs — log the state the user saw before acting; reference that event ID in subsequent events to record causal dependency.
• Conflict resolution algorithms (CRDTs, merge strategies) — useful for state but don't help with external side effects (sending a notification).

Batch and Stream Processing Together

Batch and stream processing are converging. Spark breaks streams into microbatches on a batch engine; Flink runs batch as a special case of streaming. The distinction is blurring, but batch handles reprocessing historical data, while stream provides low-delay incremental updates.

Maintaining Derived State

Deterministic, pure functions with immutable inputs and append-only outputs — same philosophy from Chapter 10. Asynchrony is what makes log-based systems robust: a fault in one consumer is contained locally, unlike distributed transactions which abort all participants when one fails.

Reprocessing for Application Evolution

Schema migration analogy: railway gauge conversion. 19th-century England had competing railway gauges. The solution: add a third rail (dual gauge), run both old and new trains, then remove the old rail when conversion is complete. Gradual, reversible migration.

Same idea with data: maintain old schema and new schema side by side. Shift users gradually. If the new view has bugs, switch back. Every stage is reversible, reducing risk.

The Lambda Architecture

Nathan Marz's proposal: record immutable events; run batch + stream in parallel. Stream produces approximate updates quickly; batch produces corrected versions later.

Problems Kleppmann identifies:

• Maintaining the same logic in two frameworks (batch + stream) is significant extra effort.
• Merging batch and stream outputs is easy for simple aggregations but hard for joins and sessionization.
• Incrementalizing batch processing adds streaming complexity, defeating the purpose.

Unified systems (Flink, Beam, Cloud Dataflow) supersede the Lambda approach: replay historical events through the same engine, exactly-once semantics, event-time windowing. One engine, not two.

Unbundling Databases

A database internally does: storage + indexing + replication + query optimization + materialized views + triggers + constraints. These are tightly integrated features.

Unbundling = taking those features and implementing them as separate, loosely coupled components connected by streams. The search index, cache, ML model, and analytics warehouse are like different index types on a single logical database.

Federated Databases (Unifying Reads)

A unified query interface over heterogeneous storage engines — PostgreSQL Foreign Data Wrappers, polystores (BigDAWG). Applications access specialized engines directly or combine data through the federated interface. Solves read-side integration only.

Unbundled Databases (Unifying Writes)

The harder problem: keeping writes in sync across systems. The unbundled approach uses CDC + event logs to propagate writes — like Unix pipes for databases. Loose coupling manifests as:

1. System-level: async event streams buffer failures; faulty consumers catch up later. No fault amplification.
2. Human-level: different teams develop and maintain components independently. Event logs provide strong consistency properties (durability, ordering) while being general enough for any data.

What's Missing?

The unbundled database needs its Unix shell — a declarative language for composing storage and processing systems. Kleppmann dreams of: mysql | elasticsearch — a pipe that captures database changes and indexes them automatically, like a declarative CREATE INDEX across heterogeneous systems.

Early research: differential dataflow (Naiad) — incrementally update materialized views as inputs change. Think spreadsheet recalculation for distributed data systems.

Designing Applications Around Dataflow

Application Code as a Derivation Function

Derived datasets need transformation functions. Some are generic (secondary indexes — CREATE INDEX). Others are application-specific (ML feature engineering, cache invalidation logic). The latter require custom code — and this is where databases fall short (triggers, stored procedures are afterthoughts). Stream processors are the better execution environment.

Separation of Code and State

Modern web apps deploy stateless services + state in databases. The database is a mutable shared variable — but you can't subscribe to changes, only poll. The dataflow approach flips this: subscribe to state changes ahead of time, store locally, query locally (no network request).

Stream Processors vs Services

The dataflow approach is faster and more robust: the fastest network request is no network request. Instead of RPC, perform a stream join between purchase events and exchange rate update events.

Observing Derived State

Write Path and Read Path

Every derived dataset has two paths:

• Write path — precomputed work done eagerly as data arrives (update search index, invalidate cache). Write path = eager evaluation.
• Read path — work done lazily when someone queries (search index lookup, cache hit or miss + compute). Read path = lazy evaluation.

The derived dataset is where write and read paths meet. Indexes, caches, and materialized views shift the boundary: more work on write path = less work on read path, and vice versa.

Twitter example from Chapter 1 reprise: celebrities get fan-out-on-write (more write work, fast reads); ordinary users get fan-out-on-read (less write work, slower reads). The boundary is drawn differently per user.

Stateful, Offline-Capable Clients

Extend the write path to end-user devices. Subscribe to state changes (via WebSocket, SSE), maintain a local cache, continue working offline, sync when reconnected. Consumer offset technique (Chapter 11) works for individual devices — each device is a small subscriber to a small stream.

Reads Are Events Too

Treat read requests as event streams. Route read + write events through the same stream processor. A read becomes a stream-table join between the query and the database partition. Logging reads adds causal tracking (what state did the user see before their action?) at the cost of additional storage and I/O.

Multi-Partition Data Processing

Complex queries that span partitions (e.g., union of follower sets for a URL, fraud detection joining reputation databases) can be expressed as streams of read events routed through a stream processor's partitioning and joining infrastructure. Storm's Distributed RPC is an example.

Aiming for Correctness

Stateful systems remember things forever — bugs corrupt data permanently. ACID transactions have been the standard tool for correctness for 40 years, but they're weaker than they seem (weak isolation confusion, Jepsen findings) and expensive at scale.

The End-to-End Argument for Databases

Classic example from Saltzer, Reed, Clark (1984): TCP suppresses duplicate packets, but that doesn't prevent duplicate HTTP POSTs if the response is lost. The application itself must handle end-to-end deduplication.

Same principle applies to databases: serializable transactions don't save you from application bugs. If the app writes incorrect data, no transaction isolation level helps. End-to-end integrity requires application-level mechanisms.

Exactly-Once and Duplicate Suppression

Making an operation naturally idempotent is the cleanest approach:

• Setting a key to a fixed value = idempotent.
• Incrementing a counter = not idempotent.
• Fix: include a unique operation ID (UUID) in every write. Use a uniqueness constraint to reject duplicates.

Example: INSERT INTO requests (request_id, ...) VALUES ('UUID', ...) with UNIQUE (request_id). If a retry inserts the same UUID, the unique constraint aborts the second transaction, preventing double charging.

Uniqueness Constraints Require Consensus

Enforcing uniqueness (usernames, seat bookings) requires consensus: concurrent requests with the same value need a single decision on which wins. Standard approach: single leader node. Can scale by partitioning on the unique value (hash of username).

In log-based systems: partition the log by the unique value (hash of username). Stream processor consumes one partition on a single thread, unambiguously orders requests, and decides first-wins. Same as implementing linearizable storage via total order broadcast (Chapter 9).

Multi-Partition Request Processing

Bank transfer from A to B spans 2+ partitions. Traditional approach: distributed transaction across partitions (expensive). Log-based approach: client generates a request ID, appends to request log (single partition). Stream processor reads the request, emits debit instruction to A's partition and credit instruction to B's partition. Downstream processors deduplicate by request ID.

Key insight: atomicity is achieved by single-object write to the request log — either the request appears or it doesn't. The multi-partition effect is derived deterministically from that single durable write. No distributed transaction needed.

Timeliness vs Integrity

Kleppmann's strong claim: integrity is more important than timeliness. Violations of timeliness are annoying; violations of integrity are catastrophic. Dataflow systems decouple them: async processing provides no timeliness guarantees, but exactly-once semantics, idempotence, and deterministic derivation preserve integrity.

Loosely Interpreted Constraints

Many real-world constraints don't need strict linearizability:

• Overlapping bookings — hotel overbooks expecting cancellations. Compensate the excess with an apology (upgrade, free room elsewhere).
• Double-booking a seat — send apology, ask customer to pick another.
• Overdraft — allow it, charge a fee.
• Stock overselling — order more stock, apologize for delay.

The cost of apology is a business decision. If it's acceptable, linearizable constraints aren't needed — go ahead optimistically and fix up later.

Coordination-Avoiding Data Systems

Two observations combine:

1. Dataflow systems preserve integrity without atomic commit or synchronous coordination.
2. Many applications tolerate loose constraints with post-facto apologies.

Result: coordination-avoiding data systems that operate across datacenters with multi-leader async replication. No synchronous cross-region coordination. Weak timeliness, strong integrity. Can still introduce synchronous coordination where needed (e.g., before an operation with irreversible consequences) — not everything needs to pay the cost.

Trust, But Verify

All our system models make assumptions (fsync works, memory doesn't corrupt, CPU arithmetic is correct). These are probabilities, not absolutes. Random bit-flips are rare but happen (rowhammer attacks, cosmic rays). Software bugs in databases themselves exist.

A Culture of Verification

Kleppmann argues for self-auditing systems: continually read back data, compare replicas, run redundant derivations in parallel. HDFS and Amazon S3 already do this — background processes that check file integrity and move data off failing disks. Most systems don't.

Designing for Auditability

Event sourcing makes auditability natural: the event log is the truth; derived state is deterministic and repeatable. Run the same log through the same code = same state. Merkle trees (used in certificate transparency, cryptocurrencies) can prove integrity cryptographically. Kleppmann sees this as an area to watch.

End-to-end integrity checks include the entire pipeline — disks, networks, services, algorithms — in one check. Continuous verification gives confidence to evolve faster (like automated testing).

Doing the Right Thing

The final section — ethics of data-intensive applications. Kleppmann argues engineers have a responsibility to consider the real-world consequences of their systems.

Predictive Analytics and Bias

Algorithms that predict recidivism, loan defaults, insurance risk directly affect people's lives. Machine learning can amplify existing biases: if input data is discriminatory, the output will be too. Postal code predicts race in segregated neighborhoods; an algorithm using postal code encodes racial bias.

Accountability gap: if a human judge makes a mistake, there's appeal. If an algorithm makes a mistake, who is accountable? Credit scores at least use relevant facts (borrowing history); ML scores use opaque correlations with no recourse for errors.

Feedback Loops

Predictive systems can create self-reinforcing downward spirals. Example: credit score affects employability; joblessness worsens credit score; further reduces chances of employment. Systems thinking is required to anticipate these effects.

Privacy and Surveillance

Kleppmann's strongest language in the book: "we have built the greatest mass surveillance infrastructure the world has ever seen." He proposes renaming "data" to "surveillance" in common phrases to expose the reality: "surveillance-driven organization," "surveillance warehouse," "surveillance scientists."

• Consent is not meaningful when privacy policies are incomprehensible, terms are non-negotiable, and alternatives impose social costs (not using Facebook = missing social participation).
• Privacy ≠ secrecy — privacy is the freedom to choose what to reveal to whom. Surveillance transfers that decision right from individuals to corporations.
• Data as toxic asset — breaches leak intimate data; bankruptcies sell personal data; governments compel or steal it.

The Industrial Revolution Analogy

The Industrial Revolution brought enormous benefits (economic growth, living standards) but also terrible harms (child labor, pollution, worker exploitation) that required regulation to fix. "Data is the pollution problem of the information age" (Bruce Schneier).

Kleppmann's call to action: self-regulate data collection, purge data when no longer needed, build cryptographic access controls, educate users. "Ubiquitous surveillance is not inevitable — we are still able to stop it."

Summary

The key ideas of Chapter 12:

1. Data integration via derivation. No single tool fits all use cases. Designate a system of record; derive indexes, caches, ML models, and analytics views via batch and stream processing.
2. Logs over transactions. Log-based derived data is more robust than distributed transactions for heterogeneous system integration — comparable integrity, better fault containment, no 2PC/XA overhead.
3. Unbundling. A database's internal features (indexes, materialized views, replication) can be decomposed into independent stream-connected components — like Unix pipes for data.
4. Write path / read path. Every derived dataset sits between eager precomputation (write path) and lazy serving (read path). Indexes, caches, and materialized views shift this boundary.
5. Integrity over timeliness. Async dataflow systems preserve integrity (no corruption) without timeliness (linearizability). Most applications can tolerate loose constraints with post-facto apologies.
6. End-to-end correctness. TCP, database transactions, and stream processor exactly-once are insufficient — application-level operation IDs and uniqueness constraints are needed for true exactly-once semantics.
7. Trust but verify. Assume hardware and software will eventually corrupt data. Build self-auditing systems: read back, compare replicas, run redundant derivations.
8. Ethics matter. Predictive analytics can amplify bias and discrimination. Surveillance is not inevitable. Engineers carry responsibility for how their systems affect the world.

Quiz

Select the correct answer for each question.

Notes

Primary source: Kleppmann, M. (2017). Designing Data-Intensive Applications, Chapter 12: "The Future of Data Systems." O'Reilly Media.
Recommended supplements: Jay Kreps' "The Log: What Every Software Engineer Should Know" (for the unbundling thesis), the Dataflow Model paper (Akidau et al., VLDB 2015), the end-to-end argument paper (Saltzer, Reed, Clark, ACM TOCS 1984), Pat Helland's "Life Beyond Distributed Transactions" (CIDR 2007), "Immutability Changes Everything" (CIDR 2015), and for ethics: Cathy O'Neil's "Weapons of Math Destruction," Bruce Schneier's "Data and Goliath," and the ACM Code of Ethics.

Ask follow-up questions to drill into: how differential dataflow works, the exact mechanics of coordination-avoiding data systems, the CALM theorem, how to design end-to-end exactly-once with operation IDs, the Lambda Architecture debate in detail, practical DB unbundling with Kafka/Debezium/Streams, engineering ethics in practice, or how to build auditing systems with Merkle trees.