This chapter is not just about serialization formats. It is really about evolvability under change: how a system keeps working while code versions, schema versions, and deployment waves overlap. In FAANG-style interviews and real systems, this is one of the deepest practical ideas in distributed systems: data almost always outlives code, so compatibility rules matter more than syntax preferences.
Mission tie-in: if you want to build production-grade systems and also answer senior-level design questions well, you need to reason about schema evolution, rolling upgrades, backward/forward compatibility, API versioning, and how data moves through databases, services, and message brokers. Chapter 4 gives that vocabulary.
Applications change continually. Features are added, fields are renamed, payloads gain new attributes, and services are rolled out gradually rather than all at once. That means old and new code often coexist.
Two deployment realities make this unavoidable:
Therefore, data written by one version must often be readable by another version.
| Compatibility type | Meaning | Why it matters |
|---|---|---|
| Backward compatibility | New code can read old data | Needed when code is upgraded before old data is rewritten |
| Forward compatibility | Old code can read new data | Needed during rolling deploys and client lag |
Backward compatibility is usually easier because the author of the new code knows the old format. Forward compatibility is harder because old code must safely ignore things it does not understand.
Programs usually keep data in pointer-rich in-memory structures such as objects, maps, arrays, and trees. But to write data to disk or send it across the network, that data must become a self-contained byte sequence.
The transformation from memory structures to bytes is called encoding. The reverse is decoding. Other common names are serialization/marshalling and deserialization/unmarshalling.
Important boundary: encoding is not encryption. Encoding makes data transportable; encryption makes data confidential.
Many languages ship with built-in object serialization: Java
Serializable, Python pickle, Ruby Marshal, and similar
libraries. They feel convenient because they save and restore in-memory objects with minimal
code. But they are usually a bad choice for persistent or cross-process data exchange.
| Problem | Why it is serious |
|---|---|
| Tied to one language/runtime | Locks you into the current language and makes cross-language systems painful. |
| Security risk | Decoders often instantiate arbitrary classes, which has historically led to remote code execution bugs. |
| Poor versioning story | Forward/backward compatibility is often an afterthought. |
| Weak efficiency | Encodings can be bloated and slow to parse. |
This is why serious distributed systems typically use language-independent formats rather than built-in object dumps.
JSON, XML, and CSV are popular because they are language-independent and human-readable enough to be inspected manually. That matters a lot for interoperability, especially across organizations.
But they carry subtle problems that are easy to ignore until they hurt you.
JSON distinguishes strings from numbers, but not integers from floating-point numbers, and it does not define precision. XML and CSV are even looser unless you rely on external schemas.
This becomes dangerous with large integers. JavaScript uses IEEE 754 double-precision numbers,
so integers larger than
2^53 cannot be represented exactly. Kleppmann's Twitter example matters because
tweet IDs are 64-bit integers; JSON consumers in JavaScript can silently corrupt them. Twitter
worked around this by returning IDs both as numbers and as decimal strings.
JSON and XML handle Unicode text well, but not arbitrary byte sequences. So binary data is often Base64-encoded as text, which is workable but increases payload size by roughly 33% and requires schema/context knowledge to interpret correctly.
XML Schema and JSON Schema exist, but they are powerful enough to be complicated. Many JSON systems skip schemas entirely, which makes payload interpretation a convention rather than an enforced contract.
CSV has almost no built-in semantics. Applications must agree manually on what rows and columns mean, and escaping rules are not consistently implemented. A column addition or meaning change is an application-level migration problem.
Practical view: JSON is often not the best technical format, but it is frequently the best organizational format because everyone can work with it.
Many binary formats such as MessagePack, BSON, Smile, and WBXML attempt to keep the JSON or XML data model while reducing size and parsing overhead. These formats are better than raw text, but usually not dramatically better.
Example: the sample JSON record
{ "userName": "Martin", "favoriteNumber": 1337, "interests": ["daydreaming", "hacking"] }
becomes 66 bytes in MessagePack versus 81 bytes in compact textual JSON. That is a modest improvement, but not a transformational one.
The core reason is structural: if you do not have an external schema, the encoded payload must
still carry field names like
userName, favoriteNumber, and interests. That metadata
dominates small records.
Thrift and Protocol Buffers take a different approach: the format is driven by an explicit schema known ahead of time. That lets them omit field names from the payload entirely.
Instead of writing full field names into the data, the payload contains compact
field tags such as 1, 2, and 3. The
schema tells the decoder what those tags mean.
That is why Thrift CompactProtocol and Protocol Buffers shrink the sample record to roughly 34 and 33 bytes respectively, while Avro can push it to 32 bytes.
In Thrift and Protocol Buffers, fields may be marked
required, optional, or in protobuf also repeated. But
the encoded bytes do not carry a marker saying "this field was required." The tag and type
drive decoding; required mainly adds runtime validation.
The crucial design rule in Thrift and Protocol Buffers is this:
Field tags are forever. You may rename a field, but you must never change or reuse its tag number once data has been written with it.
You can add a new field as long as it gets a brand-new tag number. Old readers will see an unknown tag and skip it. That gives forward compatibility: old code can read new records by ignoring fields it does not know.
New code can still read old records because the old tags mean the same thing they always meant. But there is one major restriction: new fields added after initial deployment must be optional or have defaults. If a newly added field were required, new readers would fail on older records that never had that field.
Removing a field is effectively the mirror image of adding one. You can only safely remove fields that were optional, and you must never reuse the removed tag number later.
Some type changes are partially safe, but many are dangerous. Example:
int32 to int64 is often okay when new code reads old
data.
Protobuf encodes repeated fields by repeating the same field tag multiple times. That creates a nice compatibility property:
This makes some single-to-multi-valued evolutions easier than in Thrift.
Avro is schema-driven like Thrift and Protocol Buffers, but it does something radical: the binary payload contains almost no self-description at all.
No field names. No field tags. Just the values, encoded in schema order.
That is why Avro can be the most compact of the examples in this chapter. But it creates a sharper requirement: decoding only works if reader and writer schemas are correctly related.
This is the central Avro idea.
Avro resolves differences by comparing the two schemas side by side. It matches fields by name, not by numeric tag.
| Change | Compatibility rule |
|---|---|
| Add field | Safe only if the new field has a default value. |
| Remove field | Safe only if readers can supply/ignore via defaults and resolution rules. |
| Rename field | Possible using aliases; backward-compatible, but not always forward-compatible. |
| Add union branch | Often backward-compatible, not necessarily forward-compatible. |
| Change type | Safe only if Avro can convert between the types. |
Avro does not treat everything as nullable by default. If a field may be null, you must say so with a union such as:
["null", "long"]
This is more verbose but prevents accidental ambiguity.
This depends on context:
| Context | How writer schema is obtained |
|---|---|
| Large file of many records | Store schema once in the file header (Avro object container file). |
| Database with individually written records | Store schema version ID with each record and resolve via schema registry/database. |
| Network connection | Negotiate schema version when the connection is established. |
Since Avro matches by field name and avoids numeric tags, it works well for schemas generated automatically from another source, such as relational table definitions.
That matters operationally. If a table gains or loses a column, a data-export process can simply generate a fresh Avro schema and continue. With tag-based formats, someone usually has to preserve and manage tag assignments carefully.
Thrift and Protobuf lean hard on code generation, which is especially nice in Java/C++/C#. But in Python, Ruby, and JavaScript, mandatory code generation is often a workflow tax.
Avro can be used without generated classes. If a file contains its writer schema, tools can inspect the data directly. That is a big reason Avro fits systems like Hadoop and Pig so naturally.
Contrast: Protobuf/Thrift optimize around stable, explicitly managed schemas and strong generated-code workflows. Avro optimizes around schema resolution and dynamic data pipelines.
Schema-driven binary formats are not just about compression. Their biggest value is operational discipline.
Kleppmann's summary point is subtle and important: schema evolution provides much of the flexibility people seek from schemaless systems, but with better guarantees and tooling.
The second half of the chapter asks a broader question: once data is encoded, how does it flow between processes? Compatibility requirements depend on that path.
In a database, the writer encodes the data and some later reader decodes it. Sometimes that reader is just a later version of the same application, which is why Kleppmann says storing in a database is like sending a message to your future self.
During rolling upgrades, newer and older application instances may access the same database concurrently. That means:
Suppose new code adds a field and writes it to the database. Later, old code reads the row, updates some known field, and writes the row back. If the application decodes into an object model that does not preserve unknown fields, the newly added field may be silently lost on re-encode.
This is one of the most practical warnings in the chapter: compatibility at the wire/storage format level does not automatically guarantee preservation at the application-object level.
Old application binaries may disappear in minutes after a rollout. Old database records may survive for years. That is why full database rewrites are avoided when possible. Many relational databases can add nullable columns without rewriting every old row. Avro-backed systems such as LinkedIn Espresso rely on schema resolution instead of massive rewrites.
Backups, snapshots, and warehouse exports are opportunities to rewrite data into a new consistent encoding, because the data is being copied anyway. That is why Avro object container files or analytical formats like Parquet are attractive for archival/export use cases.
Services expose network APIs. Unlike databases, their interface is usually narrower and shaped by application logic rather than arbitrary query languages. This yields encapsulation, but also creates API-evolution constraints.
In a microservices-style system, one service often becomes a client of another service. Independent deployability is the goal, which means older and newer service versions coexist. Therefore, API payloads need exactly the same kind of compatibility we have been discussing.
| Aspect | REST | SOAP |
|---|---|---|
| Philosophy | Design style built around HTTP principles | XML-based protocol stack |
| Payloads | Usually JSON or simple forms | XML |
| Tooling model | Often lightweight, manual, debuggable | Heavy tool/codegen dependence via WSDL and WS-* |
| Human approachability | High | Low |
| Cross-org fit today | Common | Legacy-heavy enterprise use |
SOAP's ambition was protocol-independence and rich standardization, but that richness became a burden. REST won mindshare partly because it keeps the network visible instead of pretending remote calls are just local method invocations.
RPC tries to make a network request look like an ordinary local function call. Kleppmann's argument is that this is conceptually misleading.
| Local function call | Remote call |
|---|---|
| Predictable and under your process control | Subject to network loss, delay, congestion, and remote machine failure |
| Returns, throws, or hangs | May also time out with ambiguous outcome |
| No duplicate side effect from retries | Retries may duplicate effects unless protocol is idempotent |
| Fast and relatively stable latency | Slower and highly variable latency |
| Can pass in-memory references | Must encode parameters into bytes |
| Single language runtime | Potential cross-language type mismatch |
This is why location transparency is seductive but flawed. A remote service should not be treated as merely "an object over there."
RPC is still widely used. Modern frameworks like gRPC, Finagle, Rest.li, Avro RPC, and Thrift RPC are more honest about remote uncertainty.
Even so, the best use case for custom binary RPC is usually inside one organization, where you control both ends and care about efficiency. For public APIs, REST remains attractive because debugging, tooling, browser/curl access, and ecosystem support often matter more than raw efficiency.
In service-to-service evolution, Kleppmann makes a simplifying assumption: servers are upgraded first and clients second. Under that model:
But across organizational boundaries, this assumption weakens. Providers cannot force all clients to upgrade. So compatibility may need to be maintained for years, sometimes indefinitely, and providers often run multiple API versions in parallel.
There is no universal best practice. Common REST patterns include:
Accept headerThe deeper lesson is that versioning is not the first line of defense. Good compatibility practices reduce the need for breaking versions in the first place.
Asynchronous message passing sits between RPC and databases. Like RPC, it moves data between processes with low latency. Like databases, it often goes through an intermediary that stores data temporarily.
| Benefit | Why it matters |
|---|---|
| Buffering | Absorbs bursts and tolerates temporarily slow consumers. |
| Redelivery | Can re-send messages after consumer crashes. |
| Location decoupling | Producers need not know consumer IP/port. |
| Fan-out | One message can reach multiple consumers. |
| Loose coupling | Publishers do not need strong knowledge of subscribers. |
The tradeoff is that brokers are usually one-way and asynchronous by default. The sender publishes and moves on; if a reply is needed, it is usually carried on a separate channel.
Brokers usually treat messages as byte arrays plus metadata and impose no data model. That makes the encoding choice your responsibility.
If publishers and consumers evolve independently, backward and forward compatible formats are extremely valuable. And if one consumer republishes a message onward, it may need to preserve unknown fields to avoid the same data-loss issue seen with databases.
Actor systems are another form of message-passing architecture. Each actor has local state and processes one message at a time, which avoids many thread-level concurrency hazards such as shared-memory races and lock contention.
In distributed actor frameworks, the same model extends across machines. Messages are transparently encoded and sent over the network.
The actor model already assumes asynchronous message passing and possible message loss, even conceptually. So the mismatch between local and remote is smaller than in RPC, where programmers are encouraged to imagine a remote call as a normal function call.
If old and new actor nodes exchange messages during a rolling upgrade, payload compatibility still matters. Kleppmann gives concrete warnings:
| Mode | Who writes? | Who reads? | Main compatibility pressure |
|---|---|---|---|
| Database | Current app version | Future app versions and concurrent older/newer nodes | Data outlives code; preserve unknown fields |
| RPC/REST | Client for request, server for response | Server for request, client for response | Independent deployability; server/client version skew |
| Message broker / actors | Producer/publisher | Consumer/subscriber | Loose coupling and independent evolution over time |
If an interviewer asks how you would evolve an API, event schema, or stored object model, answer in this order:
Interview-quality answer: "Because data will outlive the currently deployed binaries, I would treat schema evolution as a first-class design concern. For internal event streams I would use a schema-driven format with compatibility checks in CI, and I would only add optional/defaulted fields so rolling upgrades remain safe."
1. What is forward compatibility?
New code reads old data Old code reads new data Both sides use identical schema versions2. Why are language-specific serialization formats risky in distributed systems?
Language lock-in, security risks, weak evolution story They cannot encode nested objects They only support strings and numbers3. Why do binary JSON variants only modestly reduce size?
They still carry field names and structure metadata They are not allowed to use integers They are always slower to parse than text JSON4. What is the non-negotiable rule for field tags in Thrift/Protobuf?
Tags may be renumbered if field names stay the same Never change or reuse a tag once deployed Tags matter only for required fields5. Why must newly added fields usually be optional or defaulted?
So newer code can read older records safely So the field can be encrypted later Because binary formats forbid required fields6. What is Avro's key schema-evolution mechanism?
Writer-schema and reader-schema resolution Permanent numeric field tags Embedding the full schema inside every record7. What is the unknown-field trap in database-backed applications?
Older code may accidentally drop newer fields on rewrite Databases reject any row with extra fields New fields are automatically duplicated by old code8. Why is RPC fundamentally different from a local function call?
Network failure and timing semantics are fundamentally different RPC cannot pass primitive values RPC is faster because work happens elsewhere9. Why are message brokers attractive compared to direct RPC?
Buffering, redelivery, fan-out, and loose coupling They make all communication strictly synchronous They automatically enforce one universal schemaCode changes faster than data disappears → old + new versions coexist
Backward compatibility ⇒ new code reads old data
Forward compatibility ⇒ old code reads new data
In-memory objects × portable across process boundary
Need byte sequence for disk / network
Encoding ⇒ memory to bytes ; Decoding ⇒ bytes to memory
Convenient short-term ; dangerous long-term
Problems :- a) language lock-in b) security bugs from arbitrary class instantiation c) weak schema evolution d) bloated/slower encoding
Human-readable + interoperable → great for public exchange
Number ambiguity :- JSON doesn't define precision ; JS breaks for integers > 2^53
Binary data :- use Base64 → size +33%
Schema :- optional / external / sometimes skipped
CSV :- weakest semantics ; app must define meaning manually
Better than text ; still modest win
Reason :- field names still carried inside payload
Schema-driven binary encoding
Field tag ⇒ compact alias for field meaning
Field names can change ; tag numbers × cannot change
Add field → give new tag ; old reader skips unknown tag
New field must be optional/defaulted for backward compatibility
Removed tag × never reuse
No tags in payload ; no field names in payload ; values serialized in schema order
Writer's schema v/s Reader's schema :- resolve differences at decode time
Match by field name ; writer-only field ignored ; reader-only field filled from default
Add/remove field safely only with defaults
Good for dynamic schemas + Hadoop-style data pipelines
More compact payloads
Executable documentation
Compatibility checks before deploy
Code generation useful in static languages
Writing to DB ⇒ message to future self
Need backward + forward compatibility during rolling upgrade
Unknown-field trap :- old code reads newer record, rewrites it, drops unknown field accidentally
Data outlives code :- old rows survive years ; binaries may vanish in minutes
Service API narrower than DB query language → more encapsulation
REST :- HTTP-native style ; simple payloads ; debuggable ; strong ecosystem
SOAP :- XML + WSDL + WS-* ; tooling-heavy ; enterprise legacy
RPC flaw :- remote call × local call
Network request has timeout ambiguity, retries, idempotence issues, variable latency, encoding cost, cross-language mismatch
Typical assumption :- servers upgrade first, clients later
Requests need backward compatibility ; responses need forward compatibility
Public APIs may need compatibility for years → multiple versions may coexist
Between RPC and DB
Buffering, redelivery, fan-out, location decoupling, loose coupling
Usually one-way + async ; reply if needed on separate channel
Broker treats payload as bytes → encoding compatibility still your job
Actors process one message at a time ; local state not shared
Distributed actor systems use same message model across nodes
Location transparency works better than RPC because async/message-loss already assumed
Rolling upgrades still require schema/message compatibility
Primary source: Kleppmann, M. (2017).
Designing Data-Intensive Applications, Chapter 4: “Encoding and
Evolution.” O'Reilly Media.
Recommended supplements: Protocol Buffers developer guide, Apache Avro
specification, and practical articles on API versioning and schema registries.
Ask follow-up questions if you want to drill into Avro vs Protobuf, schema registries, event evolution, or how to explain compatibility strategy in a system design interview.