Every data-intensive application — Twitter, Amazon, Uber, Dropbox — is built from the same few building blocks: databases, caches, search indexes, message queues, and batch processors. What separates a system that survives Black Friday from one that crumbles under load comes down to three properties. This lesson covers the three-word answer to the most common system design interview opener: "How would you design this?"
Modern applications don't use one database. They stitch together multiple specialized tools — a primary database, a caching layer (Memcached/Redis), a search index (Elasticsearch/Solr), and a message queue (Kafka). The application code becomes the glue that keeps these in sync.
You are not just an application developer — you are also a data system designer. When your system combines a cache, a full-text index, and a message broker, you own the guarantees: cache invalidation, index consistency, exactly-once processing. The composite system must keep its promises even when individual components fail.
Kleppmann argues that nearly all design decisions in data systems boil down to three concerns. When an interviewer asks "How would you design Twitter?" or "Design a URL shortener," your thinking should thread through all three.
Definition: the system continues to work correctly even in the face of adversity — hardware faults, software bugs, and human error.
A fault is when one component deviates from its spec (a disk dies, a CPU overheats). A failure is when the entire system stops serving the user. Fault-tolerance means faults don't cascade into failures.
| Fault type | Nature | Countermeasure |
|---|---|---|
| Hardware | Random, independent (disk MTTF ~10–50 years; 10k disks = 1 dead/day) | Redundancy (RAID, dual PSU) → moving to software fault tolerance (rolling upgrades, tolerate whole-machine loss) |
| Software | Systematic, correlated across nodes (leap-second bug, runaway process) | Process isolation, crash+restart, monitoring, self-checks. Chaos Monkey to deliberately trigger faults. |
| Human | Leading cause of outages (config errors > hardware) | Sandbox environments, thorough testing, fast rollback, clear monitoring/telemetry, good abstractions that make "right thing" easy. |
Interview tip: when asked "How would you make this reliable?" don't just say "replication." Break it down by fault type: hardware → replication + redundancy; software → isolation + monitoring; human → rollback + sandbox. This three-axis decomposition is what interviewers want to see.
Definition: the system's ability to cope with increased load. Not a binary label ("scalable" / "not scalable") — it's a discussion: "If the system grows in this particular way, what are our options?"
Load is captured via load parameters — numbers specific to your architecture: requests/sec, read/write ratio, concurrent users, cache hit rate, fan-out factor.
The Twitter fan-out story — the single most-cited example in system design interviews:
The lesson: the distribution of followers per user is the real load parameter — averages hide the celebrity problem.
| Metric | Meaning | Interview relevance |
|---|---|---|
| p50 (median) | Half requests ≤ this threshold. "Typical" wait. | Good for UX dashboards, not for SLOs |
| p95, p99 | 95% / 99% of requests ≤ this threshold | Most SLOs/SLAs target these. p99 is common for backend latency budgets. |
| p99.9 (tail) | 1 in 1,000 requests exceeds this threshold | Amazon uses p99.9; 100ms increase → 1% sales drop. Tail latency amplification: 1 slow backend call slows the entire end-user request. |
Latency ≠ Response Time. Response time = service time + network delays + queueing delays (what the client sees). Latency = time the request waits before being served. Queueing delay dominates tail latencies — head-of-line blocking causes fast requests to wait behind slow ones. Always measure response times client-side.
| Approach | Description |
|---|---|
| Scale-up (vertical) | Bigger machine. Simple but expensive at high end. |
| Scale-out (horizontal) | Distribute across many smaller machines. Complex for stateful systems. |
| Elastic | Auto-add resources when load spikes. Good for unpredictable load. |
| Manual | Fixed capacity, provisioned ahead. Simpler, fewer surprises. |
The key insight: a scalable architecture is built around assumptions about load parameters. If your assumptions are wrong, the scaling effort is wasted. For an early-stage startup, iterating on product features beats scaling to hypothetical future load.
Definition: over time, many different people will work on the system (engineering and ops). They should all be able to work on it productively. The majority of software cost is not initial development — it's ongoing maintenance.
Three design principles:
| Principle | What it means |
|---|---|
| Operability | Make life easy for ops: good monitoring, automation hooks, documentation, predictable behavior, self-healing with manual override. |
| Simplicity | Manage complexity. Remove accidental complexity (implementation artifacts) vs. essential complexity (inherent to the problem). Abstraction is the best tool: SQL hides disk structures and crash recovery behind a clean interface. |
| Evolvability | Make change easy. Requirements will change. Simple, well-abstracted systems are easier to refactor at the data system level (e.g. Twitter's approach 1 → 2 migration). Agile + TDD work at the local scale; evolvability works at the architectural scale. |
When an interviewer says "How would you design X?" your first response should translate the three pillars into the domain of X:
1. Fault vs Failure
One component deviates vs system stops Both mean the same Only hardware problems qualify2. Twitter's real load parameter is:
Tweets per second Distribution of followers per user Home timeline reads per second3. p99.9 tail latency matters because:
Affects most user requests directly 1% sales drop per 100ms observed Amplified across backend → slow userModern apps stitch database + cache + search + queue → you glue the guarantees
System continues correct function despite adversity (fault)
Fault ⇒ single component deviates from spec
Failure ⇒ entire system stops serving user
Cannot prevent all faults → build tolerance; deliberately trigger faults (eg Chaos Monkey)
a) Hardware ⇒ random, independent (disk crash, power fail)
Solution :- redundancy (RAID, dual PSU) → now prefer software fault tolerance (tolerate whole-machine loss, rolling upgrades)
b) Software ⇒ systematic, correlated across nodes (eg leap-second bug crashing all servers)
Solution :- process isolation, crash+restart, monitoring, self-checks
c) Human ⇒ config errors = leading cause of outage
Solution :- sandbox env, automated testing, fast rollback, good abstractions that discourage wrong action
Cope with increased load → load parameters define growth dimension
Key load param ⇒ follower distribution (× tweet volume, × read volume)
→ fan-out :- posts go to all followers' timeline caches
→ celebrity users (30M followers) → hybrid (fan-out regular users; fetch celebrities at read time)
a) p50 (median) ⇒ typical wait
b) p95, p99 ⇒ SLO/SLA targets
c) p99.9 (tail) ⇒ Amazon metric; 100ms increase → 1% sales drop
d) Tail amplification ⇒ 1 slow backend call slows entire user request (even if parallel)
e) Latency ≠ Response Time :- latency = wait time in queue ; Response Time = client's total (service + network + queueing)
a) Scale-up ⇒ vertical ; bigger machine
b) Scale-out ⇒ horizontal ; distribute load across small machines → complex for stateful systems
c) Elastic ⇒ auto-add resources ; good for unpredictable load
d) Manual ⇒ fixed capacity ; simpler
Majority of cost = ongoing maintenance, not initial build
a) Operability ⇒ ops teams work efficiently (monitoring, automation, docs, predictable behavior)
b) Simplicity ⇒ remove accidental complexity via abstraction (eg SQL hides crash recovery)
Accidental × → inherent in problem → from implementation
c) Evolvability ⇒ adapt to changing requirements at data-system scale (not just local refactoring)
Reliability → identify fault types → propose tolerance per type
Scalability → pick load parameters → what breaks at 10×
growth?
Maintainability → name abstractions → new engineer picks up in a
day?
Primary source: Kleppmann, M. (2017).
Designing Data-Intensive Applications, Chapter 1: “Reliable, Scalable, and
Maintainable Applications.” O'Reilly Media.
Recommended supplement: Netflix Tech Blog — “The Netflix Simian
Army” (original Chaos Monkey article) at
techblog.netflix.com
Questions? Ask your agent — you can follow up on any concept, quiz answer, or how this applies to a specific FAANG interview question.