16 — The Learning Continues

This chapter is the compression: forty-six transferable rules of large-scale system design, extracted from the twenty-eight worked problems that produced them. The track is twenty-nine chapters; this one carries no problem of its own.

A rule is one line: a mechanism, plus the number that makes it true. For example: database load falls as (1-h), where h is the fraction of reads the cache answers, so each additional nine of hit rate divides database load by ten.

That rule has three parts: a mechanism (caching), a number (1-h), and something you can act on (the last few points of hit rate are worth more than the first thirty). Every row in this chapter’s rule tables has the same three parts. It is the rule, not the problem it came from, that transfers to a design question no chapter covers.

What you should be able to do when you finish. Name the pattern an unfamiliar problem belongs to, quote the arithmetic that decides it, and state which of your own assumptions the answer rests on.

Input and output. In goes a system design problem you have not seen, described in a sentence or two: “design a concert ticketing system”, “design a ride-hailing dispatcher”. Out comes a short, defensible chain:

1. Five numbers, written down before anything else.
2. One fork, taken because of the first of those numbers.
3. Three independent checks, run in every case.
4. A named architecture, with the binding constraint stated explicitly.

Worked the chart applied to a problem with no chapter runs that chain end to end on concert ticketing, from the arithmetic to the architecture.

Every idea this chapter uses is defined here before it is used. Follow a link when you want the derivation with its numbers; you should not need to follow one to understand a sentence on this page. The vocabulary everything else is written in defines the terms of art up front.

This chapter sits at the seam of the study plan. Chapters 01-15 are Volume 1. 17 through 29 are Volume 2. 1b volume 2 as twenty five more and Volume 2 the binding constraints are written to be useful both before you read Volume 2 and after.

Read it once now. Re-read Which pattern does this smell like and The numbers you cannot recompute under pressure the morning of the interview.

0. The vocabulary everything else is written in

Every term of art the rest of the chapter uses is defined here in plain English. Using a mechanism’s name without being able to define it is a common way to sound like you do not understand it.

Read it once, then treat it as a lookup table: jump back here whenever a later section uses a word you cannot define.

Terms about the systems themselves

• A node or box is one server. A fleet is however many of them a service runs on. RAM is random-access memory, the fast working memory a program computes out of, as opposed to disk.
• QPS is queries per second: how many requests a second the system handles. Offered load is the rate arriving; service capacity is the rate the system can absorb. They are different quantities that share a unit, which is the source of the single most common arithmetic mistake in this round.
• Latency is how long one request takes. p99 is the ninety-ninth-percentile latency — the time that 99 requests out of 100 come in under — so it describes the slow tail rather than the typical case. p99.9 is the same idea one decimal place further out: the time 999 requests in 1,000 beat. It matters because a request that has to wait on ten machines inherits a tail well past any one machine’s p99. RTT is a round trip time: request out, response back.
• Throughput is work per unit time; headroom is the gap between capacity and offered load, and it is what determines how fast a backlog drains.
• Utilization, written rho (the Greek letter rho), is offered load divided by capacity. Queueing delay grows as rho / (1 - rho), which is why a system at 90% utilization waits nine service times and a system at 50% waits one.
• CV is the coefficient of variation — the standard deviation divided by the mean — which is the right way to express load imbalance across machines because it is unitless and so compares across fleet sizes.
• Blast radius is how much of the system one failure takes down with it, which is why fleets are sized by the damage one lost machine does and not only by capacity.
• A DAG is a directed acyclic graph: a set of steps with dependencies between them and no cycles, which is the natural shape of a processing pipeline such as video transcoding.
• HTTP status codes appear as shorthand: 409 means Conflict — your write lost a race — and 429 means Too Many Requests, the polite way for a server to shed load.

Terms about storing and finding data

• Sharding or partitioning means splitting one dataset across several machines by some key, so that each machine holds a slice. The shard key is the column whose value decides which machine a row lives on. Replication means putting identical full copies on several machines. Sharding buys capacity; replication buys read throughput and survivability.
• The working set is the data actually touched by live traffic, which is usually far smaller than the data stored. It is the number that decides whether you need to shard at all.
• A scatter-gather query is one that has to ask every shard and combine the answers, which makes its latency the slowest of the responses.
• A hot key is a single key that receives a disproportionate share of the traffic. Partitioning cannot help it, because one key hashes to one place by construction.
• A cache is a fast copy of data in front of a slower store. The hit rate h is the fraction of reads the cache answers; (1-h) is the fraction that fall through. An LRU cache is a least-recently-used cache: when it is full, it evicts whatever has gone longest without being read. A TTL is a time to live, the age at which a cached copy must be discarded.
• Zipfian describes an access pattern where a small number of items get most of the requests — the shape of nearly all real-world popularity.
• A CDN is a content delivery network: caches placed physically near users so a request does not have to cross an ocean.
• Cardinality is how many distinct values something has. In a monitoring system, a label with unbounded cardinality is one whose set of possible values grows with traffic, which is why one label can multiply your storage bill by four orders of magnitude.
• A KV store is a key-value store: a database whose only operations are “put this value under this key” and “get the value at this key”. A B-tree is the classic index structure a relational database uses, kept sorted and updated in place. An LSM tree is a log-structured merge tree, the alternative that buffers writes in memory and writes whole sorted files out, merging them in the background — which is why it beats a B-tree on write-heavy workloads.
• ZREVRANK is the Redis command that returns an item’s position in a sorted set. It is named here only because reaching for it is the reflex answer to a leaderboard question, and it does not shard.

Terms about correctness when there is more than one machine

• A quorum is a required number of replicas that must agree. In a store with N replicas, W is how many must acknowledge a write and R how many must be read; W + R > N guarantees that any read overlaps any write.
• A partition, in the network sense, is a break that leaves two halves of a system unable to talk. CAP is the observation that during one you must choose between remaining consistent and remaining available.
• Consistent hashing places keys on a conceptual ring rather than by key mod N, so that adding or removing a machine moves the minimum possible fraction of keys. Virtual nodes are many small ring positions per physical machine, which evens out the load.
• A version vector or vector clock is a per-writer counter set attached to an object, which lets you tell whether two versions are causally ordered or genuinely concurrent. It detects conflicts; it does not resolve them.
• A sibling is one of the concurrent versions such a detection leaves behind.
• Compare-and-swap is a write that only succeeds if the value is still what you last read — the cheap single-owner alternative to a version vector.
• Last-write-wins resolves a conflict by keeping whichever version has the later timestamp, silently discarding the other. It is data loss, not a policy.
• Idempotent means an operation can be applied twice with the same result as applying it once. An idempotency key is the client-supplied identifier that makes a retry safe.
• A Merkle tree is a tree of hashes over a dataset, which lets two replicas find their differing keys in O(log n) comparisons instead of O(n).
• Anti-entropy is the background process that uses such a comparison to repair diverged replicas.
• 2PC is two-phase commit: a coordinator asks every participant to prepare, then tells them all to commit. A saga is the alternative — run the steps one at a time and undo the earlier ones with compensating actions if a later one fails.
• Linearizable means every read sees the most recent write, as if there were one copy.
• Eventual consistency is the opposite end of that scale: replicas are allowed to disagree for a while and are only guaranteed to converge once writes stop. It is what a quorum below W + R > N buys, what Dynamo chooses and Spanner refuses, and the concept the whole contrast in Seven papers and the problem each one actually solved turns on.

Terms about streams, queues and time

• A queue or log carries messages between services. Backpressure is the mechanism by which a full downstream slows an upstream instead of collapsing. Admission control is the version of that decision made at the front door: let work in only at the rate the system behind can absorb, and turn the rest away immediately.
• An offset or cursor is a consumer’s read position in a log; owning it is what makes replay free.
• Consumer lag is how far behind the head of the log a consumer is.
• Event time is when something happened; processing time is when your system saw it. A watermark is the system’s declaration that it believes all events up to a given event time have now arrived.
• An upsert writes a row if it is absent and replaces it if it is present. An absolute-value upsert writes the running total (= 47) rather than an increment (+= 1), which is what makes replaying the same message twice a no-op.
• Restatement, or amendment, is publishing a corrected figure after you have already published a fast one. It is the alternative to waiting long enough to be right the first time.
• Fan-out is one input producing many outputs — one tweet becoming a million feed writes. Fan-in is the reverse.
• Jitter is deliberate randomness added to retry or reconnect delays so that clients do not synchronize into a stampede.
• Salting a key means appending a random suffix to spread one hot key across several partitions, then merging the parts.

Terms about storage economics, money and the rest

• RF 3, replication factor three, means three full copies, so 3x the bytes. Erasure coding, written RS(k, m) for Reed-Solomon, splits an object into k data fragments plus m parity fragments; any k of the k + m reconstruct it, which gives durability at (k+m)/k times the bytes — 1.4x for RS(10,4).
• IOPS is input/output operations per second, the count of separate reads or writes a device can do. NVMe is the modern fast solid-state disk interface. Write amplification is the ratio between bytes the storage engine actually writes to the device and bytes the application handed it — a 50-byte row that costs 2,100 bytes of device writes is 42x amplified.
• A Bloom filter is a compact probabilistic structure that answers “is this item in the set” with no false negatives and a tunable false-positive (FP) rate, at about 9.6 bits per entry for 1%.
• A fold is the running result of applying an operation down a sequence — a balance is the fold of every entry in a ledger. Memoized means that result is cached rather than recomputed, so a memoized fold is a stored balance column that must be kept in step with the entries it summarizes.
• A blob store or object store holds arbitrarily large opaque byte strings under string keys, with no query language, no joins and no partial updates — S3 is the familiar example. Half of this track’s storage designs are a small mutable index in a database plus an enormous immutable byte pool in one of these.
• Content addressing means naming a blob by the hash of its bytes, so the name is the checksum and re-uploading the same content is automatically idempotent.
• Deduplication is storing one physical copy of bytes that appear in many logical places. It saves storage always and bandwidth only if the client is told what the server already holds — which is the version that leaks, because that answer is a question about other people’s data.
• CDC, content-defined chunking, cuts a file at boundaries determined by its contents rather than at fixed offsets, so inserting a byte does not shift every chunk.
• GC is garbage collection: the sweep that deletes data nothing references any more.
• Double-entry bookkeeping records every movement of money as equal and opposite entries, so a correct ledger sums to zero per currency and a half-written transfer has no representation at all.
• An SLO is a service level objective, a target such as 99.9% success. The error budget is the allowance it implies — 0.1% of requests — and burn rate is how fast you are spending it.
• Determinism means the same inputs always produce the same outputs, in the same order, with no dependence on wall clock, floating point, hash iteration order or thread scheduling. It is what makes a standby able to replay a log and get the identical state.

Two names for people and services appear repeatedly: APNs is Apple’s push notification service, and a PSP is a payment service provider — the company that actually talks to the card networks on your behalf.

1. Volume 1 as twenty-one rules

The first of the two rule tables: twenty-one mechanisms from Volume 1, each with the chapter that derives it and the one line you should be able to say about it without preparation.

The one-line rule is the deliverable: if you can state it without looking, you know the mechanism. The name and source are the handle to reach for in an interview and where to check the arithmetic; nobody will ask you to recite them.

If you can say fifteen of those twenty-one lines cold, you can hold a system design conversation about a problem nobody has written a chapter for.

Three Volume 1 rules, applied to a problem the track never covers

Some rows above carry their own worked instance, because the number is the instance — “94% against 1/(N+1)” is not abstract. Three rows are genuinely abstract, though, and they are the ones a first reader cannot picture. Here they are on a problem no chapter in this track contains: letting users attach photos to restaurant reviews.

Assume 3 million photos uploaded a day at 2 MB each, and 500 million review page-views a day. Every number below follows from those two.

“Estimate first” — the output is the name of the binding constraint, not a number. Run the arithmetic:

photo writes/s      3,000,000 / 86,400          =  34.7  /s
bytes/day           3,000,000 x 2 MB            =  6 TB/day
bytes over 3 years  6 TB x 365 x 3              =  6.57 PB
read : write        500,000,000 / 3,000,000     =  167 : 1

Thirty-five writes a second is nothing; a single database absorbs it without noticing. The constraint is bytes, not requests — 6.57 petabytes over three years, of data nobody edits after upload. That single sentence is what the estimate was for. It tells you the interview is about storage tiering and erasure coding, and that any minute spent on write throughput is a minute wasted. The 167:1 ratio then tells you the read path is a caching and CDN problem, which is the second thing worth knowing and not the first.

“Queues” — anything off the user’s critical path goes async. The user’s upload must return as soon as the bytes are durable. Everything after that — generating thumbnails, running the content-safety scan, extracting the location tag — is work the user is not waiting for, so it goes on a queue. The rule’s second half is the part people miss: the queue is not there to smooth out a spike, it is there so that when the thumbnail workers fall behind, the backlog has somewhere legitimate to sit instead of pushing latency back into the upload call. If the thumbnail fleet dies for an hour, uploads keep succeeding and thumbnails appear late. That is the whole reason the queue exists.

“Two stores, no transaction” — order the writes so the survivable failure is the one that happens. The photo bytes go in a blob store and the review row goes in a database, and there is no transaction spanning the two. So write the bytes first, then commit the row. If the process dies in between you have paid for bytes nobody references — an orphan — which a background sweep can find and delete. Commit the row first and a crash leaves a review pointing at a photo that does not exist, which the user sees as a broken image and which no sweep can repair. Same two writes, opposite failure, and the only thing you changed was the order.

1b. Volume 2 as twenty-five more

The second rule table is organized around a different question. Volume 1 answers how do I build the mechanism. Volume 2 mostly answers which constraint is actually binding, and what does the honest version of this cost.

The columns work exactly as in Volume 1 as twenty one rules: name, source, deliverable. Read the table once before chapter 17 as a preview and once after chapter 29 as recall; the rows are written to work both ways. On the first pass, a row you cannot picture is a row whose chapter you have not read yet — that is expected, and the second column tells you where to go.

One discipline runs underneath a third of that table and is worth stating on its own, because it is the single most common arithmetic mistake in this round:

State whether a rate is offered load or service capacity at the moment you derive it, and never divide one by the other without saying which is which. Both are measured in requests per second, which is why they get mixed up, and a design that divides one by the other produces a fleet size that is wrong by the ratio between them.

That discipline is stated in ch 03, spent on an ingest path in ch 18 and on partition counts in ch 20, and extended to fleet counts in ch 21, where the same rule applies to machine counts: compare capacity against capacity, or a high-availability configuration against a high-availability configuration, never one of each.

Three Volume 2 rules, applied to a problem the track never covers

Three rows above are stated as principles rather than as numbers, which makes them the hardest to picture. Here they are on a multi-tenant customer-support ticket system — many companies, each with their own agents, their own customers and their own tickets — which no chapter in this track builds.

“Pay the fan-out once, deliberately” — name which subsystem pays, per subsystem. One reply on a ticket has to reach several places: the agent’s inbox view, the customer’s email, the tenant’s analytics roll-up, and the full-text search index. Each of those either pays the fan-out — writing a copy per destination at reply time — or dodges it, by storing one copy and having each reader come find it.

The point of the rule is that you must decide this per subsystem and say so, not let the multiplier land wherever the first implementation happened to put it. Reasonable answer: the agent inbox dodges it (agents query by tenant, and the query is cheap); email pays it, because email is a copy by definition and there is no dodging available; analytics dodges it by reading the same log the inbox reads; search pays it, because an index entry per ticket is the only structure that answers a search fast. Two pay, two dodge, and you can defend each one. The failing answer is not “we fan out” or “we do not” — it is having no per-subsystem answer at all.

“Isolation by partition, not by predicate” — another tenant’s data should not be in the structure being read. The predicate version is one global ticket table with WHERE tenant_id = ? on every query. It is correct today and it stays correct only for as long as every future code path remembers the clause. One forgotten WHERE is a cross-tenant data leak, and no test that runs against a single tenant’s fixtures will ever catch it. The partition version gives each tenant its own index, so the other tenants’ rows are not in the structure the query reads at all. Then a forgotten filter returns fewer results, not somebody else’s. Access control by partition survives refactors that access control by predicate does not.

“Decomposable or not” — ask which one a query is before you shard. Two questions an agent dashboard asks look equally innocent:

• “Show me the 10 oldest open tickets.” This is a top-k selection. Ask every shard for its own 10 oldest, merge the results, keep the top 10. Each shard’s answer is a valid partial answer, so the query shards perfectly.
• “How many open tickets are older than mine?” This is a rank, which is a global count. No per-shard statistic reconstructs it — a shard can tell you how many of its tickets are older, and you only get the answer by adding up every shard, every time.

Same dashboard, same table, and one of the two queries silently becomes a scatter-gather across the entire fleet the moment you shard. The rule says to ask the question before you choose the layout, not after.

2. The twenty-eight problems, and what each one is actually for

Every problem in the track maps to the single mechanism it exists to teach and the single mistake it exists to catch — which is how to pick a chapter by what you need rather than by its title.

No chapter exists for its own sake. Each one is a delivery vehicle for one mechanism you would otherwise never derive, plus one thing candidates reliably get wrong, and compressing a chapter to those two things is the test of whether you read it or skimmed it.

The two tables below hold fifteen problems and thirteen, which is twenty-eight. This chapter is the one that carries no problem of its own.

Volume 1 — the mechanisms

These fifteen chapters each introduce a mechanism.

Chapters 09 through 14 introduce almost no new mechanism. They recombine Volume 1 as twenty one rules’s list under different constraints — which is exactly the claim you are being tested on, that the mechanisms transfer.

Chapter 15 is the exception, and Volume 1 as twenty one rules’s own second column says so. You can check this yourself against the table above rather than taking it on trust.

Count the appearances of 15 in §1’s Derived in column: seven of the twenty-one rows. Now count the rows where 15 is the only citation: three — conflict resolution, delta and cursors, and two stores with no transaction. No other chapter is the sole source of more. Only chapter 01 matches it, with read replicas, shard-last and queues.

So the recombination claim has a boundary, and it is worth stating rather than smoothing over. The six problems before 15 are recombinations. Chapter 15 earns three rules of its own, by being the first problem in the track where two stores must agree without a transaction between them.

If a chapter contributes a rule nobody else derives, saying it introduced no new mechanism would leave three rows of §1 with no home.

Volume 2 — the binding constraints

These thirteen chapters mostly reuse those mechanisms and teach instead which constraint decides the design.

Volume 2 is organized by constraint rather than by mechanism, and three of its chapters are the corners of the space. Ch 23 is pure correctness at a scale that is a rounding error; ch 29 is the same correctness inside a microsecond budget; ch 25 is durability at exabyte scale where the budget is dollars per petabyte-month. Each of those three chapters says so about the other two. Placing an unfamiliar problem against those three corners is faster than pattern-matching it against a mechanism.

3. Which pattern does this smell like?

An architecture for a problem you have never seen can be chosen with one number rather than intuition — and the failure mode to prevent is reaching for a pattern before you have any numbers at all.

Everything below is keyed off five questions you should have asked and answered in the first eight minutes (ch 03):

1. Read:write ratio — how many reads the system does per write.
2. Consistency, field by field — what must be current, and what may be stale.
3. p99 latency target, and where in the world the users are.
4. Retention — how long the data must be kept.
5. Growth — over two years.

Only the first of those five genuinely dispatches. “Dispatches” means it decides which further question you ask; the other four are checks you run in every case, and the gates below are where they land.

The chart reads top to bottom. You enter at the box that says write the five numbers down. You branch once, on the read:write ratio. Each branch asks one follow-up question, and you land in one of six leaves. Every leaf names an architecture and the chapter that prices it. There is nothing else in the chart — no components, no data flow, no arrows meaning “sends a request to”.

flowchart TD
    A["Write the five numbers down FIRST<br/>read:write · consistency per field · p99 and where users are<br/>retention · growth (ch 03 step 1)"] --> B{"read : write"}

    B -->|"under ~10:1"| W{"Is a write a NEWER VALUE<br/>or a NEW FACT?"}
    W -->|"newer value,<br/>supersedes the last"| W1["One slot per key, overwritten.<br/>A 10-deep queue serves a fix 5 TTLs stale.<br/>ch 18 section 12"]
    W -->|"new fact,<br/>must be counted"| W2["Append-only log, then windowed aggregation.<br/>Dedup key plus absolute-value upsert,<br/>never INCREMENT. ch 20, ch 22"]

    B -->|"~10:1 to ~100:1"| M{"Does a small hot set<br/>cover most of the reads?"}
    M -->|"yes, Zipfian"| M1["Cache-shaped after all.<br/>ch 08 at exactly 10:1 buys h = 0.962<br/>with 10 GB of RAM"]
    M -->|"no, flat"| M2["Build both paths and say so.<br/>The middle band is a finding,<br/>not a failure to decide"]

    B -->|"over ~100:1"| R{"Does ONE key carry a<br/>large share of the reads?"}
    R -->|"no"| R1["Shared cache plus replicas.<br/>Each extra nine divides DB load by ten.<br/>ch 02 estimation 5"]
    R -->|"yes"| R2["Hot key. In-process LRU, or replicate<br/>that one key everywhere. Partitioning<br/>cannot help: one key, one hash. ch 05"]

The six leaves in words

The chart is readable without squinting at it if you take the three branches one at a time.

Write-heavy, under about 10 reads per write. The follow-up question is whether a write is a newer value or a new fact.

• A newer value supersedes the last one — a position, a status, a current price. It goes into a single slot per key, overwritten in place.
• A new fact must be counted — a click, a payment, a trade. It goes into an append-only log and is aggregated in time windows afterwards.

The middle band, about 10:1 to about 100:1. The follow-up question is whether a small hot set covers most of the reads.

• Zipfian: the problem turns out to be cache-shaped after all. That is the leaf citing ch 08’s h = 0.962.
• Flat, meaning reads are spread evenly: build both paths and say so.

Read-heavy, over about 100 reads per write. The follow-up question is whether one key carries a large share of the reads.

• No: a shared cache plus read replicas.
• Yes: you are at a hot key, where partitioning is provably no help.

Why the chart has no colours

Every other diagram in this track uses chapter 01’s published key: blue is the authoritative copy of the data, green is read capacity, orange is a box forced by something other than processor time, and red is the one rung you cannot undo.

That key describes components. This chart has no components. Its boxes are a question, three more questions and six answers, so colouring them can only mislead: blue would land on the entry box, green on an append-only log that is itself the authority, orange on a cache, and red on a hot key, which is among the most undoable conditions in the book.

A decision chart is not an architecture diagram, and the key does not apply to it.

Five things the chart is deliberately saying

• The first box is the whole trick. Every branch below it is decided by a number, and a candidate who has not written the numbers down is guessing at every fork. The most common way to lose this round is to arrive at “we’ll shard it” without ever computing whether one machine would have done.

• The thresholds are ch 03’s, and they are asymmetric on purpose. Under about ten reads per write, the write path is the system; over about a hundred, caching stops being an optimization and becomes the design. The band between them is wide, and it is where most real problems land — including Worked the chart applied to a problem with no chapter’s ticketing example, at 12:1.

• In the middle band the ratio has stopped being informative and the access distribution takes over. Ch 08 sits at exactly 10:1 and still comes out cache-shaped, because its reads are Zipfian — a small set of links gets most of the clicks. Ten gigabytes of memory buys a 96.2% hit rate there, meaning h = 0.962, so only 3.8% of reads reach the database.

  A flat distribution at the same ratio buys nothing of the sort: a cache holding half the links would serve exactly half the reads, and more memory would not bend that line. Two systems with identical read:write ratios can want opposite architectures, and the tiebreak is a second number, not a stronger opinion.

• The write-heavy fork is the one candidates have never rehearsed, and it splits cleanly on a single question: is a write a newer value or a new fact?

  If a write supersedes the last one — a position, a status, a current price — then buffering it is actively wrong, because a buffer’s whole job is to preserve things you no longer want. Ch 18 shows a ten-deep queue delivering a location fix five TTLs past its own expiry date, and derives that the correct queue depth is one.

  If instead a write is a new fact that must be counted — a click, a payment, a trade — you are in ch 20 and ch 22. The write must then be an absolute-value upsert: store the running total (= 47), not the increment (+= 1), so that replaying the same message twice changes nothing.

• The read-heavy fork ends at the hot key on purpose. A hot key is the one place where adding machines is provably no help (ch 05), because one key hashes to one partition no matter how many partitions exist, and it recurs in five separate chapters (Seven failure shapes which is all of them).

The three gates the chart deliberately does not contain

The remaining three questions do not branch, so they are not in the chart. They are independent gates: checks that either fire or do not fire, in any of the six leaves above. A gate that fires changes the design regardless of which leaf you landed in.

Drawing them as further levels of a tree would make a flat checklist look like a decision procedure, when only the read:write node in that tree actually carries a number. Run them as a list, every time, in every problem.

Each gate below is a yes-or-no question, the number you check it against (with the chapter that derived it), and what changes in the design when the answer comes back yes.

4. Worked: the chart applied to a problem with no chapter

The claim in The twenty eight problems and what each one is actually for is that the rules transfer. This section tests that claim on concert ticketing, which the track never covers: Which pattern does this smell like’s fork and its three gates run end to end, so you can see what the procedure produces.

The problem, in one line: 50,000 people want to buy tickets to the same concert, and they all arrive at once.

Start where the framework says to start, with the arithmetic. The input is two lines of product description. The output is the six quantities below, and the last of them turns out to be the design.

assume  a 50,000-seat venue, 500,000 buyers arriving in the first 60 s,
        each refreshing the seat map every 5 s

seat-map reads/s at peak
  500,000 / 5                           =  100,000
purchase attempts/s, if every buyer tries once inside that minute
  500,000 / 60                          =  8,333
read : write
  100,000 / 8,333                       =  12
successful writes over the entire event, one per seat
  50,000
failed purchase attempts, since 500,000 buyers chase 50,000 seats
  500,000 - 50,000                      =  450,000
failure rate of the purchase call
  450,000 / 500,000                     =  0.9

Now walk the fork, and then the three gates, in that order.

• The read : write ratio is 12:1, which lands in the middle band — above ch 03’s floor of about 10:1 and nowhere near its ceiling of about 100:1. So the chart does not hand you an architecture. It hands you a second question: does a small hot set cover most of the reads?

  Here it does, absolutely. There is one seat map, and every one of the 500,000 buyers is reading it. One object at 100,000 reads/s is the most concentrated hot set in this book, so the map is served from cache and is stale by design — a seat you clicked may already be gone by the time you click it.

  The alternative is 100,000 linearizable reads a second — every read seeing the very latest write — against a table with 50,000 rows. That is an enormous amount of coordination bought to produce an answer that is obsolete a millisecond later anyway.

• But 12:1 also means the write path is not free. A single threshold would let you declare “the cache is the architecture” and stop; the middle band explicitly does not. At 12:1 you build both paths and say so. That matters here, because the write path is where the actual problem turns out to be.

• Gate 1, RAM: the working set is 50,000 seats. It fits in one process’s memory a thousand times over, so do not shard and do not spend the interview on partitioning. This is ch 01’s step 0 being right again.

• Gate 2, latency: buyers for one venue are regional, and the budget is human-scale rather than machine-scale, so the whole thing lives in one region. Put a CDN in front of the static seat-map image, and none at all in front of the write path, which cannot be cached by definition.

• Gate 3, two writers: absolutely. Two people click seat 14C at the same instant. But there is exactly one owner of that row, so the answer is a compare-and-swap on a seat version number — not a lock, and not a quorum (ch 15).

  Ch 23 is the same gate fired in a different domain, and it prices the alternatives. The free fix is to put the check inside the write itself — UPDATE ... WHERE version = 7 — so the condition is evaluated under the lock the write statement already holds, rather than under a second lock you took yourself.

The domain then overrides exactly one branch, and that is the part worth noticing. The chart’s action when two writers collide is “keep both versions”. Here you cannot: a seat is not a document that can be merged. So the losing writer gets an HTTP 409 Conflict and picks another seat.

The last line of the arithmetic is the actual design. Ninety percent of purchase calls must fail. The error path is not an edge case, it is the dominant response of the system — nine out of every ten times the endpoint is called, its job is to say no politely and tell the buyer what to do next.

That reframes the problem away from throughput and onto admission control: a virtual waiting room that lets buyers through at the rate the seat inventory can absorb. It is ch 04’s rate-limiter client contract applied to a product surface.

flowchart LR
    U["Buyer"] --> CDN["CDN: seat map<br/>cached, stale by design"]
    U --> WR{"Virtual waiting room<br/>admission control"}
    WR -->|"admitted at inventory rate"| P["Purchase<br/>compare-and-swap on seat version"]
    WR -->|"rate exceeded"| Q["429, wait"]
    P -->|"version matched"| OK["Seat booked"]
    P -->|"lost the race"| C["409 Conflict, pick another seat"]

One fork gave you a second question, three gates gave you the shape, one number gave you the actual problem, and the domain overrode exactly one branch. That is what the chart is for. It is not a decision procedure, it is a way to stop guessing.

5. The numbers you cannot recompute under pressure

What follows is a recall sheet of forty-odd derived numbers, each paired with the decision it makes. The derivations themselves are slow: every one of these costs a minute or two to rebuild on a whiteboard, and the interviewer stops learning anything new from watching you do so after the second time.

Primitive against derived, because they are memorized differently. A primitive is a hardware constant — how long a memory reference or a disk seek takes. Those live in Latency numbers and what each one forbids, with the full memorization table in Numbers worth memorizing cold. A handful appear below only because a later row depends on them.

Everything else here is derived: a number this track produced by combining primitives with an assumption. Derived numbers are the ones you cannot rebuild in your head inside a forty-five-minute conversation, which is why they are worth carrying.

The only column that matters in an interview is the last: what decision the number makes. A number that decides nothing is not on this sheet.

Volume 1

These are the numbers Volume 1 produces, in chapter order.

Volume 2

These are the numbers Volume 2 produces, again in chapter order.

The point of these tables is not recall, it is speed. Every one of these is a number you would otherwise spend ninety seconds deriving in a forty-five minute conversation, and the derivation adds nothing the interviewer wants once you have shown you can do it twice.

6. Seven failure shapes, which is all of them

Twenty-seven of the twenty-nine chapters carry a failure-modes table — every one except ch 02, which is an estimation drill, and this one — and across all of them the same seven shapes recur.

Learning the shape is worth more than memorizing the instances, because the shape tells you what to alert on. An instrument you did not decide to build is an instrument you do not have during the incident.

The instances are there so you can see the shape repeat across unrelated domains — skim them, do not memorize them. The tell is the observable signal that says this failure is happening right now, and the guard is what prevents it. In an interview you spend the tell and the guard; the instances are only evidence that the shape is real.

Three of these seven are invisible without an instrument you had to decide to build — silent data loss, hot keys, and drift — and those are the three an interviewer is most impressed to hear you name unprompted. All three share the same tell, which is that nothing is on fire: no error rate moves, no latency graph bends, and the only way to see them is a check you wrote before you needed it.

7. What this track does not cover

Eight subjects this track uses without teaching, where each gap will bite you, and what to read instead.

Being honest about the boundary is more useful than pretending there is not one. It is also the answer to the wrap-up question about what you are least sure of — an answer you can give in one sentence because you decided it in advance rather than on the spot.

8. Seven papers, and the problem each one actually solved

Seven foundational systems papers, each summarized to the level you would need to discuss it in an interview: what problem it solved, and what that solution costs.

Read the originals for the problem statement, not the implementation. Every one of them is a case of “we could not buy this, so we had to invent it,” and knowing what could not be bought is what makes the design legible. Each entry below is written in the same three moves — the problem, the mechanism, the price — because that is the shape of the answer an interviewer wants when they ask what a paper is about.

Dynamo (Amazon, 2007)

The problem. A shopping cart must accept a write during a network partition. A rejected “add to cart” is lost revenue, and a rejected checkout is worse.

The mechanism. Everything follows from refusing to ever say no to a write. Consistent hashing places data without asking a coordinator. Version vectors exist because concurrent conflicting writes are now normal rather than exceptional. Sloppy quorums and hinted handoff — accepting a write on whatever replicas can be reached, and having them hand it to the rightful owner once the network heals — mean a partition does not stop writes. Merkle trees then find and repair the divergence afterwards.

The price, and the part worth internalizing: Dynamo pushes reconciliation to the application. That is why the cart resurrects deleted items. The merge rule for a cart is “union,” and union has no delete.

Chapters 05 and 06 are this paper with the arithmetic filled in.

BigTable (Google, 2006)

The problem. Petabytes of structured data whose schema varied per row, at a write rate that made B-trees untenable.

The mechanism. A sparse, sorted, distributed map keyed by (row, column, timestamp). It is stored as immutable files called SSTables, with an in-memory buffer called a memtable in front of them absorbing writes, and a background compaction process behind them merging the files back together.

That arrangement — buffer in memory, write out whole sorted files, merge later, never update in place — is the log-structured merge tree, or LSM. It is why sequential writes beat random ones by three orders of magnitude (sql/03), and that same gap is the whole of ch 20’s 320x.

The price, and the consequence that decides designs: the row key is the only index, so the schema is the access pattern. You cannot add a query later without rewriting the data. Everything modern that calls itself a wide-column store is a descendant.

GFS (Google, 2003), and Colossus after it

The problem. Storing files larger than any single disk, on hardware that fails constantly, for a workload that appends far more than it overwrites.

The mechanism, and the design decision worth stealing: one master holding all metadata in memory, and dumb chunkservers holding all the bytes. That is exactly the metadata-versus-blob division chapters 15, 24 and 25 keep returning to.

The single master looks like a scandal until you notice two things: metadata is about a thousandth of the data, and the client talks to the master once per file, not once per byte.

Colossus is the sequel. It removed the single master and moved from whole copies to Reed-Solomon erasure coding; ch 25 is that arithmetic derived, including the parts the marketing skips.

MapReduce (Google, 2004)

The problem, stated correctly. The commonly told story is that MapReduce made parallelism easy. It did not — parallelism was already easy. It made fault tolerance easy, and that is the actual contribution.

The mechanism. If every task is deterministic and free of side effects, a task on a dead machine can simply be re-run somewhere else. Nothing needs to be undone, and no other task needs to know. A 1,000-machine job stops being a coordination nightmare and becomes bookkeeping.

That is the same determinism ch 29 buys at the opposite end of the timescale, and the same reason ch 22 can treat replay as routine. Spark and Flink changed the execution model and kept the insight exactly.

Kafka (LinkedIn, 2011)

The problem. Many consumers reading the same event stream at different speeds. A traditional queue cannot do this, because the broker tracks per-message delivery state for every consumer.

The mechanism: make the log the primary object and push the offset — the read position — out to the consumer. The broker then does sequential disk writes and nothing else. Replaying old messages is free, because nothing on the server had to be undone.

That is the same structural idea as the per-user cursor in ch 15: the reader owns its position, so the server holds no per-reader state, and both resume and replay come for free.

Ch 20 is this paper with every constant priced.

Raft (Ongaro and Ousterhout, 2014)

What consensus is, first. Consensus is the problem of getting a group of machines to agree on a single ordered sequence of decisions even while some of them fail. It is what “elect a leader” and “replicate this log” actually rest on.

The problem Raft addressed was not correctness. Paxos, the earlier protocol, was already correct. The problem was that almost nobody could implement Paxos without getting it wrong.

The mechanism is decomposition: leader election, log replication, and safety presented as three separable mechanisms, plus a membership-change protocol that Paxos papers left as an exercise.

The price. Read this paper before you say “we use consensus” in an interview, because the follow-up is what happens during an election. The answer is that writes stop for an election timeout — typically 150-300 ms. That is a real number, and you should be prepared to defend it against your latency budget.

Spanner (Google, 2012)

The problem. External consistency across datacenters: if transaction B commits after transaction A finished, then every observer anywhere must see B ordered after A. Not eventually — immediately and globally.

The mechanism is TrueTime, which stops pretending a clock reading is a point and treats it as an interval. Bound the clock uncertainty using GPS receivers and atomic clocks, then have each commit wait out that interval before releasing its locks. Waiting guarantees that the next transaction’s timestamp really is later.

The price, and the honest lesson: every commit waits roughly the clock uncertainty. Global strong consistency costs milliseconds per write, permanently, by design.

That is the counterweight to the entire eventual-consistency argument in chapter 06, and to the gap What this track does not cover admits this track never fills. Strong consistency across regions is purchasable, and the invoice is denominated in latency.

9. The other tracks in this repo

Four different design rounds share a shape and are scored on completely different things, so it matters which one you are actually in — reading the wrong framework an hour before the interview is a recoverable mistake only if you notice. Five things are listed below and only four of them are rounds: the fifth, database internals, is the substrate all four stand on and nobody interviews you on it under that name.

• ML system design — machine learning, meaning a system whose behaviour is learned from data rather than written down. It is the same skeleton as this track plus three extra stages that dominate the score: framing the learning objective, data and labels, and the gap between the metric you measured offline and the one that moved in production. The distinguishing failure is treating the model as the system — the interview is mostly about labels, features, and why the offline number improved while the online one did not. Ch 19’s ETA is this track handing a problem across that boundary and saying so out loud.
• GenAI system design — generative AI, where a model produces free-form text or images and there is therefore no single correct output, which makes evaluation the system rather than a stage of it. Cost scales with how much the model writes and superlinearly with how much context you give it, and generation is sequential — one token at a time — so the latency budget decomposes differently from anything in this track.
• Agent design — an agent being a model that calls tools in a loop to accomplish a task. This is a seven-step method for that round: clarify, tier, loop, tools, context, failure modes, then evaluations and cost. It is the closest in spirit to this track, and the phrases that signal seniority list transfers verbatim.
• Object-oriented design — a separate round with a separate rubric: clarifying questions, actors, a class diagram, the two or three decisions that imply a design pattern, then working code. Fourteen chapters cover it: what the round is, the framework, object-oriented programming fundamentals, and eleven worked problems from a parking lot to a restaurant. Do not walk into it with this chapter’s habits — nothing there is decided by an estimate.
• Database internals — the layer underneath everything here, and not a round of its own. If you cannot say why the query planner declined to use an index, or what a page is, then the storage answers in this track are recited rather than understood. Ch 23 is the chapter that spends it most directly: its whole deep dive is an argument about transaction isolation levels.

The framework chapters are interchangeable in structure and not in content. Read the framework for the round you are actually in, an hour before you are in it.

10. How to practice, in the order that works

Five practice steps, arranged so that each depends on the one before it. Doing them out of order mostly wastes the later ones.

1. Redo chapter 02’s six estimations on a timer, out loud, without notes, three minutes each. If the arithmetic is not automatic, nothing else in this track will surface under pressure, because every other skill here is downstream of a number you produced quickly.
2. Take one chapter and rebuild it from the requirements, without reading it. Chapter 08 is the right one to start with because the whole design turns on two numbers; chapter 15 is the right one to finish Volume 1 with because it turns on six. From Volume 2, rebuild chapter 23 — it is the one where the arithmetic’s entire job is to buy you permission to spend forty minutes on a race condition, which is a different skill from sizing a fleet.
3. Do a problem this track does not contain — a ticketing system, a ride-hailing dispatcher, a collaborative text editor — and use Which pattern does this smell like’s fork and its three gates to pick your two deep dives, exactly the way Worked the chart applied to a problem with no chapter does. Then check yourself against ch 03’s rubric. For each one, write down where the decisive constraint came from, and whether the chart got you there or the domain overrode a branch — that is the same move Worked the chart applied to a problem with no chapter makes at its last step.
4. Practice the wrap. Five minutes of “here is what I would build first, here is what I would measure, here is what I am least sure about” is worth more than a fourth deep dive, and it is the part almost nobody rehearses (ch 03). The last of those three clauses is what The assumption ledger is for.
5. Read one paper from Seven papers and the problem each one actually solved a week. Not to implement it, but to be able to say what problem it solved in two sentences.

11. The assumption ledger

Every design is a set of assumptions with a diagram attached, and the diagram is only correct relative to them. This chapter is no exception: its advice, its decision chart and its worked example all rest on things that could be otherwise. They are collected here, so that you can state the foundations in twenty seconds and say what replaces the advice when each one fails.

Sort each assumption into one of three bins:

• State it. You are free to pick, and being wrong costs a re-derivation and nothing more.
• Ask it. The answer moves a policy or a threshold, so it is worth an interviewer’s time.
• Load-bearing. If this is wrong the design is not suboptimal, it is invalid — a box appears or disappears, rather than the count inside a box changing.

The one-line test for which bin something goes in, from ch 03: move the assumption an order of magnitude in each direction and ask whether the set of boxes changes, or only the number of machines inside them.

The table below applies that test to the thirteen assumptions this chapter rests on; the last column — what you would build instead — is the one that turns an admission into an answer.

Why the venue size cannot move the answer

The 50,000-seat venue is the number to mark as explicitly unable to change the answer. It looks decisive: it appears in three of the six lines of Worked the chart applied to a problem with no chapter’s arithmetic, and it produces the memorable 90% failure rate.

It still cannot move the fork, and the reason is structural rather than numerical. The seat count appears in neither rate. Write the two rates out symbolically, with B for the buyer population:

reads/s         B / refresh interval        =  B / 5
writes/s        B / arrival window          =  B / 60
read : write    (B / 5) / (B / 60)          =  60 / 5     =  12

B cancels, and the seat count was never in the expression at all. The ratio is a function of the two time intervals and nothing else.

That is a stronger statement than it first looks. You can concede the venue size, and you can concede the crowd size too. The only two numbers that can move this problem out of the middle band are the refresh interval and the arrival window.

Test it: shrink the venue to 5,000 seats. The failure rate rises to 495,000 / 500,000 = 0.99, the read:write ratio stays at exactly 12:1, and every architectural conclusion holds.

What the seat count does decide is how loudly the error path dominates, and whether the inventory fits in memory. Fifty thousand rows clears the memory bar by a factor of thousands, and so would five thousand or five hundred thousand.

So concede the venue size instantly and spend the challenge on the arrival window, which is the one input that genuinely moves the design.

The sentence that makes this visible to an interviewer: “This whole approach rests on four things. One, that unfamiliar problems are recombinations of known mechanisms, which is what makes a rule table worth carrying. Two, that I have forty-five minutes and am being scored on reasoning, which is why I recall numbers instead of deriving them twice. Three, that the read:write ratio is the one question that dispatches, which is why my first move is always to compute it. Four, on this specific problem, that all the demand lands inside a minute — that is what makes it a write-path problem, and if it were spread over an hour I would be designing something else. The size of the venue is not one of them.”

Cheat sheet

This is the whole chapter compressed to twenty lines. If you read nothing else the morning of the interview, read this and Which pattern does this smell like.

Where to go from here: 03 — Interview Framework is how the forty-five minutes is spent, minute by minute; 02 — Back-Of-The-Envelope is the estimation drill you repeat until it is automatic; 17 — Proximity Service opens Volume 2 and is the next chapter in the study plan; and ML and GenAI are the same conversation scored against a different rubric.