01 — ML System Design Framework

Interview prompts of the form “design the system that ranks, detects, recommends, or scores X” have a repeatable answer: a thirteen-stage method, sized to fit a 45-minute interview.

The method is worked end to end on one concrete problem, the home feed of a short-video app. Every number in the finished design is derived from an earlier number rather than asserted.

After working through it you should be able to take any “design the system that predicts X” prompt and state, within the time budget:

• the model
• its inputs
• where its training labels come from, and when they arrive
• how it is trained
• how it is served
• how you would know it works
• which of its failures no offline metric can detect

You should also be able to justify every threshold and item count with arithmetic done in front of the interviewer.

Every term of art is defined where it first appears. Links to other chapters offer extra depth, not a substitute for a definition needed to keep reading.

What “an ML system” means here: the input, the output, and the five questions

Fix the vocabulary and the shape of the thing being designed first, so that everything after can be read literally.

Every system in this chapter fits one skeleton: something goes in, a model turns it into scores, and a policy turns the scores into one action. State it in the first minute of the round.

flowchart LR
    IN["Input<br/>request: who is asking, in what context, when<br/>candidate set: items that could be returned or acted on<br/>features: numbers on the user, item, and pair"] --> MODEL["Model<br/>a function that turns one<br/>(user, item, context) row into one score"]
    MODEL --> OUT["Output<br/>one score per candidate, which a policy converts<br/>into one action: a ranked slate of k items,<br/>a yes/no decision, or a number a human may override"]

Three terms in that skeleton carry the rest of the chapter, so they are defined once here and then used freely:

• A feature is one number, or a short list of numbers, computed about the user, the item, or the user-item pair and handed to the model as input. “How many videos this user finished this week” is a feature; so is “how many hours old this video is.”
• A label is the recorded answer the model is trained to reproduce — a value attached to a past row saying what actually happened, such as y = 1 if the user played the video, else 0. Labels are not handed to you. You manufacture them out of logged events, and choosing the rule that manufactures them is among the highest-leverage decisions in the round.
• An embedding is a fixed-length list of numbers, typically 32 to 256 of them, that stands in for a user or an item. It is learned so that things behaving alike land near each other, and “near” almost always means a large dot product — multiply the two lists element by element and add up the results.

Five questions your design must be able to answer at every stage. They are the reason the thirteen stages are ordered the way they are, because each answer is an input to the next. The right-hand column of the table says which stage of the chapter answers each one, so you can use it as a map:

A design that cannot answer all five is incomplete. The most common way to fail this round is to answer question 1 well and never reach the other four, because the clock ran out while choosing an architecture.

Which round this is, and what it is scored on

There are three design rounds that look alike from the outside, and this is the middle one.

It is not the generative-AI (“GenAI”) design round — the one where the system is built around a large language model, a model that produces free-form text and therefore has no single correct output to compare against. It is not the distributed-systems round either, although it borrows a diagram from each.

In GenAI design you win on evaluation, because there is no label (genai 01). In classic ML design you win on the label — where it comes from, when it arrives, and whether it is the thing you actually care about.

Everything downstream is a consequence of that answer: features, model, metric, serving. Candidates who skip it spend thirty minutes designing a system that optimizes the wrong quantity very efficiently.

Where the deeper derivations live

This chapter is the spine that the problem chapters 02-11 apply. Some mechanisms have a longer derivation elsewhere in this repository. Where that happens, this chapter states the idea in a sentence or two and then links, so you never have to leave the page to follow the argument:

• evaluation metrics — ml 06
• features and train/serve mismatch — ml 01
• rare-class handling and drift — ml 07
• round-agnostic material (how to narrate your thinking, how to handle “I don’t know,” how to timebox) — ch 10
• the distributed-systems round whose shape this one borrows — system-design 03

One problem, carried all the way through

Every stage below is run on the same problem. The chapter states the prompt before Stage 1 and carries its numbers through all thirteen stages, each one feeding the next:

18 million daily active users → 5,000 requests per second at peak → a 20-million-item catalog → 1,000 candidates per request → 16 machines → the one line of the cost table that turns out to dominate.

A few individual traps borrow a different domain, because the trap is clearest there: a feature that peeks at the future (subscription churn), a training set that only contains survivors (lending), a label that arrives months late (card fraud). Each one says which domain it has borrowed and then returns to the video feed.

The spine

With the vocabulary fixed, here is the method itself: thirteen stages, the minutes each one gets, and three places where the ordering is counter-intuitive. The pipeline below is its shape — each box consumes the box above it, the order is fixed, and three of the boxes are shaded.

flowchart TD
    F["1 Problem framing<br/>5 questions, one of them<br/>about the label"] --> O["2 ML objective<br/>business goal to learnable<br/>target, then the operating point"]
    O --> D["3 Data and labels<br/>and what they cost"]
    D --> FE["4 Features"]
    FE --> M["5 Candidate generation<br/>then the ranking model"]
    M --> TR["6 Training"]
    TR --> OFF["7 Offline metrics"]
    OFF --> ON["8 Online metrics<br/>and the A/B"]
    ON --> S["9 Serving<br/>batch · online · streaming"]
    S --> C["10 Scale and cost"]
    C --> FM["11 Failure modes"]
    FM --> A["12 Alternatives rejected"]
    A --> P["13 Pushback"]

    style O fill:#1d3557,color:#fff
    style M fill:#1d3557,color:#fff
    style FM fill:#1d3557,color:#fff

The three shaded boxes are exactly the three named in Three sections that look optional and are not below — the ML objective, candidate generation, and the failure table. The other ten boxes are peers in a pipeline, and how much each deserves is a question the minute table two sections down answers properly.

What each box actually is

Walk the boxes once, because several of them are not what their names suggest.

1. Problem framing — five questions, and one of them is about the label: when the recorded answer you train on actually arrives.
2. ML objective — turns a business goal into a learnable target, then fixes the operating point, meaning the rule that converts a continuous score into the one action the product takes.
3. Data and labels and what they cost — prices the labels in dollars, because a labeling plan without a number cannot be evaluated.
4. Features — sketches the input vector.
5. Candidate generation, then the ranking model — narrows the catalog to a shortlist first, and only then scores that shortlist with the expensive model.
6. Training — the split, the negative sampling, and the refit cadence.
7. Offline metrics — the gate you can compute on logged data, before you ship anything.
8. Online metrics and the A/B — the live randomized experiment. “A/B” means splitting traffic into an A arm and a B arm and comparing them. This is the only thing that decides a launch.
9. Serving — picks the delivery mode: batch predictions computed ahead of time on a schedule, online predictions computed inside the request, or predictions computed from a stream of events.
10. Scale and cost — turns the design into machines and dollars.
11. Failure modes, 12. Alternatives rejected, 13. Pushback — where seniority is visible.

The clock

Here is the 45 minutes, allocated the way the round actually scores. The Clock column is the one to memorise: minutes without a clock cannot tell you whether you are behind.

5 + 8 + 8 + 5 + 8 + 7 + 4 = 45

Note the shape: objective and metrics together are 16 of the 45 minutes, and modelling is 5. Candidates invert that ratio and lose the round while talking confidently about architectures.

Four minutes for the last three stages is the tightest block in the chapter, and it is deliberate. It is also less than genai 01 gives the same three stages.

The reason is that this round front-loads its senior signal. Three places earlier in the round are already asking you to show judgement: the label question in Stage 1, the proxy inversion in Stage 2a, and the operating point in Stage 2e. The GenAI round has no equivalent of any of them, so it has to spend its seniority budget at the end.

If you find yourself at minute 41 with nothing said that a mid-level candidate would not have said, the four minutes will not rescue you.

Three sections that look optional and are not

Three stages sit somewhere a first reading of the pipeline would not put them, and each of the ten problem chapters performs all three. They are worth flagging before you meet them, because in each case the surprise is that the work happens earlier than expected:

All three sit earlier, or closer to the objective, than the pipeline picture suggests, and that is the point of naming them. In one line each:

• Candidate generation is not a modelling detail. It is where the inventory constraint lives.
• The operating point is not a metrics detail. It is half of the target definition.
• The failure table’s fourth column is Control rather than Guard because what you owe the interviewer is the knob you would turn, not the fence you would build.

How this differs from the two rounds it resembles

Thirty seconds into a round you should know which of the three you are in, so you stop reaching for the wrong toolkit. The side-by-side below is how to tell.

Two terms in the table are worth glossing first:

• proxy — an observable stand-in for the thing you actually care about, such as using a click to stand in for satisfaction.
• FLOP — one floating-point operation, the unit compute cost is counted in.

The middle column is your round. Read down it first, then read across the rows to see what the other two rounds would have said instead.

The single reframe to say out loud in the first two minutes: “Before I pick a model I need to know what decision this prediction drives, what a mistake costs in each direction, and whether a label exists — because if the label arrives 90 days late, that changes the training set, the metric, and the retraining cadence, and I would rather find out now.”

The running problem

The running problem: design the home feed for a short-video app.

Every stage from here on is worked against this one problem. Its constants are fixed once, below, and the finished system is assembled in a single box before any of it is derived.

The numbers chain forward. The answers in Stage 1 size Stage 5. What Stage 5 picks sizes Stage 10. And the dominant line of Stage 10’s cost table points back at Stage 3.

Restate the problem first, with the exclusions, before you ask anything. This takes two sentences and it is the cheapest way to discover you are about to design the wrong system:

“So: the ranked list a user sees when they open the app — not search, not notifications, not the creator-side tooling. I am assuming the catalog is on the order of tens of millions of videos and that this is a personalization problem rather than a cold-start-a-new-product problem. I am going to spend most of my time on what we are actually optimizing and on the candidate path, because I think those are where this system is won. Tell me if that is the wrong emphasis.”

The constants

Now fix the constants that everything downstream divides by, stated once so they are never re-derived on the clock. Three abbreviations appear in the block and are used for the rest of the chapter:

• DAU — daily active users, the count of distinct people who open the app on a given day.
• p99 — the 99th-percentile latency, the number that only 1 request in 100 exceeds. It is the honest budget, because averages hide the requests users complain about.
• diurnal — the day-night traffic cycle. Peak here is assumed to be twice the daily average.

The top half of the block below is assumed; the bottom half is arithmetic on the top half. 86,400 is the number of seconds in a day (60 × 60 × 24).

DAU                                            18,000,000
feed requests per user per day                         12
items rendered per response                            10
catalog                                        20,000,000
p99 budget, whole ranking service                  100 ms

requests/day       18e6 × 12                = 216,000,000
requests/s avg     216e6 / 86,400           =       2,500
requests/s peak    2,500 × 2 (diurnal)      =       5,000     <- the sizing number
impressions/day    216e6 × 10               =   2.16e9

Two numbers do all the work below: 5,000 requests per second — “QPS,” queries per second, from here on — and a catalog of 20 million videos.

The first sizes the machine fleet in Stage 10. The second forces the two-stage retrieve-then-rank pattern in Stage 5, and it forces it by a factor of 8,000, which is not a close call.

The 2.16 billion impressions per day is the third — an impression being one item shown to one user once. It changes the experiment design in Stage 8 in a way most candidates get backwards.

The finished system in one box, with its assumptions marked

Here is where the thirteen stages are going, stated as the five questions from the top of the chapter. Each line is derived in the stage its parenthetical names.

The box describes two models running one after the other, and each model gets five lines — inputs, label, trained, served, judged — which are the five questions from the top of the chapter in the same order. The parenthesised stage number at the end of a line is where that line is derived. Everything marked [assumed] is a number or a choice you would state as an assumption in the room and invite the interviewer to overturn.

Several terms in the box are not defined yet. That is on purpose — you are meant to see the shape now and collect the definitions as the chapter reaches them.

THE SYSTEM: two models in series, plus a policy layer

MODEL 1  two-tower retriever.  Two small neural networks, one reading the
         user, one reading the video, each emitting a 64-number embedding;
         the score of a pair is the dot product of the two embeddings.
         WHY this family: only a score that splits into f(user)·g(video)
         lets you precompute every video's side and index it  (Stage 5)
  inputs user side: long-term topic and creator affinities
         video side: topic, creator, duration, quality  (Stage 4)
  label  (user, video) pairs the user actually engaged with = positive;
         other videos sampled from the catalog = negative  (Stage 6)
  trained sampled softmax with in-batch negatives and a logQ correction;
         index rebuilt hourly  (Stage 6)
  served ANN index, ~1.2 ms per query, 20M -> 1,000 candidates  (Stage 5)
  judged recall@1,000 against the ranker's own exhaustive top-10,
         reported per item-frequency decile, never in aggregate  (Stage 7)

MODEL 2  ranking model.  A 4-layer neural network scoring one
         (user, video, context) row at a time, ~40 microseconds each.
         WHY this family: it must NOT factorize -- the whole value of the
         expensive stage is the user x video interaction terms  (Stage 5)
  inputs 200 features: 140 video-side, 40 user-side, 20 context and
         cross  (Stage 4)
  label  a COMPOSITE: y = 0.2·click + 0.5·completed + 0.3·no-skip-away,
         weights fit on a bought satisfaction sample  (Stage 2a, Stage 3)
  trained temporal split, refit weekly, floor of 1 h from label
         arrival  (Stage 6)
  served online, 1,000 candidates per request, 16 machines  (Stage 9, 10)
  judged NDCG@10 offline as the gate, play-through rate in a live A/B
         test as the decision, worst slice not the mean  (Stage 7, 8)

POLICY   slate of 10 = 7 ranked + 2 diversity + 1 exploration  (Stage 2e)

[assumed] 18M DAU · 12 requests/user/day · 10 slots · 20M catalog ·
          100 ms p99 · peak = 2x average · $0.35/machine-hour
[assumed] clicks are a corrupted proxy for satisfaction, so a composite
          label is required -- Stage 2a turns this into a measurement

Two abbreviations in the box get their full treatment later, and both are worth having now so the box reads cleanly. ANN is approximate nearest neighbour: an index that finds almost the closest embeddings without ever comparing against all 20 million. NDCG is normalized discounted cumulative gain: a ranking score that rewards putting good items near the top of the list, discounting them the further down they sit.

Notice what the box does not contain: a claim that any of this is right. Two numbers in it — the composite label’s weights and the candidate count of 1,000 — are the two the interviewer is most likely to attack, and both are derived rather than chosen, in Stage 2a and Stage 5 respectively. That is the difference between a design you can defend and a design you can recite.

Stage 1 — Framing: five questions, and each one moves the design

The round opens with five questions, and each answer moves the design.

The test for a good framing question is not that it sounds thorough. It is that you spend the answer within a minute. “How many daily active users are there?” fails that test, because you will not use the number for twenty minutes. Each of the five below changes your very next sentence.

Terms used in the table

Four terms appear throughout the table and are worth having before you read it.

• Precision — of everything the model said yes to, how much of it was right. The share of the rows you flagged that really were positive.

• Recall — of everything that was really positive, how much you caught. The share of the truly positive rows that you flagged.

  These two trade against each other. Lower the threshold and you catch more of the positives, and you are right about a smaller fraction of what you flagged. So “recall 0.9955” in the table means 99.55% of the faces in the imagery were found, and 4.5 in 1,000 were not.

• Calibrated — a model is calibrated when its stated probabilities are true frequencies. Of all the rows it scores 0.30, close to 30% actually turn out positive. This matters when something downstream multiplies the number by a dollar value. It does not matter at all when you only need the items in the right order.

• CTR — click-through rate, clicks divided by impressions.

The five questions

Each row of the table gives one question, one possible answer, and the design consequence of that answer. Read it as: if the interviewer says this, my next sentence is that.

Question 5 in detail: when does the label arrive?

Question 5 is the one candidates skip and it is the one that reorganizes the design.

The diagram below is the whole taxonomy of answers, and each of the four is a different system — not a different setting of the same system. The arrow labels are the answers; the box contents are what each answer forces you to build.

flowchart TD
    Q{"When does the label arrive?"}
    Q -->|"seconds · click, play, dismiss"| A["Implicit feedback<br/>huge volume, heavy bias<br/>-> position-bias correction<br/>-> retrain daily"]
    Q -->|"days to months · chargeback,<br/>churn, default, return"| B["Censored window<br/>-> maturity-weighted labels<br/>-> retrain on matured data only<br/>-> calibrate for the lag"]
    Q -->|"only when a human labels it"| C["Labeling is a budget line<br/>-> active learning<br/>-> weak supervision to bootstrap"]
    Q -->|"only when the system acts"| D["Selection bias by construction<br/>-> exploration traffic<br/>-> propensity logging is mandatory"]

    style Q fill:#1d3557,color:#fff
    style A fill:#2d6a4f,color:#fff
    style B fill:#2d6a4f,color:#fff
    style C fill:#2d6a4f,color:#fff
    style D fill:#2d6a4f,color:#fff

The four answers are peers — four different systems, none worse than another, and the choice between them is made by your product rather than by you. Selection bias is not more dangerous than a censored window; it is a different problem with a different fix.

Walk the four branches one at a time.

Branch 1 — labels arriving in seconds: click, play, dismiss.

This is implicit feedback, meaning the label is inferred from ordinary product behaviour rather than asked for. Volume is enormous, and every row is contaminated by where on the screen the item was shown.

Two design consequences. You need a position-bias correction, derived in Stage 3. And your retrain cadence is measured in days rather than months, because nothing stops you.

Branch 2 — labels arriving in days to months: chargeback, churn, default, return.

Three of those words are domain jargon. A chargeback is a card payment the bank reverses after a customer disputes it. Churn is a subscriber cancelling. Default is a borrower failing to repay. All of them land long after the prediction was made.

The recent window is therefore a censored window: a row that has not gone positive yet is not a negative, it is a row whose verdict has not arrived.

Two fixes, and then one thing you must do either way:

1. Maturity-weighted labels — discount a young row by how much of its verdict window has elapsed. Or,
2. Retrain on matured data only — older data, but honest.
3. Either way, calibrate for the lag, because the observed positive rate is a fraction of the true one (Stage 2d).

Branch 3 — labels arriving only when a human labels them.

Labeling is now a budget line rather than a data-engineering task. Active learning stops being a refinement and becomes the thing that decides whether the project is affordable at all (Stage 3).

Weak supervision — combining several cheap imperfect rules into one probabilistic label — is how you bootstrap before the budget exists (Stage 3).

Branch 4 — labels arriving only when the system acts.

The system observes outcomes exclusively on the rows it chose to act on. So selection bias is a property of the design, not an accident of the data.

Both fixes are structural. First, an exploration slice that acts regardless of score. Second, propensity logging — a propensity being the probability that the policy in force would have chosen this row, written down next to the row at the moment of acting.

Without that propensity number, no later estimator can undo the selection, and the number cannot be reconstructed after the fact.

What interviewers probe: whether your questions are load-bearing. “How many users?” is a question whose answer you will not use for twenty minutes. “When does the label arrive?” changes your next sentence. Ask questions whose answers you spend immediately.

The five questions, answered on the running problem

Here are the five answers for the short-video home feed, which is what the rest of the chapter is built on. Give each one in about ninety seconds, and say not only what the answer is but what it costs you — what it buys and what it takes off the table:

Question 5 is the one that reorganizes this design, and here is the reorganization: the label is not one thing with one arrival time. It is four things with four arrival times.

What each of the four is for:

• Play-start, arriving in milliseconds — the fast term, and the one the clickbait trap in Stage 2a corrupts.
• Completion, arriving after one video length (median 45 s) and skip-away, arriving when the session closes (~1 h) — the two terms that make the fast one honest.
• Survey satisfaction, weekly and only on a sample — far too sparse to train on. Its job is to set the weights on the other three, and it doubles as the evaluation set.

That structure — a fast biased signal, a slower honest one, a sparse true one — is the shape of the label in most consumer ranking problems. Saying it in Stage 1 buys you the whole of Stage 2.

Where the problem chapters actually answer question 5

In a finished design, the label-arrival answer almost never shows up in the framing section that asked it.

Across the ten problem chapters, question 5 is asked once and then spent in three different places. Knowing which three tells you what to look for when you read one. The middle column of the table is where to look:

So the answer to question 5 gets spent in three places: the training window, the calibration check, and the failure table. That is why a chapter organised by system component scatters it rather than opening with it.

Ask it at minute two anyway. An answer you spend in three places later is worth more than one you spend immediately, not less — and it is the cheapest way to find out that your prediction target does not exist at all.

Stage 2 — Framing the ML objective

Framing done, the goal itself has to be translated: a sentence a product manager would say must become a quantity a model can minimize, and that translation silently breaks in four ways.

This translation is the skill the round most tests. A business goal is not a loss function, and a wrong translation stays invisible for six months.

Two definitions before the diagram. A loss function is the number training tries to make small. Binary cross-entropy is the standard loss for yes/no outcomes: it is small exactly when the predicted probability is close to 1 on rows where the thing happened, and close to 0 on rows where it did not.

The block below is that translation, drawn as four levels with a decision on each arrow. Read it top to bottom — English at the top, a number a computer can minimize at the bottom.

business goal      "a session should be worth the time the user spent in it"
        |
        v  choose an observable
proxy target       "user plays the video"
        |
        v  choose a label rule
learnable label    y = 1 if a play started within the impression
        |
        v  choose a loss
objective          binary cross-entropy on (impression, play)

Each arrow is a decision, and each one can silently break the chain. The five sections below are the four ways it breaks plus the fix that ends the stage:

• 2a — the proxy is not the goal
• 2b — the feature saw the future
• 2c — the training set contains only survivors
• 2d — the label arrives too late
• 2e — fixing the operating point

2a. Proxy-metric mismatch, derived

The first break: a proxy that is right on average can still be wrong on most pairs. Exactly how often is derivable — and so is the fix.

The setup

Suppose you want satisfaction s and you can only observe clicks. Write the click probability for item i as

P(click | i)  =  a_i  +  b · s_i

Two symbols, and they are the whole model:

• b — how much satisfaction actually drives the click. A dial between “clicks are pure satisfaction” and “clicks are noise.”
• a_i — the item’s intrinsic clickiness, the part of its click rate that has nothing to do with whether the item is any good: thumbnail, title framing, novelty.

The naive argument that “clicks track satisfaction on average” is true and irrelevant, because the ranker does not average, it sorts.

Item B outranks item A whenever B’s click probability is higher. Write that out and move the terms around:

a_B + b·s_B  >  a_A + b·s_A        <=>        a_B - a_A  >  b · (s_A - s_B)

So a pair inverts — comes out in the wrong order — whenever the gap in clickiness exceeds b times the gap in satisfaction.

There is no threshold at which that stops happening. Inversions only get rarer, as b · sd(s) grows relative to sd(a). Here sd(·) is the standard deviation, the usual measure of how widely a quantity is spread across items.

One pair of items, worked

Take b = 0.25 and two videos. Read the P(click) column as a + b·s computed on that row, and then read the Ranked column, which is the order a click-trained ranker puts them in.

Substituting, one row at a time:

A:  a + b·s  =  0.10 + 0.25 × 0.80  =  0.10 + 0.20  =  0.300
B:  a + b·s  =  0.28 + 0.25 × 0.20  =  0.28 + 0.05  =  0.330

A is four times as satisfying and ranks second, and nothing about the model is broken. The model estimates P(click) correctly. The objective was the mistake.

How often, exactly

“Inversions get rarer” deserves an exact percentage, because a rule of thumb you cannot price is not a rule you can defend.

If the algebra below is not your thing, skip to the k table — the formula it produces is P(inversion) = arctan(k)/pi, and the table is what you would actually quote in a room.

Step 1 — define one ratio and one unit.

Take a random pair of items whose clickiness a and satisfaction s are drawn independently from bell curves. Define

k  =  sd(a) / (b · sd(s))

Read k as: the spread of clickiness, measured in units of the spread of satisfaction-driven clicks. Small k means satisfaction dominates and clicks are a fine proxy. Large k means clickiness dominates.

Normalise sd(s) = 1 throughout. This costs nothing. k is a ratio, and the inversion rate depends on the two spreads only through k, so fixing the unit of satisfaction fixes nothing else.

Step 2 — write down the two gaps between the pair.

ds  =  s_A - s_B                 the satisfaction gap
da  =  (a_A - a_B) / b           the clickiness gap, IN SATISFACTION UNITS

da is divided by b on purpose, and the derivation is wrong without it. Here is why. The click score is a + b·s, so the raw click-order gap between the two items is

(a_A - a_B)  +  b·(s_A - s_B)   =   b·da  +  b·ds   =   b·(ds + da)

A positive b cannot change a sign, so the click order is just the sign of ds + da — but only once the clickiness gap has been rescaled into satisfaction units this way.

Read da as the raw gap a_A - a_B and the whole result acquires a spurious dependence on b: it lands at 17% rather than 9.3% when b = 2, and at 4.7% when b = 0.5. The rescaled definition is what makes the printed formula below true for every b.

Step 3 — say what an inversion is, in those terms.

An inversion is sign(ds) ≠ sign(ds + da): the pair’s true order and its click order disagree.

So the question is now about a pair of quantities, (ds, ds + da), and how often they disagree in sign. That pair is a bivariate normal — two bell-curve quantities whose joint behaviour is fully described by how strongly they move together.

Step 4 — get the spreads.

Both gaps are differences of two independent draws, so each has twice the variance of a single item. Var(·) is the variance, which is just sd(·) squared. Work in variances rather than standard deviations here, because variances of independent quantities add and standard deviations do not.

Var(ds)  =  2 · sd(s)^2               =  2
Var(da)  =  2 · sd(a)^2 / b^2         =  2 · k^2 · sd(s)^2  =  2k^2

Step 5 — get how strongly the two move together.

That is the correlation rho. Since da is independent of ds, the covariance of ds with ds + da is just Var(ds):

rho  =  Var(ds) / sqrt( Var(ds) · Var(ds + da) )
     =  2 / sqrt( 2 · (2 + 2k^2) )
     =  2 / sqrt( 4 + 4k^2 )
     =  2 / ( 2 · sqrt(1 + k^2) )
     =  1 / sqrt(1 + k^2)

Step 6 — turn the correlation into a probability.

For a mean-zero bivariate normal, the probability that the two coordinates disagree in sign is arccos(rho)/pi. And arccos(1/sqrt(1+k^2)) simplifies to arctan(k). So:

P(inversion)  =  arctan(k) / pi          k = sd(a) / (b · sd(s))

What the formula says at real values of k

One row per plausible value of k. The middle column is arctan(k)/pi evaluated at that k.

The k = 0.3 row, substituted: arctan(0.3) = 0.2915 radians, and 0.2915 / 3.1416 = 0.0928, so 9.3%.

arctan(k)/pi is zero only at k = 0, which is the formal version of “there is no threshold at which that stops happening.”

So the common sd(a) < 0.3 · b · sd(s) rule of thumb is not a safety line. It is a budget. It says you have decided that nine per cent of pairs inverting is a price worth paying to avoid building a composite label. Say the 9% out loud when you invoke the rule, because an interviewer who knows the formula is listening for whether you know what you just accepted.

How you would measure k on a real system

One thing here is easy to skip and is the whole difficulty: a_i is a per-item latent — a quantity that is never directly observed and can only be inferred from repeated observations of the same item. So you cannot get sd(a) from a single regression of click on satisfaction. (A regression here just means fitting a straight line through points and reading off its slope.) You need repeated impressions of the same item.

The estimator is two steps.

1. Get b. Regress each item’s observed click rate on its satisfaction score, across a satisfaction-labeled sample. The slope of that line is b.
2. Get sd(a). Take how much the points scatter between items around that line. Subtract the scatter you would expect from coin-flip noise alone, given each item’s impression count. What is left is var(a), and its square root is sd(a).

Drop items with fewer than a few hundred impressions rather than shrinking them. The quantity you want is a between-item spread, and low-impression items contribute almost none of it while contributing most of the noise.

The fix, run back through the same two items

Now the repair, re-scored on the exact same two items — because a fix you do not re-score is a fix you have not shown.

Fix a proxy gap with a composite label, not with a better click model.

A composite label is a weighted sum of several observed signals used as one training target. For example:

y  =  0.2 · click  +  0.5 · (completed the video)  +  0.3 · (no immediate skip-away)

Why this works, in terms of the two symbols from Step 1: it moves b up and sd(a) down. Clickbait — content engineered to be clicked rather than to be worth clicking — is negatively correlated with the completion term, so the terms that clickbait wins and the terms it loses now sit in the same target.

Run the same two items through it. The completion and skip-away rates are the kind a 12-minute explainer and a rage-bait post actually produce.

Term by term, so nothing is hidden:

A:  0.2 × 0.300 = 0.060      0.5 × 0.86 = 0.430      0.3 × 0.91 = 0.273    ->  0.763
B:  0.2 × 0.330 = 0.066      0.5 × 0.11 = 0.055      0.3 × 0.34 = 0.102    ->  0.223

A now outranks B by 3.4x — 0.763 / 0.223 = 3.42 — having lost the click term by exactly the margin it lost by before.

The click column is unchanged. B still wins it, 0.330 to 0.300. It no longer matters, because it carries 0.2 of the weight against 0.8 carried by two terms on which B is catastrophic.

The completion term did most of the work:

completion's contribution to the gap  =  0.5 × (0.86 − 0.11)  =  0.375
the whole gap                         =  0.763 − 0.223        =  0.540
                                         0.375 / 0.540        =  69%

That 69% is what “clickbait is negatively correlated with the completion term” looks like as a number, and it is why the weights matter more than the model does.

Where the weights come from

The weights come from a satisfaction-labeled sample, not from taste. Fit them by regressing surveyed satisfaction on the three observed signals, then rescale the resulting coefficients so they sum to 1.

On the running problem that sample is the 20,000 items a month priced in Stage 3 and paid for in Stage 10. This is the first place in the chapter where two stages meet.

The mismatch compounds

The damage does not merely persist, it grows. Ranking by P(click) shows high-clickiness items more often. That generates more training data for high-clickiness items. That sharpens the model’s estimate of their clickiness and lifts them further.

A proxy mismatch is not a static bias. It is a positive feedback loop (ml 07).

2b. Label leakage, with a trace

The second break is a feature that peeked at the future. The way to learn the smell of one is to follow a real bug from the offline score all the way down to the single row that reveals it.

Leakage is any feature that encodes information not available at the moment the prediction has to be made. It always looks like a great model, which is why it survives review.

The trap is clearest on a subscription-churn problem — predicting which subscribers will cancel — so this section leaves the video feed and comes back at the end.

Terms in the trace

• AUC — area under the receiver-operating-characteristic curve. Plain reading: pick one positive row and one negative row at random; AUC is the probability the model scores the positive one higher. A coin flip is 0.5, a perfect ranker is 1.0.
• Gain — a feature-importance number from a tree model: the share of the model’s total improvement that splitting on that feature accounted for.
• Shadow — a deployment stage where the new model scores 100% of live traffic and serves none of it. Its outputs are logged and compared, never shown to a user (Stage 9).
• JOIN ... AS OF — the database operation that attaches to each row the feature values as they stood at that row’s own timestamp, rather than as they stand now.

The trace

Read the block top to bottom. It opens with the task and the suspicious feature, then shows the two numbers that do not agree (offline 0.941, shadow 0.712), then walks the pipeline until it lands on one user whose feature value is different depending on when you compute it.

TASK    predict churn in the next 30 days, scored on the 1st of each month
FEATURE support_contacts_30d   (count of support tickets, last 30 days)

offline  AUC 0.941   -- by far the strongest feature, gain 0.38
shadow   AUC 0.712

TRACE
  the feature table is rebuilt nightly and keeps only the latest row per
      user, holding
      support_contacts_30d = COUNT(ticket WHERE created_at
                                   BETWEEN load_date - 30d AND load_date)

  the March cohort (t = 2024-03-01) was assembled on 2024-03-31 and joined
      its labels to that snapshot with no time predicate:
      JOIN features ON features.user_id = label.user_id     <- no AS OF
  so every row's 30-day window is [2024-03-01, 2024-03-31]
      which is exactly the LABEL window, not the 30 days before t

  the churn label was "cancelled between t and t+30d", and 61% of
      cancellers open a "how do I cancel" ticket in the 72 h before
      they cancel -- which is INSIDE [t, t+30d]

  one row, both ways:
      user 8814, t = 2024-03-01, cancelled 2024-03-20
      as of t          tickets in [2024-01-31, 2024-03-01]  =  0
      as of load_date  tickets in [2024-03-01, 2024-03-31]  =  3
                       (all three opened 2024-03-17..19)

The bug is not in the window, it is in what the window is relative to.

BETWEEN load_date - 30d AND load_date is a perfectly correct 30-day window. It is simply anchored to the day the pipeline ran rather than to the day the prediction was made. Here those two anchors are 30 days apart.

So the feature’s window and the label’s window are the same 30 days. The feature is not merely contaminated by the future — it is a count of events drawn entirely from it.

At serving time the anchor and the prediction time coincide, so the feature collapses back to its honest value (0, for user 8814), and the model loses 0.23 of AUC: 0.941 − 0.712 = 0.229.

The tell is the gap between offline and shadow, not anything visible in the offline run. Nothing about 0.941 is suspicious on its own — that is exactly what a genuinely strong feature looks like. Shadow is the first moment features are read the way production reads them, and therefore the first moment this class of bug can appear at all.

Three structural defenses stop it, in descending order of value:

One subtler form is worth naming unprompted, because it does not look like a time bug at all: the label is in the feature by policy.

Suppose the current rules engine declines every transaction above a risk score. Then was_declined is a perfect predictor of fraud in the training data — not because it saw the future, but because policy made the two the same thing. At serving time it is unavailable. Or worse, it is your own model’s output, so the model is being fed its own answer.

2c. Survivorship bias in the training set

The third break: the training set contains only the rows some earlier policy let through — and the fix has a price you can compute.

The classic shape is a loan-default model trained only on loans that were approved. It learns

P(default | applicant, AND this applicant was approved)

and is then deployed to answer a different question:

P(default | applicant)          for everyone, including applicants it has never seen

The block below puts numbers on the gap. Notice the third line: labels exist only for the 340,000 approved rows.

applications        1,000,000
approved (policy)     340,000     <- the ONLY rows with a label
defaulted              23,800     observed default rate 7.0%

the 660,000 rejected applications have no label, and the rejection rule
was itself a function of risk -- so the region of feature space where
default is most likely is the region the model has never seen

The consequence is not “slightly worse on the tail.”

The model is asked to score applicants in a region where it has literally no training examples. And a gradient-boosted decision tree (GBDT) — an ensemble of small trees, the workhorse model for tabular data — answers such a question by holding the last leaf value constant.

Here is why. A tree cuts the space of applicants into boxes and stores one number per box. Outside the range it saw, the outermost box simply extends forever. So the score goes flat exactly where you need it to be sharpest. (ml 02 shows why distance-based models fail in the same region for a different reason.)

Three fixes, in ascending order of cost:

• Reject inference. Guess labels for the rejected applications using a model fit on the accepted ones, weighting each guess by the inverse of how likely that applicant was to be accepted. It is cheap, and it is circular: it can only interpolate what the accepted population already told you, so it cannot discover anything about the region that was never observed.
• Exploration budget. Approve a small random sample regardless of score. 0.5% of 1M applications is 5,000 unbiased rows per cycle, and it is the only source of genuinely uncensored data. Pricing it correctly takes four steps and is worked below.
• Log the propensity. Whatever policy decides who gets acted on, write down next to each row the probability that policy would have chosen it. Without that number, no amount of later cleverness recovers an unbiased estimate, and the number cannot be reconstructed afterwards.

Pricing the exploration budget, step by step

Two mistakes are available here, and both make the number look better or worse than it is. Do it in four steps.

Step 1 — count the incremental approvals, not the gross ones. Policy already approves 34% of applicants anyway, so a third of your random sample would have been approved regardless. Only the rest is a change:

5,000 × 0.66  =  3,300 approvals that would not otherwise have happened

Step 2 — price those 3,300 at the rejected population’s default rate, not the portfolio’s. They are exactly the applicants policy declines, so by construction they are the risky tail. You have a pooled random-approval default rate of 22%; back the rejected rate out of it:

pooled  =  0.34 · (approved rate)  +  0.66 · (rejected rate)
0.22    =  0.34 · 0.070            +  0.66 · r
0.22    =  0.0238                  +  0.66 · r
r       =  (0.22 - 0.0238) / 0.66  =  0.1962 / 0.66  =  0.297   <- 29.7%, not 22%

Step 3 — turn a default rate into dollars per approval. Assume an $8,000 average loan, $4,800 lost when a borrower defaults, and $900 of margin on a loan that is repaid:

per incremental approval:
  0.297 × 4,800  -  0.703 × 900   =  1,426 - 633  =  $793
  × 3,300                                          =  $2.62M per cycle

Pricing the same 3,300 at the pooled 22% instead gives 0.22 × 4,800 − 0.78 × 900 = $354 each, so $1.17M — an understatement of 55%. It charges the risky tail at the average of a pool that is two-thirds risky tail and one-third the safest applicants in the book.

Step 4 — say what the $2.62M buys, because the cost alone is not the argument. A single point of default rate on the 340,000 loans the policy does approve is worth:

340,000 × 0.01 × 4,800  =  $16.3M

So the unbiased training set pays for itself if it improves the default rate by 2.62 / 16.3 = 0.16 of a point — about a sixth. That comparison, not the $2.62M alone, is the argument.

Every system that acts on its own predictions has this problem. Recommenders only observe feedback on what they showed; fraud systems only observe outcomes on what they let through; face-blurring systems only see complaints about what they missed. Say it in the framing, not as an afterthought at the end.

2d. Delayed labels, and the maturity correction

The fourth break is time: the truth arrives months after the prediction. Ranking and calibration break very differently under delay, so the right move depends on which one you need.

The worked case is card fraud. A chargeback — the bank reversing a disputed payment — can arrive up to 90 days after the transaction. If you retrain weekly on the last 30 days of transactions and treat “no chargeback yet” as a clean negative, you are mislabeling every fraud whose chargeback has not landed.

The maturity curve

Let m(a) be the maturity of a label at age a: of the rows that will eventually turn out positive, the fraction whose positive verdict has already arrived by age a.

So m(30) = 0.52 means that after 30 days, just over half the frauds have announced themselves and just under half are still silent. Measured on this business the curve looks like this:

age  7d   m = 0.18
age 14d   m = 0.31
age 30d   m = 0.52
age 60d   m = 0.83
age 90d   m = 0.97

The mistake almost everyone makes

Now suppose you train on every transaction from the last 30 days. Ages are spread evenly from 0 to 30, so the average row is 15 days old.

The tempting move is to look up m(15). That is wrong. The factor by which your labels understate the truth is the mean of m across the whole window, not m evaluated at the mean age:

m(15)                                     =  0.3231     <- the tempting number
mean of m over a uniform 0-30d window     =  0.2995     <- the correct one
                                             ------
m(mean age) / mean(m)                     =  1.079

The two differ because m is concave: it rises quickly at first and then flattens, so its graph bends downwards.

The direction of the error is not luck. Jensen’s inequality — for a curve that bends downwards, the curve’s value at the average input is at least the average of the curve’s values — guarantees m(E[age]) >= E[m(age)], where E[·] means “the average of.”

So evaluating the maturity curve at the mean age always makes your data look more mature than it is. Here it is 8% optimistic. For a wider training window it is unboundedly worse. Use m̄ = 0.30.

Two consequences, and they are not the same consequence

1. RANKING is barely hurt.  A missed positive is a false negative in the
   training labels, uniformly at random across the score range if the
   chargeback delay is independent of the fraud score. Rank order survives.

2. CALIBRATION is destroyed.  observed p = 0.30 · true p.
   A model that outputs 0.010 is describing a 0.033 event.
   And the threshold you derived from the cost matrix is now wrong by 3.3x.

Substituting the second one: a model outputs 0.010, and 0.010 / 0.30 = 0.033, so the event it is actually describing happens 3.3% of the time. Every threshold derived from a cost matrix is off by that same 3.3x.

So the answer to “labels are delayed” depends on whether you need an ordering or a probability — which is framing question 1 arriving again twenty minutes later.

Three moves are available:

“Weight it accordingly” means two different things, 5.5x apart

The middle row of that table says to weight a young negative. That instruction has two plausible readings, and they differ by a factor of 5.5 at age 7. Only one of them is right.

The tempting reading: weight the row by the maturity itself, m(7) = 0.18. “The label is 18% arrived, so count this row as 0.18 of a row.”

The correct reading: use the posterior probability that a row seen without a positive event really is negative. “Posterior” means the probability after taking the observation into account. At age 7 with a 1% base rate that number is 0.9918.

The factor of 5.5 is exactly that ratio:

0.9918 / 0.18  =  5.5

The honest weight is five and a half times the tempting one. The reason is that m describes how much of the positive verdict has landed. It says nothing at all about a row that was overwhelmingly likely to be negative from the start.

Here is the posterior written out. p is the overall positive rate — the base rate, meaning the fraction of all rows that eventually go positive:

P(truly negative | no event by age a)
    =  (1 - p) / [ (1 - p) + p·(1 - m(a)) ]
    ~= 1 - p·(1 - m(a))                          (to first order in p)

Substituting p = 0.01 and m(7) = 0.18:

(1 - 0.01) / [ (1 - 0.01) + 0.01 × (1 - 0.18) ]
    =  0.99 / (0.99 + 0.0082)
    =  0.99 / 0.9982
    =  0.9918

The complement of that posterior is the row’s soft positive label, p(1 - m(a)) / (1 - p·m(a)) — the chance the verdict has not yet arrived.

That pair is self-consistent in a way that weighting by m(a) alone is not. The check is one line: add up the expected positive weight in a group of rows all of age a, and you should get back the base rate p.

p·m(a)              observed positives
  + (1 - p·m(a)) · p(1 - m(a))/(1 - p·m(a))      soft mass on the negatives
  =  p·m(a) + p(1 - m(a))  =  p                  exactly, for every a

Soft labelling recovers the true base rate at every age, which is the property you actually wanted. Weighting each row by m(a) does not have it.

The code below implements all four pieces: the measured curve, the window average, the rate correction, and the soft label. The output block after it is what it prints — those are the numbers quoted above.

MATURITY_POINTS = [(0, 0.0), (7, 0.18), (14, 0.31),
                   (30, 0.52), (60, 0.83), (90, 0.97)]


def maturity_curve(age_days: float) -> float:
    """Linear interpolation of the measured maturity points.

    `m(a)` = fraction of true positives whose label has arrived by age `a`.
    Flat beyond the last measured point: nothing licenses extrapolating a
    curve you stopped measuring.
    """
    if age_days <= 0:
        return 0.0
    if age_days >= MATURITY_POINTS[-1][0]:
        return MATURITY_POINTS[-1][1]
    for (x0, y0), (x1, y1) in zip(MATURITY_POINTS, MATURITY_POINTS[1:]):
        if x0 <= age_days <= x1:
            return y0 + (y1 - y0) * (age_days - x0) / (x1 - x0)
    raise AssertionError("unreachable")


def window_maturity(max_age_days: float, curve=maturity_curve,
                    steps: int = 10_000) -> float:
    """Mean of m over a uniform training window, NOT m at the mean age.

    m is concave, so m(mean age) >= mean(m) by Jensen -- evaluating the
    curve at the mean age always overstates how matured the window is.
    """
    return sum(curve(max_age_days * i / steps)
               for i in range(steps)) / steps


def maturity_correct(observed_rate: float, maturity: float) -> float:
    """Recover the true positive rate from a censored label window.

    observed = true * maturity, where maturity is m AVERAGED OVER THE AGES
    in the training window. Only valid when label delay is independent of
    the feature vector -- if fraud rings deliberately delay chargebacks,
    this is a lower bound.
    """
    if not 0 < maturity <= 1:
        raise ValueError("maturity must be in (0, 1]")
    return observed_rate / maturity


def censored_soft_label(age_days: float, base_rate: float,
                        curve=maturity_curve) -> float:
    """Soft positive label for a row that has NOT gone positive by `age_days`.

    Its complement is the negative's weight, 1 - p(1-m)/(1-p*m), which is
    the table's `1 - p*(1 - m(a))` to first order. Returning the soft label
    rather than a weight keeps the base rate exact at every age.
    """
    m = curve(age_days)
    return base_rate * (1.0 - m) / (1.0 - base_rate * m)


m_bar = window_maturity(30)
print(f"m(15)            = {maturity_curve(15):.4f}")
print(f"mean m over 0-30d = {m_bar:.4f}")
print(f"true rate from an observed 1.0% = "
      f"{maturity_correct(0.010, m_bar):.4f}")
for age in (7, 30, 90):
    soft = censored_soft_label(age, 0.01)
    print(f"age {age:2d}d: soft positive {soft:.5f}, "
          f"negative weight {1 - soft:.5f}")

m(15)            = 0.3231
mean m over 0-30d = 0.2995
true rate from an observed 1.0% = 0.0334
age  7d: soft positive 0.00821, negative weight 0.99179
age 30d: soft positive 0.00483, negative weight 0.99517
age 90d: soft positive 0.00030, negative weight 0.99970

Notice how little the negative weights move: 0.992 at age 7 against 0.9997 at age 90.

At a 1% base rate a young negative is still 99.2% likely to be a genuine negative, because most rows were never going to be positive at any age.

That is the whole reason ranking survives delay and calibration does not. The correction is nearly invisible in the per-row weights, and it is a factor of 3.3 in the aggregate rate.

What interviewers probe: whether “the label is delayed” produces a design change or a shrug. The strong answer names which of ranking and calibration breaks, and picks the move from the answer to framing question 1.

2e. The operating point, which belongs here and not in Stage 7

Surviving the four breaks leaves one decision that completes the objective: the operating point. There are three ways to derive one — and a fourth case with no threshold at all, which is the running problem’s case and the one candidates fumble.

The model emits a continuous score. The product does exactly one thing. The operating point is the rule that maps between them.

It belongs to the objective rather than to the metrics, because it determines which metrics you are even allowed to report. You cannot report precision without having chosen a cutoff. Every problem chapter in this track argues the operating point in its second or third section, before a model is chosen, and that ordering is correct.

The three regimes

There are three ways to pick a threshold, and naming which regime you are in is most of the answer. Two symbols and one term before the table:

• C_fp — the cost of a false positive: flagging something that was fine.
• C_fn — the cost of a false negative: missing something that was not.
• quantile — a cut point in a distribution, stated as a percentage. The 99th quantile of the score is the value that only the top 1% of scores exceed.

The fourth case: no threshold at all

This is the one people forget, and it is the running problem’s case.

The home feed always returns exactly ten items, so no score is ever compared to a cutoff. What stands in for the operating point is the slate policy: how many of the ten slots go to the ranker’s top scores, how many to diversity, and how many to exploration.

That is still an operating point in the sense that matters. It is a knob with a measurable cost, chosen before the model exists.

The block below is the allocation and then the price of the exploration slot. The two conversion rates in it are measured on the ramp, not assumed.

slate of 10, allocated
  7 slots   ranker's top scores
  2 slots   diversity-constrained (max 1 per creator, max 2 per topic)
  1 slot    exploration, sampled from candidates with < 100 impressions

exploration cost, measured on the ramp:
  the explore slot converts at 0.031 vs 0.052 on a ranked slot
  10% of slots × (0.052 - 0.031) / 0.052   =  4.0% of feed engagement

Check the arithmetic in two steps:

how much worse one explore slot is:   (0.052 - 0.031) / 0.052  =  0.404   (40.4%)
it is one slot in ten:                 0.404 / 10               =  0.0404 (4.04%)

Four per cent of engagement is the price of the only counterfactual data this system will ever have. Counterfactual means data about what would have happened had the ranker chosen differently, which by construction the ranker’s own choices can never supply.

Say the number out loud. It turns “we should have exploration” from a platitude into a trade the interviewer can push back on.

That one 4% line item pays for three separate things:

1. The fix for the cold-start row in Stage 11.
2. The source of the unbiased traffic slice that Stage 3 needs in order to estimate position bias — which, as that section insists, cannot be recovered from observational data at all and requires an intervention.
3. The exploration term in the slate policy itself.

Stage 3 — Data and labels, and what they cost

The second of the five questions — what are its inputs — starts on the label side: where labels physically come from, what each source is biased by, and what each costs in dollars. Four sources are worth considering at all, taken below in the order you should consider them.

Explicit vs implicit feedback

Start by separating the two kinds of signal a product can give you.

• Explicit feedback — a user deliberately telling you something. A star rating, a thumbs-up, a survey answer.
• Implicit feedback — behaviour you infer a judgement from. A click, a play, dwell time (how long the item stayed on screen), a purchase.

They differ on every axis that matters. The row to read first is Negatives, because it is the one that decides how you train:

The missing-negatives problem is the defining feature of implicit feedback, and the standard fix is to sample negatives rather than to treat every non-click as a zero.

Two ways to sample, with different failure modes:

• Uniformly from the catalog. The negatives come out too easy. The model learns to tell a cooking video from a random unrelated one, which it could already do, so training stops teaching it anything. (The loss and why random negatives stop teaching derives exactly how much of the learning signal that destroys.)
• In proportion to popularity. Harder, more informative negatives — at the cost of a popularity-dependent bias you then have to correct.

Position bias, and the one number it needs

Position bias is the fact that where an item was placed on the screen changes how often it is clicked, independently of how good it is.

It is worth one number. Write e_r for the probability that a result at rank r is even looked at. Then the click rate you observe is e_r times the true relevance. A good item buried low therefore looks worse than a bad item at the top — and the block below shows exactly that happening:

e_1 = 1.00, e_5 = 0.31, e_10 = 0.14      (typical for a 10-blue-links page)

an item at rank 10 with relevance 0.60 shows CTR 0.084
an item at rank  1 with relevance 0.15 shows CTR 0.150

-> training on raw clicks teaches the model that rank-1 items are better,
   which is a statement about your previous ranker, not about the items

Each of those CTRs is relevance × e_r: 0.60 × 0.14 = 0.084 and 0.15 × 1.00 = 0.150. The four-times-better item shows a click rate 44% lower.

The fix is inverse-propensity weighting: count each click by 1/e_r instead of by 1, so a click on a rarely-examined slot counts for more. That exactly undoes the examination bias.

You get e_r by running a small slice of traffic in which results are deliberately swapped between positions at random, then comparing click rates for the same item in different slots.

You cannot estimate position bias from observational data alone. It requires an intervention. That sentence is worth saying out loud, because it is the thing candidates most often assume away.

Human labeling economics

Human labels are the one input to an ML system that costs real money per row. Price the plan in full, because a labeling plan without a number cannot be evaluated — and then face the accuracy ceiling the raters themselves impose.

Two terms in the block below:

• gold questions — items whose answer is already known, mixed into the stream to measure each rater’s accuracy.
• adjudication — a senior rater breaking ties on items where the three raters disagreed.

The first line does the volume, the middle lines add the overheads, and the last line is what you would actually quote. The first line unpacks as 50,000 × 3 × 40 = 6,000,000 rater-seconds, and 6,000,000 / 3,600 = 1,667 rater-hours.

50,000 items · 3 raters · 40 s per item        =  1,667 rater-hours
at $22/hour fully loaded                       =  $36,700
+ 12% gold questions and rater qualification   =  $41,100
+ adjudication of the 9% non-unanimous items:
    4,500 items · 90 s · $44/h (senior rater)  =  $ 4,950
                                                  -------
                                                  $46,000
                                          ≈ $0.92 per labeled item

The adjudication line is the one to show your work on, because it is the only line whose unit cost exceeds the blended figure the block concludes with.

$4,950 / 4,500  =  $1.10 per adjudicated item      against $0.92 all-in

Adjudication takes longer per item and is done by someone paid double.

Say which denominator you are quoting it against, because the two readings differ by more than a point:

4,950 / 41,100  =  12.0%     adjudication as a surcharge on the pre-adjudication subtotal
4,950 / 46,000  =  10.7%     adjudication as a share of the finished budget

The 12% is the more useful of the two, because it scales with the disagreement rate while the base it sits on stays fixed.

Read that way, consider what happens if a vague rubric drives disagreement from 9% to 20%:

20% of 50,000 = 10,000 items · 90 s · $44/h  =  $11,000
                             $11,000 / $41,100  =  26.8%

The disagreement rate grew by 11 points; the adjudication surcharge grew by 15, from 12.0% to 26.8%. That is the argument for spending a week on the rubric before spending $46,000 on raters.

On the running problem this is the satisfaction-labeled sample that the composite weights in Stage 2a are fit on, and Stage 10 prices it at 20,000 items a month. It is not a one-off purchase — the weights drift as the content mix does, so it is a standing line in the budget.

The rater-agreement ceiling

Now the ceiling almost nobody computes.

Take three raters and use the majority answer. If each rater is right with probability a, independently, then the majority is right with probability

3a^2 - 2a^3

which is the chance all three are right, plus the chance exactly two are. At a = 0.85:

3 × 0.85^2  -  2 × 0.85^3   =   3 × 0.7225  -  2 × 0.614125
                            =   2.1675      -  1.2283
                            =   0.9393

The table extends that to two panel sizes:

At 0.85 rater accuracy, 6.1% of your “ground truth” is wrong — 1 − 0.939 = 0.061.

Be precise about what that caps, because the sentence people say next is really three different claims wearing one set of words. Read the three cases below as three different models measured against the same flawed labels:

a PERFECT model scores exactly 0.939 measured, because it disagrees
  with the labels on precisely the 6.1% the labels got wrong

a model whose errors are INDEPENDENT of rater error scores at most
  0.939, and its measured accuracy understates its true accuracy

a model that has learned the raters' biases -- same rubric ambiguity,
  same edge cases -- can score ABOVE 0.939 while being worse

So 0.939 is a ceiling on measured accuracy for any honest model. It is not a ceiling on true accuracy at all.

If your model reads 91% and you are trying to reach 95%, you are chasing a number the labels cannot express. And you cannot tell from the 91% alone whether the remaining 2.9 points (93.9 − 91.0) are model error or label error.

The move is a better rubric — clearer written instructions to the raters — because that raises a and lifts the ceiling for everyone.

The move is not more raters. The marginal rater has a sharply diminishing return, visible in the table:

a = 0.85, 3 -> 5 raters:   0.973 - 0.939  =  +3.4 points   for 67% more spend
a = 0.95, 3 -> 5 raters:   0.999 - 0.993  =  +0.6 points   for the same 67%

Weak supervision

When labels are expensive and rough rules are cheap, you can manufacture a large noisy training set instead of buying a small clean one.

Write k labeling functions. Each one is a cheap rule that votes on some rows and abstains on the rest: a text pattern, a business rule, an existing model, a lookup against a reference database. Each has an unknown accuracy and covers only part of the data.

You then fit a small statistical model over which functions agree with which. That is enough to infer how trustworthy each one is without ever seeing the true labels, and the model emits a probability per row.

In the block below, “LF” is short for labeling function, and coverage is the fraction of rows on which that function votes at all. Notice the trade in the two columns: kb_join is the most accurate and votes on 11% of rows, while old_model votes on nearly everything and is wrong a fifth of the time.

5 labeling functions on 2M unlabeled rows

LF          coverage   empirical accuracy on a 500-row gold set
regex_A       0.31          0.88
rule_B        0.62          0.71
old_model     0.94          0.79
kb_join       0.11          0.96
heuristic_E   0.44          0.66

union coverage 0.97, mean 2.4 LFs fire per row
label-model output vs gold:  0.86 accuracy

Weak supervision buys you a 0.86-accuracy label on 2M rows for a week of engineering. Human labeling buys a 0.94-accuracy label on 50,000 rows for $46,000.

Which is the better buy depends entirely on where your model sits on its learning curve — the plot of validation error against training-set size. If doubling the data still moves the error, take the 2M noisy rows. Plot that curve before choosing, because the answer flips depending on the model.

Class imbalance

Class imbalance means one outcome is far rarer than the other — fraud, spam, a click on an ad. It is the last of the four label problems.

The one-line version: imbalance is a threshold problem wearing a data problem’s clothes (Imbalance is usually a threshold problem wearing a data problems clothes). The model usually learns fine. What breaks is the cutoff you compare its scores against.

So do not resample by reflex. Throwing away negatives or duplicating positives changes the base rate, which changes what the model’s probabilities mean and destroys calibration (Resampling and the calibration it breaks). Use class weights instead, keep the probabilities honest, and derive the threshold from costs.

The one real exception is extreme imbalance, where the rare class is too rare to show up in a training batch at all. A batch is the group of rows the model looks at before each weight update.

Put numbers on it. At 1 positive in 10,000 with a batch of 256, the chance a batch contains at least one positive is about 256 / 10,000 = 2.5%. So 97.5% of weight updates see nothing but negatives, and the learning signal is swamped by negatives the model already gets right.

That is precisely the case focal loss was built for: a modified loss that shrinks the contribution of examples the model already gets right, so the rare hard ones dominate (ml 07). It is exactly the situation in chapter 03.

Stage 4 — Features

Labels were half of what are its inputs; features are the other half: which numbers go in, grouped into five families, and one structural fact about those families that decides the whole serving architecture.

In the interview this is two paragraphs, not ten. Here is what to say and what to skip.

The table has five rows, one per family. The column that matters is the third one — what you would actually say out loud about each family.

The structural point, and the one that connects features to architecture: entity and user features can be precomputed. Cross features cannot.

That asymmetry is what makes two-stage retrieval possible:

• Stage one is restricted to features that split cleanly into a user side and an item side. That lets you compute every item’s side in advance and put it in an index.
• Stage two is where the pair-specific features live. There is no way to precompute those, because there are 20 million × 18 million of them.

The feature sketch for the running problem

Give one line per family, with a count. Those counts are what Stage 10 divides by, so they are load-bearing rather than decorative.

Read the block in two halves. The top half is the split by family. The bottom half explains what each part of the split buys you architecturally, and then converts 200 feature slots into the 512 numbers the model’s first layer actually sees.

ranking feature vector, 200 slots

  140  item-side     age, duration, creator, topic embedding, quality
                     score, CTR/completion counters at 1h / 1d / 30d
   40  user-side     long-term topic affinities, creator affinities,
                     recency-decayed watch history summary
   20  context+cross  time of day, device, session position, slot index,
                     user x topic affinity, user x creator affinity

the 140 depend only on the item      -> precomputable, indexable, cacheable
the  40 depend only on the user      -> fetched ONCE per request, not per candidate
the  20 exist only at rank time      -> computed on the box, from the other two

of the 200 slots, 176 already hold numbers and 24 hold identifiers
  (creator, topic, device, country), expanded through embedding tables
  at the model's input at 14 numbers each:
      176 + 24 x 14  =  512 numbers reach the first layer   (Stage 5)

That 140 / 40 / 20 split is the entire Stage 10 architecture, written down here in Stage 4. One consequence each:

• The 140 is why every copy of the ranking service can hold all the item features in its own memory.
• The 40 is why the per-request user fetch is 160 bytes (40 × 4) rather than 160 kilobytes.
• The 20 is why there has to be a second stage at all.

State the split here and the serving section becomes arithmetic rather than a debate.

Everything else about how values are encoded, how target encoding leaks, and how training and serving drift apart is derived in ml 01. Do not re-derive any of it in the interview; cite it and move.

Stage 5 — Model choice, and the two-stage pattern derived

What is the model gets answered twice here, because a catalog of 20 million items against a 100 ms budget forces two models rather than one.

The order matters, and it is deliberately not the order you would guess:

1. Derive why one model cannot work.
2. Design the shortlist stage.
3. Size the shortlist.
4. Then pick a model family — because that choice is nearly forced once the first three are settled.

Say the mid-round checkpoint before you start. It is a scored moment, and being asked what you want to go deeper on costs you the point you would have earned by naming the hard part first:

“That is the skeleton. The two things I think are actually hard here are the candidate path — 20 million items against a 100 millisecond budget is not close, and I want to show you the factor — and the label window, because the honest terms in my composite target only close out at the end of a session and that puts a floor under the retraining cadence. I am going to start with the candidate path. Stop me if you would rather see the other one.”

Why two stages exist

The two-stage architecture falls out of arithmetic, and it should be presented that way rather than named as a pattern, because the interviewer is scoring the derivation and not the name.

The inputs are the two constants fixed earlier: a catalog of 20 million items, and a 100 ms budget at the 99th percentile for the whole ranking service.

Step 1 — what one full scoring costs

A production ranker costs about 40 microseconds per item. That number is derived below rather than quoted.

The model is a 4-layer MLP — multi-layer perceptron, the plainest kind of neural network, a stack of matrix multiplications with a squashing function between them. It reads a 512-number input vector, and its layer widths are 512, 256, 128, 1.

Three units before the arithmetic:

• Each layer costs two floating-point operations per weight: one multiply and one add.
• A kFLOP is a thousand such operations; a GFLOP is a billion.
• 50 GFLOP/s is what one modern CPU core sustains on batched matrix work using its wide vector instructions (AVX-512).

Where the 512 came from, since Stage 4 said 200 features. The two counts are different things and have to be kept apart. A slot holding an identifier — creator, topic, device, country — is not a number a matrix can multiply, so it is looked up in an embedding table and its embedding concatenated with the numeric slots:

176 numeric slots  +  24 identifier slots × 14 numbers each
   =  176 + 336
   =  512 numbers reaching the first layer

200 is what the feature path moves, and what Stage 10 bills at 4 bytes apiece. 512 is what the first matrix multiply sees. The table lookups that turn one into the other are a large part of the gap between 17 and 40 microseconds below.

Now the cost. Each term in the sum is one layer’s weight count — inputs × outputs — and the leading 2 is the multiply-and-add:

FLOPs = 2 · (512·512 + 512·256 + 256·128 + 128)
      = 2 · (262,144 + 131,072 + 32,768 + 128)
      ≈ 852 kFLOP per item

at 50 GFLOP/s per core (AVX-512, batched)      =  17 us
plus embedding gathers and feature assembly    ≈  40 us per item

Check the division: 852,224 / 50,000,000,000 = 1.7e-5 seconds, which is 17 microseconds.

The gap between 17 and 40 microseconds is not arithmetic at all. It is fetching each embedding row out of memory and stitching the 200 feature values into one contiguous vector.

That gap matters enormously in Stage 10, where sizing off the 17 rather than the 40 is the single easiest way to under-build a fleet by a factor of two.

Step 2 — try to score the whole catalog

20,000,000 items × 40 us  =  800 seconds per request
budget                    =  0.1 seconds
                             ---------------------
                             8,000x over

Step 3 — try the cheapest thing that could possibly work

Eight thousand times over budget is not a tuning gap. So drop the expensive model entirely and score every item with a single dot product: the user’s 64-number embedding against the item’s, with no feature fetch at all. That is 64 multiplies and 64 adds, so 128 operations per item.

Two independent paths are priced below — how much arithmetic it takes, and how many bytes must be pulled out of memory. Both land on the same number.

128 FLOP per item × 20M   =  2.56 GFLOP
at 50 GFLOP/s per core                      ->  51 ms of compute
memory scanned            =  20M × 64 × 4 B =  5.1 GB
at 100 GB/s effective                       ->  51 ms

Both paths land on 51 ms. One query has consumed half the latency budget and an entire core’s memory bandwidth — at one query per second.

Memory bandwidth is how many bytes per second a machine can pull out of its main memory (RAM, random-access memory). It is a hard ceiling, independent of how fast the arithmetic is.

Scale that to peak traffic:

5,000 QPS × 5.12 GB per query   =  25.6 TB/s of memory bandwidth
at 100 GB/s per machine          =  256 machines doing nothing but dot products

So even the cheapest possible scan of every item does not survive contact with scale.

Step 4 — the conclusion, and what buys it

Stage one must be sublinear, not merely cheap. Sublinear means its cost grows slower than the catalog does, so doubling the catalog does not double the work.

That is what an approximate-nearest-neighbour index buys. The specific one used here is HNSW, hierarchical navigable small world: a graph in which every item is a node linked to a few near neighbours, plus a few deliberate long-range shortcuts. A search walks from wherever it starts to the right neighbourhood in a handful of hops.

Two symbols in the block: efSearch is HNSW’s one important knob — how many candidate nodes the walk keeps alive at once, trading recall for speed — and N is the number of items indexed.

HNSW, N = 20M, efSearch = 64:  ~3,000 distance computations per query
                               (four orders of magnitude below 20M)
                               ~1.2 ms, ~10 MB touched

Neither the 1.2 ms nor the 10 MB follows from any constant above it, so derive both rather than quoting them. The block has three parts — compute, latency, bytes — and the surprise is that the compute line is 156 times too small to explain the 1.2 ms:

compute      3,000 × 128 FLOP  =  0.384 MFLOP
             at 50 GFLOP/s     =  7.7 us          <- 156x below the 1.2 ms

latency      3,000 DEPENDENT random accesses, each waiting on the previous
             hop to know where to go next, at ~400 ns of memory latency
             3,000 × 400 ns    =  1.2 ms          <- this is where it comes from

bytes        3,000 × 64 × 4 B  =  0.77 MB of actual vector data
             but 3,000 nodes scattered across a 5 GB region touch
             ~3,000 distinct 4 KB pages
             3,000 × 4 KB      =  12 MB           <- "~10 MB", page-granular

An approximate-nearest-neighbour query is limited by memory latency, not by arithmetic, and the graph walk is serial by construction. You cannot fetch hop n+1 early, because only hop n can tell you where it is.

That is the same lesson as the full scan two blocks above, arriving from the opposite direction: at this scale everything here is limited by memory, not by arithmetic.

Two practical consequences. efSearch is the tuning knob that matters, because it sets how many serial hops you pay for. And batching queries together improves throughput but never latency, because the hops of one query still have to happen one after another.

How much memory each vector costs, what building the index costs, and why deleting from one is hard are all derived in Hnsw memory per vector derived. Do not re-derive them here.

The four-stage funnel

The diagram below is the design those four steps forced. The item count drops at every arrow — 20M -> 4M -> 1,000 -> 100 -> 10 — while each stage is more expensive per item and smarter than the one before it, which is the whole idea.

flowchart LR
    Q(["Request"]) --> R1["Stage 0 · Filters<br/>eligibility, region, stock<br/>20M -> 4M"]
    R1 --> R2["Stage 1 · Retrieval<br/>ANN + heuristics + recent<br/>sublinear · 1.2 ms<br/>4M -> 1,000"]
    R2 --> R3["Stage 2 · Ranking<br/>MLP/GBDT + cross features<br/>40 us/item · 40 ms<br/>1,000 -> 100"]
    R3 --> R4["Stage 3 · Policy<br/>diversity, dedupe, business rules<br/>100 -> 10"]
    R4 --> O(["Response"])

    style R2 fill:#2d6a4f,color:#fff
    style R3 fill:#bc6c25,color:#fff
    style R4 fill:#1d3557,color:#fff

Walk the four boxes:

• Stage 0 · Filters — eligibility, region, stock. Is this item allowed here, is it still available, has this user blocked its creator. Cuts 20M to 4M.
• Stage 1 · Retrieval — the sublinear part: the ANN index, plus cheap heuristics, plus a source of recently published items. 1.2 ms, cutting 4M to 1,000.
• Stage 2 · Ranking — scores those 1,000 with the expensive model and its cross features at 40 microseconds each, so 40 ms in total. Keeps 100.
• Stage 3 · Policy — diversity rules, near-duplicate removal, business rules. Cuts 100 to the 10 that are rendered.

Stage 0 is the arrow the diagram makes look free

“Eligibility, region, stock — 20M to 4M” is drawn as a filter sitting in front of the index. Filtering a graph index is the genuinely hard case in this whole area, and there are two obvious approaches that both fail.

Pre-filtering deletes the ineligible nodes before searching. That severs the long-range shortcut edges the fast search depends on, so recall collapses on exactly the queries whose eligible set is small.

Post-filtering leaves the graph intact and discards most of what it returns. So you must over-fetch by roughly the inverse of the eligible fraction:

eligible fraction     4M / 20M         =  0.2
over-fetch needed     1,000 / 0.2      =  ~5,000 asked of the index
cost of going deeper  1.2 ms           ->  3-4 ms

The production answer is neither. Partition the index by the filter that has the most distinct values and is nearly always applied — region, here — so each partition is already filtered. Accept that a partitioned index costs memory in proportion to how many partitions an item can appear in.

State the over-fetch factor and the partition key, or that arrow is decoration.

Candidate generation, which is a design surface and not a call to an index

The shortlist stage is not one model call. It is a blend of several sources with hand-set quotas, and the quotas are a product decision rather than a training outcome.

Look back at the diagram’s Stage 1: “ANN + heuristics + recent.” That plus sign is doing more work than the index is.

In every recommendation chapter in this track, candidate generation is the largest section and it is kept separate from model choice (Candidate generation, Candidate generation a graph query with an expiry date, Candidate generation the 2 hop explosion derived). Retrieval is a union of sources with quotas, and no loss function will set those quotas for you.

A two-tower model, named in the first row below, is the family this stage requires. Two separate small networks: one reads only the user, one reads only the item, each producing an embedding, and the score of a pair is the dot product of the two. That split is exactly what lets you compute every item’s tower output once, offline, and put the results in an index.

Here is the blend for the running problem. The Quota column sums to the 1,000 candidates the ranker will score, and the third column is the argument you would have to make in the room for each row:

Three things to say about that table, each of which distinguishes a candidate who has run one of these systems from one who has read about them:

1. Every source needs its own recall number, measured separately, with an owner. Recall@1,000 here means the fraction of the truly best items that survive into the shortlist of 1,000. An aggregate recall@1,000 of 0.908 is perfectly consistent with the recent-items source contributing nothing at all, and that is usually what has happened when it breaks, because it is only 20% of the pool and the aggregate barely moves.
2. The quotas, not the scores, decide the blend. If you merge across sources by score you are comparing an ANN cosine similarity against a heuristic’s rank position, and those are not on the same scale, so whichever source emits the most confident-looking numbers silently wins. Quotas are the honest answer. A learned blender is the expensive answer and needs a label of its own that does not exist.
3. Sources have different freshness clocks, and the slowest one bounds the system. An hourly index rebuild means a video published at 12:05 cannot be retrieved by the main source until 13:00, which is the entire reason source two exists.

How many candidates, and why the margin does not pick it

One number is still unset: the shortlist size. The real lesson in picking it is that the usual justification for such a number does not survive being checked.

The candidate count is not 1,000 because 1,000 is a round number. The story people tell is that it sits at the “knee” where extra recall stops paying for extra latency. On this data there is no knee, and finding that out is the point.

The measurement

The table below is retrieval recall measured against the ranker’s own top 10, on a held-out sample where every item was scored exhaustively. So the “truth” being recalled is what the ranker itself would have chosen given the whole catalog.

The last column is the one to read: recall bought per extra millisecond of ranking time. Latency is just n × 40 us.

There is no knee. The last column falls 0.00931, 0.00235, 0.00082, 0.00019 — dropping steadily by roughly 4x, then 3x, then 4x, with no bend anywhere.

That is what always happens when a benefit that saturates is charged against a cost that is perfectly linear. There is no distinguished point, only the point you decide to stop at.

Marginal analysis alone cannot pick n. It can only tell you what a given n costs, which is a different and much lesser thing. Claiming a “knee” here is the most common way to make a chosen number look derived.

Putting a price on recall

To actually pick a number you need an exchange rate between recall and latency, in a common currency.

The block below does that in three parts. First it converts retrieval recall into end-to-end recall by multiplying through the ranker. Then it states the two measured exchange rates. Then it checks each rung of the ladder against the price. When you do this honestly, it says buy more, not less.

end-to-end effect, since the ranker is also lossy:
P(best item in the shown top-10) = recall@n × P(ranker top-10 | retrieved)

n = 500:    0.861 × 0.78  =  0.672
n = 1,000:  0.908 × 0.78  =  0.708      +3.6 points end to end for 20 ms
n = 2,000:  0.941 × 0.78  =  0.734      +2.6 points more for another 40 ms

exchange rate for this service, both measured not assumed
  ramp:      +1.0% engagement per end-to-end recall POINT (0.01 absolute)
  latency:   -1.0% engagement per 100 ms of added p99
  so 1 ms costs 0.01% engagement, and 0.01% engagement buys
  0.01 / 1.0 = 0.01 recall points  =  1e-4 of ABSOLUTE end-to-end recall

what each rung actually delivers, end to end (retrieval per-ms x 0.78)
  500 -> 1,000      0.00235 x 0.78  =  1.8e-3      18x the price
  1,000 -> 2,000    0.00082 x 0.78  =  6.4e-4       6x the price
  2,000 -> 5,000    0.00019 x 0.78  =  1.5e-4     1.5x the price

Where the 0.78 came from

The × 0.78 is not decoration. It carries the entire argument above, so say where it came from rather than letting it appear.

It is the ranker’s own top-10 hit rate, measured on the same held-out sample as the recall column but against a different ground truth. Constructed like this:

1. For each request in the sample, take the item the composite label says was genuinely best.
2. Keep only the requests where that item did survive into the shortlist of 1,000.
3. Count how often the ranker then placed it in the visible top 10.

That count is 78%.

The two numbers multiply rather than overlap, because they ask different questions. Recall asks whether retrieval handed the ranker the right item. The 0.78 asks whether the ranker then did anything with it. The recall column is scored against the ranker’s own exhaustive top-10, so it cannot see this second loss at all.

The 0.78 is also not a constant. It moves when the label definition or the slate size moves, so it is remeasured alongside them and never carried forward from a previous quarter.

Stages multiply, so a loss at retrieval is damped by the lossiness of every stage after it — the same argument as Why recall below 10 is usually fine. Retrieval improvements are worth 1 − 0.78 = 22% less than they look on their own.

Even after that discount, every rung clears its price, and the last one clears it by only 1.5x. So the margin says take 5,000 and stop somewhere just past it.

What actually stops you: the p99 tail

The tail is a different argument from the marginal one, and the distinction is the whole point.

The marginal argument above is about average value. The budget was stated at p99, where a single slow replica or a garbage-collection pause — the runtime reclaiming unused memory, which stops the program briefly — costs you the whole budget.

So the ranking stage has to fit inside half the remaining budget on average, leaving the other half as room for its own bad days.

Two constants do the work in that sentence, and each is the difference between n = 1,000 and some other number entirely. State them out loud rather than leaving them as defaults in a function signature:

• overhead_ms = 10 — everything in the request that is neither retrieval nor ranking: request deserialization, the Stage 0 eligibility filter, the policy layer, response assembly, and the two network hops in and out. Measured on the service you already run, not modelled.
• tail_factor = 2 — the half-budget rule itself. The ranking stage is sized so its mean latency is at most half the time it was handed, leaving the other half for its own p99.

Those two are the entire derivation:

budget for ranking   100 − 1.2 (retrieval) − 10 (overhead)   =  88.8 ms
halved for the tail  88.8 / 2                                =  44.4 ms of mean
candidates it buys   44.4 ms / 40 us                         =  1,110
                                                                -> take 1,000

Move either constant and n moves with it:

tail_factor 2 -> 1        ceiling doubles     1,110  ->  2,220
overhead 10 ms -> 20 ms   ceiling falls       1,110  ->    985

That sensitivity is why an interviewer is entitled to hear both constants rather than find them buried in a signature.

The 2x here and the 2.5x in Stage 10 are not the same multiplier

There is a 2.5x headroom multiplier in Stage 10, and a reader is right to ask whether that double-counts this 2x. It does not. They are different quantities applied to different things.

• The 2x here is a latency reservation inside a single request. It decides how many candidates one request may score before its own tail eats the service budget.
• The 2.5x there is a capacity multiplier on the fleet. It decides how many machines the aggregate 5,000 QPS needs so that queues stay short and losing a machine does not take the service down.

One sizes n, the other sizes the machine count. Neither is computed from the other, and dropping one does not compensate for dropping the other: drop the 2x and every request blows its budget on a bad day; drop the 2.5x and there is no spare machine.

The honest caveat to volunteer is that they are not perfectly independent. A fleet carrying more spare capacity queues less and therefore has a tighter latency tail, so the rigorous treatment is one queueing model rather than two multipliers. Say that, and then say you would still ship the two-factor version, because it is conservative in the direction that matters.

The function below is that derivation as code. Its output block follows, and the two printed numbers are the 2,220 and 1,110 above.

def stage_budget(total_ms, index_ms, per_item_us, overhead_ms=10.0,
                 tail_factor=2.0):
    """How many candidates the ranking stage can afford.

    total_ms      p99 budget for the whole service
    index_ms      stage-1 retrieval latency
    per_item_us   measured cost of one full-feature scoring, including the
                  amortized feature fetch -- not just the model FLOPs
    tail_factor   the stage must run at MEAN latency below budget/tail_factor,
                  or its own p99 -- GC pauses, a cold feature shard, a slow
                  replica -- eats the whole service budget. Sizing a stage at
                  the number its mean can afford is the standard way to ship
                  a service that meets its budget in the median and misses it
                  in the p99, which is the only place the budget was stated.
    """
    remaining_ms = total_ms - index_ms - overhead_ms
    if remaining_ms <= 0:
        raise ValueError("retrieval alone exceeds the budget")
    return int(remaining_ms * 1000 / per_item_us / tail_factor)


def end_to_end_recall(retrieval_recall, ranker_recall):
    """Stages multiply. A 4-point retrieval loss is worth less than 4 points."""
    return retrieval_recall * ranker_recall


print("mean-affordable  ", stage_budget(100, 1.2, 40, tail_factor=1.0))
print("p99-safe         ", stage_budget(100, 1.2, 40))
print("end-to-end @1000 ", round(end_to_end_recall(0.908, 0.78), 3))

mean-affordable   2220
p99-safe          1110
end-to-end @1000  0.708

2,220 is what the arithmetic permits, 1,110 is what you can actually run, so the answer is 1,000 — a round number below a tail-corrected ceiling, not a knee in a recall curve.

Say it that way. The honest sentence is:

“The margin says take 5,000 and the p99 says I can afford 1,110, so I take 1,000, and the binding constraint is latency variance, not recall.”

That sentence also tells the interviewer exactly what would change the answer: cut the per-item cost from 40 us to 20 us and n doubles, with no new information about recall at all.

Choosing the model family

This is the part of the round that takes the least time and that candidates want to spend the most on. One line per data shape is enough.

Two words in the table need defining first:

• tabular — a plain table of rows and columns, the kind a spreadsheet holds.
• cardinality — how many distinct values a column takes. “country” is low cardinality; “video id” with 20 million values is high cardinality.

Find the row matching your data and read across. The Why column is the sentence you say; you do not need the rest.

Three names in that table are answers rather than explanations, and this chapter promised not to do that:

• A CNN (convolutional network) slides one small set of weights across the whole image. A pattern learned in one corner is recognized in every other corner, so the network needs far fewer parameters than one weight per pixel.
• A ViT (vision transformer) instead cuts the image into fixed patches, treats each patch as a token, and runs a transformer over them. Any patch can attend to any other, rather than only to its neighbours.
• A transformer is the architecture underneath both: a stack of attention layers. Each layer lets every element of a sequence look at every other element and take a weighted average of them, with the weights computed from the elements themselves.

That last definition is precisely why “a transformer reading the action history” beats a recency counter. The counter fixes in advance how much a 30-day-old event matters. Attention lets the model decide, per user and per prediction, which of the 200 events in the history this one depends on.

The one architectural constraint you cannot negotiate

A score of the form score(user, item) = f(user) · g(item) is required in stage one and forbidden in stage two.

• Stage one must split that way, so g(item) can be computed once, offline, and indexed.
• Stage two must not split that way, because the whole value of the expensive stage is exactly the user-item interaction terms — and a dot product of two independently computed vectors cannot express them.

Stage 6 — Training

Both models chosen, the third of the five questions — how is it trained — comes due. Exactly four things are worth saying about it in this round:

1. How to split the data.
2. How to choose negatives.
3. How often to refit.
4. Why warm-starting hides problems.

Nobody is scoring your optimizer. Say these four and move.

Split temporally, always

A random split on data with any time structure leaks information from the future into training. Yesterday’s session lands in the training set and today’s in the test set, and every counter feature — a running count over a recent window — bridges the two.

Report the time gap explicitly: train on weeks 1-8, validate on week 9, test on week 10. The size of that gap is what tells you how fast the model goes stale.

Negative sampling is a modelling decision, not preprocessing

For a two-tower retriever you need negative examples. The cheapest source is in-batch negatives: within a training batch, treat every other user’s positive item as a negative for this user.

It is free, and it is biased toward popular items. A popular item appears in more batches, so it is drafted as somebody’s negative more often — and every appearance as a negative pushes its score down.

The correction is one term. Here it is explicitly, because “apply the logQ correction” is useless without the formula. A logit is the raw score before it is squashed into a probability.

logit'(q, i)  =  f(q) · g(i)  -  log Q(i)

Q(i)  =  the probability item i appears as an in-batch negative,
         estimated as a streaming frequency per item id -- a decayed
         count of appearances, updated as batches stream past

That correction is exact, not a heuristic. Here is why, in three steps.

A softmax turns a set of scores into probabilities that sum to one, by exponentiating each and dividing by the total. A sampled softmax does the same using only the handful of negatives in the batch, instead of the whole catalog.

1. Sampling that way estimates P(item | user) under the sampling distribution rather than the true one.
2. That estimate comes out proportional to exp(f·g) · Q(i) — the extra Q(i) is the popularity bias.
3. Subtracting log Q(i) inside the exponent divides that Q(i) back out, leaving exactly the unsampled softmax.

Skip it and every popular item is pushed down in proportion to how often it turned up as somebody else’s negative. The scores end up systematically distorted along the popularity axis. That is a bias in the score scale, not a bug in any one item’s embedding.

Note what the correction is not. It does not fix tail recall. A rarely-seen item’s problem is that it appears in too few positive pairs to have learned a usable embedding at all — a different problem from having its score scale distorted. Those are two separate failures in Stage 11 with two separate controls, and conflating them is common.

Retraining cadence is derived from the decay curve, not chosen

The shape of this problem is an inventory trade-off, not a threshold: a cost that accumulates while you wait, against a fixed cost each time you act.

Measure the decay first. Freeze a model and score it each week against fresh labels:

AUC   0.812 (w0)  ->  0.809 (w1)  ->  0.803 (w2)  ->  0.791 (w4)  ->  0.762 (w8)

Two further inputs turn that curve into a cadence, and you have to ask the interviewer for both, because neither is a modelling quantity. V is what accuracy is worth and Y is what a retrain costs. The deficit line converts the AUC readings above into “how much AUC you have lost by week w”:

V   value of AUC:  a 0.01 AUC drop costs $1,200/day of margin
                   -> $840,000 per AUC unit per week
Y   cost of one retrain:  $900 of compute + half an engineer-day  =  $1,500

deficit d(w) = 0.812 - AUC(w):   0, 0.003, 0.009, 0.021, 0.050

retraining every k weeks costs, per week,
    [ V · (area under d from 0 to k)  +  Y ] / k

Evaluate that formula at four cadences:

Take the weekly row apart, since the staleness column is the one nobody shows their work on. Over one week the deficit d runs from 0 to 0.003, so the area under it is the area of a triangle:

area under d, 0 to 1 week   =  0.5 × 1 × 0.003        =  0.0015
staleness cost per week     =  $840,000 × 0.0015 / 1  =  $1,260
retrain cost per week       =  $1,500 / 1             =  $1,500
                                                         ------
                                                         $2,760

And the daily row, to show the other extreme: retraining every 1/7 of a week means paying the $1,500 seven times a week, which is $10,500, while the staleness you avoid is worth only $180.

Weekly wins, and by nearly 4x over daily ($10,680 / $2,760 = 3.9).

The closed form, and two things it tells you

This is the classic economic-order-quantity trade-off from inventory management: how often to place a restocking order when holding stock costs money and each order has a fixed fee. Staleness plays the role of holding cost; the retrain plays the role of the order.

So long as the decay stays roughly linear at rate r per week, the best interval has a closed form:

cost/week  =  V·r·k/2  +  Y/k          minimised at   k* = sqrt(2Y / (V·r))

k* = sqrt(2 × 1,500 / (840,000 × 0.003))
   = sqrt(3,000 / 2,520)
   = sqrt(1.19)
   = 1.09 weeks  =  7.6 days

First consequence: the square root is why halving the retrain cost does not halve the interval. It shortens it by sqrt(2) = 1.41x, not 2x. Automation buys less cadence than people expect, and the honest reason to automate is reliability rather than freshness.

Second consequence: the printed curve accelerates, so which r you quote matters. Across the four measured segments the per-week decay runs:

w0 -> w1    0.003 / 1              =  0.0030
w1 -> w2    (0.009 - 0.003) / 1    =  0.0060
w2 -> w4    (0.021 - 0.009) / 2    =  0.0060
w4 -> w8    (0.050 - 0.021) / 4    =  0.0073

The r to quote is the whole-window average, 0.050 / 8 = 0.0063, which weights each segment by how long it lasted. The unweighted mean of those four segment rates is 0.0056 and is the wrong summary, because it gives the four-week final segment exactly as much say as the one-week first one.

So “a model that decays 0.003/week” describes week one only. Quoting it is quoting the most flattering point on your own curve.

Beyond about two weeks the linear model — and therefore the closed form — understates the cost of waiting. That is why the table’s monthly row is computed from the actual area under d rather than from V·r·k/2.

The floor the arithmetic cannot see

On the running problem there is a hard lower bound the economics know nothing about.

Framing question 5 established that the slowest trainable term in the composite label needs the user’s session to close out. So labels run about an hour behind, and no cadence below one hour exists regardless of what the economics say.

Here that leaves enormous slack: weekly is 168 hours, so 168 times above the floor. But on a system whose decay is fast enough to want minute-scale refreshes, the label window rather than the pipeline is what you would have to fix — and no amount of infrastructure spending touches it.

Warm starting hides drift

Warm starting means initializing this week’s training run from last week’s weights instead of from scratch.

It converges 5 to 10 times faster. It also quietly makes the model a function of its own history, so a gradual corruption in the data can be carried forward indefinitely without ever showing up as a training failure.

Train one model from scratch each month as a control, and compare the two.

Stage 7 — Offline metrics

The fifth question — how do you know it works — has two halves, and the first is the one you can compute from logged data without shipping anything.

The metric theory itself is derived in ml 06 and should not be re-derived in the room. What belongs in this round is knowing which metric each kind of decision demands.

Metric terms in one line each

• ROC-AUC — area under the receiver-operating-characteristic curve. Equal to the chance a random positive scores above a random negative.
• PR-AUC — area under the precision-recall curve. Unlike ROC-AUC it does not flatter a model when positives are rare, because it never rewards being right about the abundant negatives.
• F1 — the harmonic mean of precision and recall. A convenience number with no connection to what your errors cost.
• Log loss — the penalty for a stated probability being both wrong and confident.
• Reliability diagram — plots predicted probability against observed frequency. A well-calibrated model traces the diagonal.

Which metric for which decision

Find the row describing your decision, use that metric, and stop arguing.

Two things the metrics chapter does not cover

1. Report per slice, always.

An aggregate metric hides the slice that will produce the incident. Cut the data along the dimensions where you expect the model to behave differently:

• new versus established users
• popular versus rarely-seen items
• each language and country
• each class of device

Report the worst slice right next to the mean. Metrics and why map is the wrong headline number goes further and makes the worst slice the pass condition itself, which is the strong form of this.

2. Have a baseline you would actually be embarrassed to lose to.

A baseline here is a deliberately cheap alternative system whose score your model has to beat before any of your work counts. Candidates: ranking by raw popularity, predicting yesterday’s value, always predicting the most common class, and whatever rules engine the company already runs.

A model that beats “random” is not evidence of anything at all. A model that merely ties popularity is a very expensive popularity ranker.

Careful — the word “baseline” is about to mean something else. In Stage 8, a baseline rate is the control arm’s current measured value of a metric (“baseline play-through rate 4.20%”). That is a number, not a system, and it is what an effect size is stated relative to. Both usages are standard and they are not the same thing.

The offline gate, written out for the running problem

An offline gate is the set of pass conditions a model must clear on logged data before it is allowed near live traffic.

It is not one metric. It is a table with one pass condition per row, and writing it down in advance is what stops a launch argument three weeks later.

Every stage of the pipeline gets its own metric. Two rows are worth watching as you read: the retrieval row is deliberately not a quality metric, and the calibration row deliberately has no gate at all.

The row that is easiest to leave out is the one that says “not gated.” Saying out loud which properties you are deliberately not measuring — calibration here, because nothing downstream consumes the probability — is a stronger signal than adding another metric, and it follows directly from the answer you gave to framing question 1 forty minutes earlier.

Stage 8 — Online metrics and why they disagree with offline

The other half of how do you know it works requires live traffic — and it is on live traffic that an offline win can evaporate, for four structural reasons.

You need three tiers of metric, and all three do different jobs. The Property column is why you cannot collapse them into one:

Why the offline number improved and the online number did not

There are four structural reasons an offline gain fails to appear online, and you should be able to name all four without being prompted. None of them is a bug; all of them are consequences of evaluating on logs produced by a different system.

1. Position bias. The offline set was logged under the old ranker, so relevance labels are entangled with the old ranker’s positions. An offline metric computed on those logs is partly measuring agreement with the incumbent.

2. The distribution changed because you changed it. Offline eval assumes the data distribution is fixed. Deploy a new ranker and it shows different items, which produces different clicks, which is a different distribution (Why offline ranking metrics disagree with online ctr derives the mechanics). Offline evaluation of a policy that changes its own data is fundamentally extrapolation.

3. The improvement was on a slice nobody sees. Offline evaluation sets are usually sampled evenly across items, while real traffic follows a power law — a small head of items takes most of the impressions and a very long tail takes the rest. A 3-point NDCG gain concentrated in that tail, in a mix where 80% of impressions go to the head, moves the online number by almost nothing.

4. Something downstream clamps it. A diversity rule, a business rule, a cap on how many results one seller may occupy. The ranker genuinely got better and the policy layer threw the improvement away before a user saw it.

Sizing the A/B

The A/B has to be sized before it runs, and the running problem’s answer is the reverse of the usual worry: the fast metric has far too much data rather than too little.

Three statistical terms first:

• Alpha — the false-positive rate you accept. 0.05 means a 1-in-20 chance of declaring a win that is not real.
• Power — the probability of detecting a real effect of the stated size. 0.80 means you would catch it four times in five.
• Delta — that stated effect size, in absolute terms.

Now the sizing. Note the conversion on the first two lines: a “2% relative lift” on a 4.20% baseline is 0.042 × 0.02 = 0.00084 in absolute terms, and it is the absolute number the formula wants.

baseline play-through rate 4.20%, want to detect a 2% relative lift
  (0.084 points absolute)
alpha 0.05, power 0.80

n per arm ≈ 16 · p(1-p) / delta^2
          = 16 × 0.042 × 0.958 / (0.00084)^2
          = 0.6438 / 7.06e-7
          ≈ 912,000 impressions per arm

The 16 is not a constant of nature. It is

2 · (z_{alpha/2} + z_{beta})^2

evaluated at the alpha and power stated one line above it. z is the standard normal cut point — how many standard deviations out you have to go to leave a given tail probability. So z = 1.96 leaves 2.5% in each tail and gives the familiar 95% interval.

Evaluate it at two power levels:

alpha 0.05, power 0.80    2 × (1.96 + 0.8416)^2  =  15.70   -> "16"
alpha 0.05, power 0.90    2 × (1.96 + 1.2816)^2  =  21.01   -> "21"

So moving from 80% power to 90% costs 34% more traffic (21.01 / 15.70 = 1.34). If you quote 16 without knowing where it came from, you cannot answer that follow-up.

The leading 2 is the two arms. Everything else is the two-sided test plus the power requirement.

The proxy metric: too much data, not too little

Now spend that formula on the running problem, where the answer is the opposite of what candidates expect. At 2.16 billion impressions a day, the fast proxy metric is not limited by sample size at all:

allocate 1% of traffic to the experiment, split 50/50
  impressions/arm/day   2.16e9 × 0.01 / 2   =  10,800,000
  time to 912,000       912,000 / 10.8e6    =  0.084 day  =  2.0 hours

The experiment reaches statistical significance in two hours.

So the readout time for the proxy metric is decided by novelty, day of week and who happens to be online — not by sample size. Anyone who ships at hour three because “it is significant” has measured Tuesday morning and nothing else.

The north star metric: the opposite problem

Retained users, gross merchandise value and tasks completed are per-user quantities. So they must be sized in users and randomized on users. Sizing them in impressions is exactly the mistake warned about in the bullets below.

Take D7 retention — the fraction of users who come back on the seventh day after joining — at a 42% baseline. A 2% relative lift in retention is not something anyone ships, so size for 0.2%:

delta  =  0.002 × 0.42  =  0.00084 absolute
n per arm  =  16 × 0.42 × 0.58 / 0.00084^2  =  5.52M USERS per arm

available at 18M DAU, split 50/50   =  9M users per arm

It is just barely powered, and only if the experiment consumes essentially the whole population. And a 7-day retention number cannot exist before day 7 in any case.

So these two metrics are not two rows of the same table:

• The proxy gates the launch, on a 1% traffic slice over a few days.
• The north star is a whole-population question you answer on the permanent 1% holdback, over quarters (Stage 9).

State both, say which one gates the launch, and say explicitly that the other one is not a gate. Offering a north-star readout you cannot power is worse than not offering one.

Three things to volunteer unprompted

All three are derived in Ab testing:

• Randomize on the unit the effect acts on. Splitting by individual impression when the change is a per-user personalization understates the variance and manufactures significance that will not replicate, because the many impressions belonging to one user are not independent observations.
• Interference between arms. In a marketplace, promoting an item to the treatment arm removes that inventory from the control arm, so the arms are no longer independent. Splitting by user does not fix it. You need to randomize whole clusters, such as a city or a region, or use a switchback design that flips the whole system between treatment and control in alternating time blocks.
• Novelty and primacy effects. Users react to anything new, in both directions, and the reaction fades. Read the result at week 3, and commit to that in writing before you start, so the decision is not made by whoever looks first.

Stage 9 — Serving

The fourth of the five questions — how is it served — comes in three parts: where the prediction is computed, how the features reach it without changing meaning on the way, and how the model gets from zero traffic to all of it without breaking anything.

The three inference modes

There are exactly three places a prediction can be computed, and the choice is decided by which features you need rather than by taste.

• Batch — computed on a schedule and stored, so serving is a lookup.
• Online — computed inside the request, while the user waits.
• Streaming — computed as events flow past, seconds after they happen.

“Online” has just changed meaning, and nothing in the industry’s vocabulary warns you.

In Stage 8, an online metric is one measured on live traffic, as opposed to an offline metric computed on logged rows. That is a statement about where the evaluation happens.

Here, online inference means the prediction is computed inside the request, as opposed to precomputed in batch. That is a statement about where the computation happens.

Both usages are standard, they are unrelated, and a batch-served model is still evaluated online. Say which one you mean the first time you use it in the room.

Compare the three modes on every axis that matters. The row that decides the architecture is Freshest feature:

The right default is more batch than people think.

If a prediction depends only on features that change daily, precompute it and serve a lookup. Roughly 100x cheaper per prediction, and the p99 latency is a cache read.

The online path exists only for features that arrive with the request: the query text, the current session, the item being viewed right now. So say which features force you online, and if none do, do not build an online path.

Running that test on the running problem

Go back to the Stage 4 split and check each family:

• The 140 item features — all precomputable.
• The 40 user features — all precomputable. Computing user affinities nightly is fine, because nobody’s topic preferences turn over in an hour.
• The 20 context-and-cross features — this is where the online path comes from, and only some of them force it. Time of day and device could be handled by precomputing a few variants. Session position, slot index and user × item affinity cannot, because they depend on what this user has already been shown in this session.

So the honest answer is a hybrid. And it falls out of the source table in Stage 5 rather than being a separate decision:

• The ANN source can be precomputed. A job riding the hourly index rebuild writes out each user’s top few hundred ANN candidates in advance. That is legitimate because the user embedding it queries with is computed nightly and does not move between rebuilds.
• The recent-from-followed and exploration sources must be assembled at request time, since freshness is their entire purpose.
• The online path then re-ranks the union of all of them, using the session-dependent features.

A stage whose inputs are all precomputable should itself be precomputed, and that test is applied per source, not per system. Running it here moves the largest retrieval source off the request path, and leaves the 16 machines in Stage 10 sizing the ranker and nothing else.

Feature stores and train/serve skew

Train/serve skew is the class of bug where the number a feature had during training is not the number it has at serving time, so the model is fed something it never learned on.

There is a standard architecture for preventing it, and one check that actually detects it. The underlying mechanism and worked traces are in Feature stores and trainingserving skew; here is the design-round version.

One event source feeds two transform paths, batch and stream. The load-bearing edge is the dotted line at the bottom, which loops the served features back through the training-time transform — the only part of the architecture that detects skew rather than merely reducing it.

flowchart TD
    SRC[("Event log")] --> TR["Batch transform<br/>point-in-time correct"]
    SRC --> ST["Stream transform<br/>same code path"]
    TR --> OFF[("Offline store<br/>columnar · history")]
    ST --> ONL[("Online store<br/>KV · latest value")]
    OFF --> TRAIN["Training<br/>join AS OF prediction time"]
    ONL --> SERVE["Serving<br/>read latest"]
    TRAIN --> REG["Model registry<br/>version + feature schema hash"]
    REG --> SERVE
    SERVE --> LOG[("Prediction log<br/>features AS SERVED")]
    LOG -.->|"skew check:<br/>replay logged features<br/>through the offline path"| OFF

    style ONL fill:#bc6c25,color:#fff
    style REG fill:#1d3557,color:#fff
    style LOG fill:#2d6a4f,color:#fff

Walk the boxes:

• Event log — the append-only record of everything that happened. One source, feeding both paths.
• Batch transform and stream transform — these must run the same code path. If they are two separate implementations of “30-day average,” you have shipped the skew rather than prevented it.
• Offline store — where the batch side lands. Held columnar (organized by column rather than by row, which is what makes scanning one feature across billions of rows fast) and keeping full history, because training needs to ask what a feature was on an arbitrary past day.
• Online store — where the stream side lands. A key-value store, meaning a simple lookup from an identifier straight to its current values. It keeps only the latest value, because serving only ever asks about now.
• Training joins the offline store as of each row’s own prediction time. Serving reads the latest value.
• Model registry — what stops those two from drifting apart. It pins a model version together with a hash of the feature schema it was trained against — a short fingerprint that changes if anything about the schema changes — so a model can never be served against a feature set it never saw.
• Prediction log — writes out the feature vector as served, not as recomputed afterwards. It is the only artifact that makes the dotted line possible.

The dotted line is the whole design. Log the feature vector exactly as it was served, replay it through the training-time transform, assert equality.

Anything else — shared code, careful review, a schema registry — reduces skew. Only the replay assertion detects it. Budget for it explicitly, because it is one of the highest-yield pieces of ML infrastructure and it is usually the first thing cut.

There are three kinds of skew, and the replay check catches two of them:

Rollout: shadow, canary, ramp

A model does not go from zero to full traffic in one step. It goes through four stages, each of which can detect a different class of problem — and each of which is blind to something only another stage can see.

The share of traffic served rises from 0% to 100% across the four stages. Rollback, hanging off the end of the sequence, is the exit, not a fifth stage.

flowchart LR
    A["1 Shadow<br/>100% traffic, 0% served<br/>compare scores, not outcomes"] --> B["2 Canary<br/>1% served<br/>watch guardrails only"]
    B --> C["3 Ramp<br/>5% -> 20% -> 50%<br/>the A/B reads out here"]
    C --> D["4 Full<br/>+ holdback 1% forever"]
    D -.->|"regression on any guardrail"| E["Rollback<br/>previous version pinned<br/>in the registry"]

    style A fill:#1d3557,color:#fff
    style B fill:#bc6c25,color:#fff
    style C fill:#40916c,color:#fff
    style D fill:#2d6a4f,color:#fff
    style E fill:#9d0208,color:#fff

Walk the four boxes:

1. Shadow — the new model scores 100% of traffic and serves 0% of it. You compare scores against the incumbent, not outcomes, because nobody saw the output.
2. Canary — serve the new model to 1% of traffic. Watch guardrails only, looking for the kind of breakage that shows up immediately.
3. Ramp — step the share up through 5%, 20% and 50%. The A/B reads out here, because this is the first stage with enough traffic to trust.
4. Full — serve everyone, keeping a 1% holdback forever on the old system.

The dashed edge is Rollback, triggered by a regression on any guardrail. It works by repinning the previous version in the model registry.

What each stage can and cannot tell you. Read the Blind to column first — it is the one that explains why you cannot skip a stage:

The dashed edge out of the diagram is the part to design before you need it.

A regression on any guardrail triggers rollback, and rollback means repinning the previous version in the registry — not redeploying a build.

Those are different operations with different latencies:

• Repinning is a config change. Seconds, and reversible.
• Rebuilding is a pipeline run. An hour, and it reintroduces every risk you just backed out of.

The registry keeps the previous model and its feature schema hash live and warm for exactly this reason. A rollback plan that does not name which artifact gets repinned is not a plan.

The permanent holdback is the only way to answer “is the sum of 40 shipped 0.3% wins actually a 12% win?” The usual answer is no, because wins overlap and metrics drift, and without a holdback nobody ever finds out.

Stage 10 — Scale and cost: the back-of-envelope template

Now the design turns into machines, and the machines into dollars. It takes five lines of arithmetic, and you should say them out loud rather than write them silently.

The whole point is that two independent paths get computed separately — how much arithmetic the fleet must do, and how many bytes it must move — and whichever is larger sizes the fleet. It is almost never the arithmetic.

Here is the template. Lines 3 and 4 are the two paths; line 5 takes the larger.

1. requests/day       =  QPS × 86,400
2. item-scorings/s    =  QPS × candidates_per_request        <- the sizing number
3. feature bytes/s    =  item-scorings/s × features × bytes
4. FLOP/s             =  item-scorings/s × FLOP_per_item
5. machines           =  max(FLOP path, bytes path) × headroom

Headroom on line 5 is the multiplier that leaves room for traffic spikes, for a machine dying, and for the difference between a typical request and a slow one. It is not optional and it is not decoration.

Worked: 5,000 QPS, 1,000 candidates, 200 features, 852 kFLOP/item

State the machine once, because every line below is denominated in it:

• 32 vCPU — virtual CPU cores, one hardware thread each.
• 128 GB of RAM.
• A 25 Gbps NIC — network interface card, the chip that connects the machine to the network, here rated at 25 gigabits per second.
• $0.35 per machine-hour.

That price is a committed-use or spot rate. The on-demand list price for that shape is roughly 4x, and if you are pricing a design that has to survive a procurement conversation, use the list price and say that you did.

The block below runs both paths. The compute path is computed twice: once off FLOPs, then again off measured wall-clock time. The second one is the real answer, and the paragraph after explains why.

item-scorings/s  =  5,000 × 1,000            =  5,000,000 /s

compute path
  5e6 × 852e3 FLOP                           =  4.26 TFLOP/s
  at 50 GFLOP/s per core (batched, AVX-512)  =  85 cores     <- FLOPs ONLY
  but the measured cost is 40 us/item, not the 17 us of FLOPs
  5e6 × 40 us                                =  200 core-seconds/s
                                             =  200 cores    <- true demand
  × 2.5 for p99 headroom and failure domains =  500 cores
  at 32 vCPU/machine                         =  15.6  ≈  16 machines

feature path
  5e6 × 200 features × 4 bytes               =  4.0 GB/s
  over a 25 Gbps NIC (3.1 GB/s usable)       =  1.3 NICs of pure feature traffic
  as random KV reads at 100 B/read           =  40M reads/s across the fleet

The two-step on the compute path is the single easiest place in this chapter to fool yourself.

The 85 cores is a count of arithmetic. The 200 cores is a demand for wall-clock time. They are not the same quantity, and the gap between them is exactly the 40 / 17.04 = 2.35 ratio of measured cost to FLOP cost.

Consider what happens if you apply headroom to the arithmetic-only figure:

85 cores × 2.5 headroom  =  213 cores
                            213 / 200  =  1.065

That looks like 200 cores plus 150% headroom. It is in fact 200 cores with 6% headroom. The multiplier has been almost entirely eaten by the embedding gathers and feature assembly it was supposed to sit on top of.

Apply headroom to measured wall-clock demand, never to a count of floating-point operations, or the headroom you think you have is almost nothing.

Sized honestly this fleet is 16 machines; sized off the arithmetic it would be 7. That 2.3x increase strengthens the section’s conclusion rather than weakening it: the compute path just became 2.3 times more expensive, and the feature path is still the one that binds.

Three design decisions that follow immediately

1. Co-locate item features with the ranker. Keep them in the ranking machine’s own memory rather than fetching them over the network.

From the Stage 4 split, 140 of the 200 features are item-side:

20M items × 140 features × 4 bytes  =  11.2 GB

That fits inside 128 GB of RAM on every copy of the ranking service, with plenty of room for the model itself and the operating system’s file cache. Sixteen copies hold 16 × 11.2 = 179 GB of duplicated data between them. You are paying for the same table sixteen times, and it is free, because you already bought that RAM when you bought the cores.

2. Fetch user features once per request, not once per candidate.

The 40 user-side features are identical across all 1,000 candidates, so the naive implementation fetches them 1,000 times over. Fetched once instead:

5,000 QPS × 40 features × 4 bytes  =  0.8 MB/s

Against the 4.0 GB/s you started with, the two fixes together cut feature traffic by a factor of 5,000 (4.0e9 / 0.8e6 = 5,000). The remaining 20 context-and-cross features are computed on the machine itself, from the two vectors already in hand.

3. Quantize the item features.

Quantizing means storing each number in fewer bits — here going from 32-bit floating point to 8-bit integers, a quarter of the size. That shrinks the resident table from 11.2 GB to 2.8 GB and roughly quadruples how much of it stays in fast cache.

What is being quantized is the stored feature values, not the model’s weights. The ranker is a neural network that does arithmetic with every input, so the accuracy cost has to be priced rather than waved through. Price it in four steps:

1. Give each feature its own scale and zero point, so its 255 steps span that feature’s own observed range.
2. Take that range as roughly ±3 sd, so the whole span is 6 sd and one step is 6 sd / 255.
3. Rounding error is uniform on one step, and the root-mean-square of a uniform error on a step of width w is w / sqrt(12). So the error is 6 / (255 × sqrt(12)) = 0.68% of the feature’s standard deviation.
4. Every pre-activation in the first layer is a weighted sum of those inputs. The per-feature errors are independent and zero-mean, so noise and signal both scale with sqrt(Σ w²). That means the 0.68% lands on the pre-activation however many of the 512 inputs it sums — it does not accumulate.

0.68% sits far below the sampling noise already inside a CTR counter estimated from a few hundred impressions, which is why this is cheap.

The one thing that breaks the argument is a heavy-tailed feature. A raw count whose range is set by a single outlier collapses every ordinary value into two or three buckets, so log- or rank-transform those slots before quantizing them.

And verify rather than argue: re-score a held-out slice both ways and compare NDCG@10. That is the only form of this claim an interviewer should accept.

Two things co-location quietly changes

Volunteer these rather than being caught by them.

• The constraint that sizes the fleet moves from cores to memory per machine. You can no longer scale out with lots of small machines, because every one of them must hold the whole item table.
• The Stage 0 eligibility filter does not help here. The cut from 20M to 4M happens per request, so the table sitting in memory is still all 20M items — any of them may be eligible for somebody.

The function below is the five-line template as code, returning both paths so you can see which one binds. Its printed output follows.

def size_ranking_fleet(qps, candidates, per_item_us, features_per_item,
                       bytes_per_feature=4, headroom=2.5, cores_per_machine=32):
    """Five-line back-of-envelope for a two-stage ranking service.

    per_item_us is MEASURED wall-clock per scoring, including embedding
    gathers and feature assembly -- not FLOPs / GFLOP-per-core. Sizing off
    FLOPs and then multiplying by a headroom factor double-counts: the
    factor gets eaten by the non-FLOP work and you ship with none.

    Returns both paths so you can see which one binds. In practice the
    feature path binds far more often than the compute path, which is why
    'how many GPUs' is usually the wrong first question.
    """
    scorings = qps * candidates
    cores = scorings * per_item_us * 1e-6
    feature_bytes = scorings * features_per_item * bytes_per_feature
    return {
        "item_scorings_per_s": scorings,
        "cores": cores * headroom,
        "machines": cores * headroom / cores_per_machine,
        "feature_GB_per_s": feature_bytes / 1e9,
        "NICs_at_3.1GBps": feature_bytes / 3.1e9,
    }


fleet = size_ranking_fleet(5_000, 1_000, per_item_us=40, features_per_item=200)
for k, v in fleet.items():
    print(f"{k:22s} {v:,.2f}")

item_scorings_per_s    5,000,000.00
cores                  500.00
machines               15.62
feature_GB_per_s       4.00
NICs_at_3.1GBps        1.29

The cost of a percentage point

The arithmetic finishes by converting engineering into money, which is what turns a design into a decision.

Add up every recurring line, in dollars per month, and see which one dominates. A GPU-hour is one hour on one graphics processor, the standard unit training compute is billed in. The × 4.3 on the training line converts weeks to months.

The serving line unpacks as 16 machines × $0.35/hr × 24 hr × 30 days = $4,032. The labeling line is 20,000 × $0.92 = $18,400, using the per-item price derived in Stage 3.

16 machines × $0.35/hr × 24 × 30         =  $ 4,032/month serving
training: 40 GPU-hours/week × $2 × 4.3   =  $   344/month
feature store: 11 GB hot, 4 TB history   =  $   900/month
labeling: 20k items/month at $0.92       =  $18,400/month
                                            ----------
                                            $23,676/month

-> labeling is 78% of the run cost. The optimization worth doing
   is active learning, not a smaller model.

Sizing off arithmetic alone would have given 7 machines; sizing off measured wall-clock gives 16 — a 2.3x difference that moves this total by only 11% and changes no conclusion.

Check that 11%: seven machines cost 7 × 0.35 × 720 = $1,764, so the total would have been $21,408 instead of $23,676, and 23,676 / 21,408 = 1.11.

That is the useful thing about doing the arithmetic properly rather than approximately: it tells you when a factor of two does not matter. Labeling still dominates everything else combined by 3.5x (18,400 / 5,276).

And here is the line that makes the round cohere. The 20,000 items a month are the same satisfaction-labeled sample that the composite weights in Stage 2a are fit on. So the largest line in the cost model exists to answer the objective question from Stage 2, not to train the model at all.

Say that out loud. The follow-up it earns — “so how would you cut it?” — has a real answer: active learning on the items where the three composite terms disagree with each other, because those are exactly the items where the weights are least determined.

Stage 11 — Failure modes

The last part of how do you know it works is the failures a working system has: how each one announces itself, and what you would turn to fix it.

Use the four columns every chapter in this track uses — Failure · Mechanism · Detection · Control — and fill the third one hardest.

Anyone can name a failure and most people can propose a fix. The column that separates candidates is how you would find out, because a failure you cannot detect is one you will ship.

The fourth column is Control rather than Guard deliberately. A guard is a fence you build once; a control is a thing you can turn. An interviewer asking “what would you do about it” is asking which knob.

Two words used in the table:

• decile — one tenth of a sorted population. “The bottom item-frequency decile” is the 10% of items that get shown least often.
• entropy — a measure of how spread out a distribution is. Maximum when every item gets an equal share of impressions, falling as a few items take everything.

Ten rows follow. Do not try to memorise them as ten — the three paragraphs after the table collapse them into four ideas.

Ten rows collapse into fewer problems

Ten rows is more than anyone can hold in their head. The senior move is to point out that they are not ten separate problems.

Rows 1, 2 and 4 are one problem arriving by three routes. Train/serve skew, label leakage and silent feature death all say “the feature at training time is not the feature at serving time.” The three routes are: two implementations of the same logic, one wrong time anchor, and one dead upstream job.

That is why the replay assertion in Stage 9 is worth more than any three monitors. It is a single mechanism covering three of these ten rows, and it is always the line item that gets cut.

The feedback loop, cold start and tail recall collapse are one problem seen at three points in the pipeline:

• Cold start is the loop closing at retrieval.
• Tail recall is the loop already closed and now visible in the embeddings.
• Drift in impression entropy is the same loop at the level of the whole system.

They share one control — exploration with logged propensities — which is exactly the 4% of engagement priced in Stage 2e. That 4% now looks cheap, because one line item is the control for three of the ten rows.

The one to volunteer unprompted is still the feedback loop, because it is the only failure here with no offline signature at all.

Every other row eventually surfaces in a metric somebody is already watching. The loop looks like success the whole way down: engagement is fine, AUC is fine, the model is scoring well on data it selected for itself, and the range of items it can see is steadily narrowing.

Stage 12 — Alternatives rejected

Naming what you did not build, and why, is worth more than another component — provided the table reads as judgement rather than as a rehearsed list. Four rules keep it that way:

1. Reject on a number. “Too slow” is weak; “800 seconds against a 100 millisecond budget” is strong.
2. Name what is genuinely good about the alternative first, before the rejection, or the rejection reads as a reflex.
3. Include one you would revisit, and give the trigger that would make you. For example: “revisit end-to-end learned retrieval above 50 million items, or when tail recall drops below 0.5.”
4. Include the embarrassingly simple thing, meaning popularity ranking, a rules engine, or just repeating yesterday’s answer. If you did not consider it, the interviewer will, and they will be right to.

Here is the table for the running problem. Every number in the Rejected on column was derived in a stage above rather than invented at the end — that is what makes it a rejection rather than an opinion.

And the refusal to state out loud at least once, because rejecting complexity on a number scores better than adding it:

“I am not building a learned blender over the four candidate sources. The whole quota table is four numbers and I can move them in an afternoon; a learned blender needs a label for slot allocation that does not exist, and it would be trained on the impressions the current quotas produced. I would rather spend that engineer on the replay assertion, which catches a third of my failure table.”

Stage 13 — Interviewer pushback

These are the eight questions this design invites, with the answer written the way you would say it.

Each one names what the question is really testing, because the point is not the answer but knowing which quality is being probed. Every number in the answers was derived in a stage above.

“Why not just score every item with the good model?”

Testing: whether the two-stage pattern is a memorized name or an arithmetic result.

Because it is 8,000 times over budget. A full-feature scoring costs about 40 microseconds — 852 thousand floating-point operations of neural network, plus the embedding lookups — and 20 million items is 800 seconds against a 100 millisecond budget.

Even the cheapest possible full scan, a 64-number dot product with no feature fetch at all, reads 5.1 GB per query and takes 51 milliseconds limited by memory bandwidth. That is half the budget at a single query per second.

So stage one has to be sublinear, not merely cheap: a graph index touches around 3,000 vectors instead of 20 million. And that is exactly why stage one is restricted to scores that split as f(user)·g(item) — nothing else can be indexed.

“Your offline NDCG went up 4 points and the A/B is flat. What happened?”

Testing: whether you know the gap is structural.

Four candidates, and I would check them in this order:

1. The offline labels were logged under the old ranker, so the metric partly measures agreement with the incumbent. I would check by re-scoring a randomized-position slice.
2. Offline evaluation of a policy that changes its own data is extrapolation. The new ranker shows different items and therefore lives on a different distribution.
3. The gain may be concentrated in a slice with almost no impressions, since offline sets are uniform and traffic is power-law. Segment the online metric by the same buckets as the offline set.
4. Something downstream clamps it: a diversity rule or a per-seller cap that discards the improvement before a user sees it.

I would look at the fourth first, because it is the cheapest to check.

“The labels arrive 90 days late. Design around it.”

Testing: whether label latency produces a design change.

First I ask what the prediction drives, because the answer differs.

Ranking survives delay. Missing positives are roughly uniform across the score range, so the order is intact.

Calibration does not. Over a 0-30 day training window the average maturity is 0.30 — not the 0.32 you get by evaluating the curve at the mean age, which is optimistic because maturity is concave. So the observed rate is 0.30x the truth, and every cost-derived threshold is off by 3.3x.

If I need a probability, I run two models: a fresh ranker on censored recent data, plus a matured calibration map fit on data older than 90 days. And I weight young negatives by the maturity curve rather than treating them as clean zeros.

Then I monitor predicted rate against matured actuals at the 90-day mark, which is the only honest calibration check available.

“Give me the fleet size. Now.”

Testing: arithmetic under pressure, and whether you know where the cost sits.

At 5,000 QPS with 1,000 candidates that is 5 million item-scorings per second.

The FLOPs say 852 kFLOP each, 4.3 TFLOP/s, 85 cores at 50 GFLOP/s. I would not size off that, because the measured per-item cost is 40 microseconds and only 17 of those are FLOPs. So the real demand is 5 million × 40 microseconds = 200 cores, and with 2.5x for p99 and failure domains that is 500 cores, or 16 machines at 32 vCPU.

If I had multiplied the 85 by 2.5 I would have said seven machines, and shipped with six per cent of headroom instead of a hundred and fifty.

But the binding number is still the other one. 5 million scorings × 200 features × 4 bytes is 4 GB/s of feature traffic, more than a 25-gig NIC. So this is a feature-serving problem with a model attached.

The fix is two moves. Co-locate — 140 of the 200 features are item-side, so 20M items at int8 is 2.8 GB resident, which fits on every replica. And fetch the 40 user features once per request rather than once per candidate. Together that is a five-thousand-fold cut in feature traffic.

“Your model is 94% accurate. Ship it?”

Testing: whether a headline number survives contact with you.

Not on that number, for three reasons.

1. What is the base rate? If 94% of rows are negative, then 94% is exactly what you get by always saying “no.”
2. What do the two kinds of error cost? Accuracy weights a false positive and a false negative equally, and almost nothing in production does.
3. What is the worst slice? An aggregate of 94% is entirely consistent with 71% on new users or in one country, and that slice is where the incident will come from.

I would want PR-AUC if positives are rare, a threshold derived from the cost matrix, and a slice table with the worst cell as the pass condition.

“How do you get labels with no budget?”

Testing: whether you can start.

Three sources, in this order.

1. Implicit product signal is free and unlimited. I would take it while being explicit that a non-click is not a negative: I sample negatives rather than assume them, and I log the position of every result so I can weight by the inverse examination probability later.
2. Weak supervision. Five or six cheap rules drawn from existing business logic, text patterns, and any model already in production, combined by looking at where they agree. That typically yields 0.86-accuracy labels over millions of rows for about a week of work.
3. Whatever small human budget exists, spent entirely on the evaluation set rather than the training set. A thousand carefully adjudicated rows that let me decide something, rather than fifty thousand noisy ones that let me train marginally better.

“Everything looks fine in the dashboards and the product is getting worse. Where do you look?”

Testing: whether you know the failure with no offline signature.

The feedback loop. Every model metric is computed on data the model itself selected, so a system that is narrowing its world scores well the whole way down.

Two things I would look at that are not model metrics:

• The entropy of the impression distribution over time. If the head’s share went from 61% to 78% over eight weeks, the loop is closing.
• Catalog coverage — the fraction of items with any impression in seven days.

The structural fixes are an exploration budget with logged propensities, and a permanent 1% holdback that has never been touched by the model. That holdback is the only thing that can answer whether forty shipped 0.3% wins actually add up.

“Batch or online?”

Testing: whether you default to the expensive one.

Whichever one the features force.

If every input changes at most daily, I precompute and serve a lookup: roughly 100 times cheaper per prediction, and the p99 latency is a cache read.

The only reason to be online is a feature that arrives with the request — the query text, the current session, the item the user is looking at right now. So I list the features, mark which ones are request-time, and if none are, I do not build an online path at all.

The middle case is streaming, for features that must be seconds fresh, such as “how many transactions has this card attempted in the last minute.” Its characteristic failure is a silently growing backlog, so monitor how far behind the consumer is, not how much throughput it is achieving — a healthy-looking throughput number is exactly what a saturated consumer produces.

The whole round on one problem, in numbers

Here is the chain of derivations, laid end to end — the thing you should be able to reproduce from memory.

Read it as three columns doing three jobs: what the number is, the arithmetic that produced it, and what it forced next. Every figure was derived in a stage above, out of a figure derived in a stage before that.

This table, not the architecture diagram, is the artifact of the round. If you can produce a chain like it, you have run the method. If your numbers do not depend on each other, you produced thirteen sections rather than one design.

Follow the last row backwards and the whole round is in it:

1. The dominant cost is labeling.
2. The labeling exists to fit the composite weights.
3. The composite weights exist because clicks invert 9.3% of pairs.
4. The inversions matter because the ranker sorts rather than averages.

Four steps from a cost table to the sentence in Stage 2 that the round was actually about. A candidate who can walk that chain in either direction has demonstrated something no amount of architecture vocabulary substitutes for.

When you are behind

The method is only useful if it degrades gracefully, so it comes with a recovery plan: one checkpoint, three possible positions, and a fixed order in which to cut material.

The whole plan is one decision, made once, at minute 26. Where you are at that moment branches three ways; two of the branches funnel into a cut list; and the box at the bottom is the part you protect no matter which branch you took.

flowchart TD
    T{"Minute 26:<br/>where are you?"} -->|"still framing or on the objective"| P1["Emergency: state all five<br/>framing answers as assumptions,<br/>skip features, go to candidates"]
    T -->|"on candidates or the model"| P2["Compress: name the model family<br/>in one line, go to metrics"]
    T -->|"on metrics"| P3["On track"]

    P1 --> CUT["Cut order:<br/>1 the feature catalog<br/>2 training details<br/>3 alternatives rejected<br/>4 the A/B sizing arithmetic"]
    P2 --> CUT
    CUT --> KEEP["Never cut:<br/>the label-arrival answer<br/>one operating point or candidate<br/>count derived from arithmetic<br/>the failure with no offline signature"]

    style KEEP fill:#2d6a4f,color:#fff
    style P1 fill:#9d0208,color:#fff

Minute 26 is the checkpoint because it is the boundary between the half of the round that sets up a decision and the half that defends one. There are three places you can be when you look at the clock, and each has a different recovery.

Still framing or on the objective. This is the emergency, and the recovery is to stop asking and start asserting: state all five framing answers as assumptions in one breath, skip features entirely, and go straight to candidates. That is the same move as system-design 03, and it works for the same reason — a wrong assumption you flagged produces a local correction, and a missing section produces a missing score.

On candidates or the model. Compress rather than skip: name the model family in one line with one clause of justification, drop the family table entirely, and move to metrics. Nobody is scoring your architecture selection; they are scoring whether the metric matches the decision.

On metrics. You are on track, and the thing to protect is the last four minutes rather than the next four.

Never cut three things:

1. The label-arrival answer.
2. One number derived rather than chosen.
3. One failure mode with no offline signature.

Those are the three places this round differs from every other design round. A candidate who covers all thirteen stages shallowly scores below one who covers eight and has those three.

At minute 38, offer the interviewer the choice: “I have the failure table and the cost model left — which is more useful to you?”

Stage index: which chapter instantiates each stage best

This chapter is the method; the problem chapters are the worked instances. When you want to see a stage done properly rather than described, here is the best worked version of each:

If you read one problem chapter after this one, read 08, because it is the 1:1 instantiation of this spine; then 03, because it is the clearest case of the operating point being the entire design.

The scoring rubric for this round

This is what the interviewer is actually writing down, signal by signal.

Each row gives the weak version and the strong version of the same signal. Read it as a checklist against your own last mock round: for each row, ask which column your last answer landed in.

The one-line version: find the label before you find the model, check whether the target is the thing you actually want, derive every threshold and every candidate count from arithmetic, and assume the system is corrupting its own training data until you have built the thing that proves otherwise.

Cheat sheet

One page, indexed by where you are on the clock, for the last review before the round.

The Clock column matches the minute table in The spine, so you can find the row for wherever you are. Every line here is derived somewhere above.

Next: 02 — Visual Search — the two-stage pattern made concrete, and the chapter where the loss function itself has to be derived.