06 — Video Recommendation

Build the home feed: what videos do we show this user next? The task is a large-scale video recommendation feed. The aim is to derive its shape rather than recite it.

Three ideas run through the chapter:

1. The two-stage funnel is forced, not chosen. A short arithmetic calculation rules out every one-stage design.
2. A single engagement objective is gameable in three separate ways. They are three different mechanisms and each needs its own defence.
3. The training data is produced by the system’s own past decisions. That corrupts both the model and the evaluation unless traffic is spent to prevent it.

The chapter derives how to size the fleet, where each label comes from and when it arrives, and why a two-week experiment systematically gives the wrong answer about exploration.

What goes in, and what comes out

The input is one request: a user id, that user’s watch history, their device, the time, their locale, and how many slots the surface has to fill.

The output is a slate — an ordered list of about 20 videos filling those slots. Each video shown is one impression, and the impression is the unit everything downstream is counted in.

In between, four models run in sequence. Every one of them is derived later in the chapter; this is the roster, not the argument.

Seven words this chapter cannot avoid

Defined once, here, so the rest of the chapter can use them without stopping.

• Embedding — a fixed-length list of numbers standing in for a user or an item, learned so that similar things land near each other.
• FLOP — one floating-point operation. MFLOP, GFLOP, TFLOP, PFLOP and ZFLOP are a million, a billion, a trillion, a quadrillion and a sextillion of them. This is the standard way to price arithmetic.
• QPS — queries per second.
• DAU — daily active users.
• A100 — a specific data-centre GPU, used throughout as the unit of compute you can actually rent.
• MLP (multi-layer perceptron) — the plainest kind of neural network: a stack of matrix multiplications with a nonlinearity between them.
• Head — a small output layer bolted onto a shared network body, one per thing you want to predict.

Three ideas from elsewhere in this repo

Restated here so this page stands alone.

• Bandit — an algorithm for repeatedly choosing between options whose payoffs you only learn by trying them, balancing exploiting what already looks good against exploring what might be better. UCB (upper confidence bound) does this by ranking options by an optimistic estimate. Thompson sampling does it by drawing a plausible payoff from each option’s posterior and picking the winner. Mechanics: Exploration vs exploitation and Contextual bandits the practical middle ground.
• Off-policy evaluation — estimating how well a new policy would have done, using only logs generated by the old one.
• ANN index (approximate nearest neighbour) — a data structure that returns the closest few vectors out of hundreds of millions without comparing against all of them, at a cost roughly logarithmic rather than linear in the corpus. Internals: Vector index internals.

The spine this chapter hangs on is 01 — Framework.

The one thing to say in the first three minutes

This is the canonical recommender question, and it is also the one where candidates most reliably describe an architecture without ever deriving why it has to be that shape.

The two-stage structure is not a design pattern. It is forced by an arithmetic gap of six and a half orders of magnitude. Showing that gap early is worth more than any amount of correct vocabulary about towers and embeddings. Framing and the arithmetic that forces two stages computes it.

1. Framing, and the arithmetic that forces two stages

One calculation determines the entire architecture: what it would cost to score every video for every request. The answer is absurd by six and a half orders of magnitude, and every structural decision in the rest of the chapter is a consequence of closing that gap. The calculation needs the problem’s dimensions first, so write them down completely — four of these five lines are load-bearing later:

• Input: a user id, their history, device, time, locale, and a slot count.
• Output: an ordered slate of ~20 videos.
• Corpus: 800 M eligible videos, +500 k/day.
• Traffic: 500 M DAU × 40 recommendation requests/day.
• Budget: 100 ms end to end, ~80 ms for ML.

The traffic line is the one everything else is multiplied by, so turn it into a per-second number now:

requests per day   500e6 DAU × 40 requests  =  20e9/day
mean QPS           20e9 / 86,400 s          =  231,000/s
peak QPS           assumed 2.2x the mean    =  509,000/s

Traffic is not spread evenly over 24 hours, so the fleet has to survive the busy hour rather than the average one. Every fleet number in this chapter is sized at 509 k/s, not at 231 k/s.

Score everything: the number that ends the discussion

The simplest possible design is to run the good model on every video and sort. Price it, because the price is the argument.

Start with the cost of scoring one (user, item) pair. A serious ranking model takes ~1,500 numbers in — concatenated embeddings and dense features, itemized in Ranking — and pushes them through hidden layers of width 1024, then 512, then 256, then eight small per-task heads.

Each layer is a matrix multiplication. A matrix multiply of n inputs to m outputs costs 2nm operations, because every one of the n × m weights is used once in a multiply and once in an add. That is where the leading 2 comes from:

FLOPs per (user, item) scored
  = 2 × (1500·1024 + 1024·512 + 512·256 + 256·8)
  = 2 × (1,536,000 + 524,288 +  131,072 +  2,048)
  = 2 × 2,193,408
  = 4.39 MFLOP

Now run that model over the entire corpus, once per request, and then over the whole fleet. Three lines, each one multiplying the one above it:

score the whole corpus, one request:
    800e6 items × 4.39e6 FLOP  =  3.51 PFLOP

time on one A100 at 150 TFLOP/s effective:
    3.51e15 / 1.5e14           =  23.4 seconds     (budget: 0.08 s)

arithmetic rate at peak:
    3.51e15 × 509e3 req/s      =  1.79 ZFLOP/s

GPUs required:
    1.79e21 / 1.5e14           =  11.9 MILLION A100s

In one sentence: scoring all 800,000,000 videos with the ranker costs 3.51 PFLOP for a single request, which is 23.4 seconds on one A100 against an 80 ms budget, and 11,900,000 A100s to serve the fleet at peak.

One number in there is not a spec-sheet figure. “150 TFLOP/s effective” is the throughput a real A100 sustains on this kind of work, well below its headline number, because a small batch never fills the machine. Scale and cost shows the same effect costing 1,800x on the real ranker.

Twelve million GPUs to answer one product’s requests. The corpus is 800 M items and a request can afford to score about 300 of them, so the gap is exactly that ratio:

log10(800e6 / 300)  =  log10(2.67e6)  =  6.4 orders of magnitude

No model compression, no quantization — storing weights and activations at lower precision, so each number takes fewer bytes and fewer cycles — and no batching trick closes six and a half orders of magnitude. Those tricks buy 2x to 10x.

The only structural fix is to stop scoring most of the corpus. That means an operation whose cost does not scale with corpus size: a precomputed index lookup, or an ANN query whose cost is roughly logarithmic in the corpus.

The staged budget, derived backwards from the fleet you can afford

If one stage is impossible, the replacement is a funnel of stages, each earning a larger per-item budget than the last.

Read the table left to right as a narrowing: each row takes the previous row’s output as its input, and each row is allowed to spend more per item because there are fewer items left. The last column is what that stage’s arithmetic alone would cost in GPUs at peak traffic.

Two of those rows deserve their arithmetic written out:

full rank:  300 × 4.39 MFLOP × 509,000 /s  =  671 TFLOP/s  =  4.5 A100s

retrieval:  one HNSW query at efSearch=64 touches ~3,000 nodes, each a
            128-d fp16 dot:  3,000 × 128 × 2  =  0.77 MFLOP
            six sources                       =  4.6 MFLOP/request

Two terms in that block need glossing. efSearch is the ANN index’s search-effort dial — how many graph nodes one query is allowed to visit. fp16 is 16-bit floating point, two bytes per number; a 128-dimensional dot product is 128 multiplies and 128 adds, hence the × 2.

Check the retrieval unit, because the unit is the whole story

The retrieval row is the one number in the chapter worth recomputing rather than copying.

At 4.6 MFLOP a request, retrieval costs 4.6e6 × 509e3 = 2.3 TFLOP/s at peak. That rounds to zero against the ranker’s 671, which is why Scale and cost’s FLOP floor omits it entirely.

Now slip one letter and write the same figure as 4 GFLOP:

4e9 × 509e3          =  2,036 TFLOP/s
2,036e12 / 1.5e14    =  13.6 A100s
13.6 / (4.5 + 0.3)   =  2.8x the pre-rank and full-rank stages combined

Under that typo retrieval becomes the largest arithmetic line in the system, which inverts the chapter’s entire thesis — and nothing else in the text changes to warn you.

That is the general hazard: a stage whose cost you assert instead of derive is a stage nobody will audit, and a unit error there is invisible, because there is no second number to disagree with it.

What the funnel bought, and what it did not

Four and a half GPUs of raw arithmetic against twelve million. The two-stage split is that entire difference.

The second half is the counterintuitive part: 4.5 A100s is not what a ranking fleet actually costs. Once the work is cut to 300 items, FLOPs stop being the binding constraint. The fleet is then sized by embedding-table memory, feature fetch, and p99 tail latency, and it lands one to two orders of magnitude above the FLOP floor — Scale and cost works out the real number, 108 nodes.

The arithmetic tells you which stage is impossible. It does not tell you what the possible stage costs.

Why there is a pre-ranker as well

The pre-rank stage exists for the same reason, one level down: a cheap model guarding an expensive one. Whether that guard is worth having is a ratio you compute, not a pattern you invoke.

pre-ranker cost   2,000 items × 0.05 MFLOP  =  0.10 GFLOP
work it avoids    1,700 items × 4.39 MFLOP  =  7.46 GFLOP
payback           7.46 / 0.10               =  75x

The two ratios behind that: the pre-ranker is 4.39 / 0.05 = 88x cheaper per item, and it cuts the expensive stage’s input by 2,000 / 300 = 6.7x.

A guard that returns less than 1x is a stage you should delete. Computing the ratio is what tells you which one you have.

The pre-ranker’s 0.05 MFLOP carries that whole payback argument, so it needs an architecture rather than an assertion. Its input is 128 numbers, all assembled from things retrieval already produced — nothing new is fetched. (A one-hot encoding is the standard way to represent “which of six sources” as six numbers: five zeros and a one.)

The block below lists those 128 numbers and then prices the network over them. Note the second half: three small matrix multiplies, and the total lands at a twentieth of a MFLOP.

pre-ranker input, 128-d
    64   projection of the user tower output
    48   projection of the precomputed item embedding (already in the index)
     1   the two-tower dot product itself, as a feature
     6   which retrieval source supplied this candidate (one-hot)
     9   cheap item scalars: age, duration, residualized CTR, channel affinity

MLP 128 -> 128 -> 64 -> 1
  = 2 × (128·128 + 128·64 + 64·1) + 2×128 (the dot)
  = 49,536 FLOP  =  0.050 MFLOP

The pre-ranker is cheap because every embedding it consumes was already fetched by retrieval — it adds no feature-store round trip, which is the cost that actually matters (Scale and cost). That is the design rule for a guard stage generally: it is allowed to be a smaller model, but it is required to be a model over features you already have in hand.

The two designs, side by side

The diagram below draws both candidate designs from the same starting point — the 800 M corpus at the top — so they can be compared. The upper branch is the one-stage design and it dead-ends in red. The lower branch is the funnel. Follow the item counts down the right-hand path: 800 M, 2,000, 300, 20.

flowchart TD
    C(["Corpus · 800,000,000"]) --> X["Score everything with the ranker<br/>3.51 PFLOP/request · 23.4 s<br/>11,900,000 A100 at peak"]
    X --> NO(["IMPOSSIBLE<br/>an impossible gap<br/>of 6.4 orders of magnitude"])

    C --> R["Retrieval · 6 sources<br/>ANN + precomputed item embeddings<br/>cost independent of corpus size"]
    R --> P2(["2,000 candidates"])
    P2 --> PR["Pre-rank stage<br/>0.05 MFLOP/item<br/>0.3 A100 of FLOPs"]
    PR --> P3(["300 candidates"])
    P3 --> RK["Full rank · 4.39 MFLOP/item<br/>4.5 A100 of FLOPs"]
    RK --> P4(["300 scored"])
    P4 --> SL["Slate re-rank<br/>diversity · freshness<br/>integrity demotion"]
    SL --> P5(["20 slots"])

    style X fill:#9d0208,color:#fff
    style R fill:#1d3557,color:#fff
    style PR fill:#40916c,color:#fff
    style RK fill:#2d6a4f,color:#fff

Reading the lower path in words: retrieval’s cost does not depend on corpus size, the pre-rank stage runs a tiny model over 2,000 candidates, the full ranker runs a real one over 300, and a final slate re-rank picks 20 with attention to diversity, freshness and the integrity demotions from ch 05.

Each arrow is a place where the per-item budget goes up because the item count went down. Read the two paths as a single claim: every reduction in the funnel exists to make the next stage’s per-item cost affordable, and each stage’s cutoff is set by the ratio of its own cost to the next stage’s.

The colour key, used by both diagrams in this chapter

One colour key covers this diagram and the system diagram in Ranking. It is the repo key from system-design/01 with each row read for a ranking pipeline rather than a storage ladder.

What must be true for this framing to hold

The whole funnel rests on three assumptions. Name them, because each one has a regime where it fails and the design changes.

1. The item side of the retrieval model can be computed without knowing the user. That is what makes precomputation, and therefore sublinear retrieval, possible at all. Candidate generation shows exactly what it costs you.
2. The corpus is large enough that scoring it exhaustively is hopeless. At 100,000 items rather than 800 million, the correct design is one stage and everything here is over-engineering.
3. Latency is bounded at 100 ms of server time. That is what stops you buying your way out with more machines. You can always add fleet, but you cannot add fleet to make one request’s sequential work finish sooner.

2. Candidate generation

The first model in the funnel is the one that turns 800 million videos into 2,000 plausible ones, and nearly everything about its design follows from a single requirement: 500 k new uploads a day have to be retrievable before anyone has watched them. The obvious classical answer fails that requirement structurally, the two-tower design that replaces it passes, and a serious system ends up running six retrieval sources rather than one.

Matrix factorization, and why cold start is structural

Before building anything, it is worth being precise about why the textbook recommender fails here, because the failure is definitional rather than practical.

Matrix factorization (MF) treats the whole history as a giant table R of users by items and looks for two thin matrices whose product approximates it: R ≈ U V^T, one latent vector — a learned list of numbers with no predefined meaning — per user and per item, fit by minimizing squared error over the interactions actually observed.

For a brand-new item, the gradient of the loss with respect to that item’s vector v_i is a sum over that item’s interactions. There are none, so the gradient is exactly zero, so v_i is exactly its random initialization forever. The model’s prediction for it is noise with a magnitude set by the init scale.

That is not a data-sparsity problem you can regularize your way out of. It is a definitional one: the parameter for an item is defined only by that item’s interactions, so with zero interactions the parameter does not exist. At 500 k new videos a day against 800 M in the corpus, a system whose retrieval cannot represent a new item has forfeited its entire supply side.

Two-tower retrieval, and why the separation is the whole point

The fix is to make the item’s representation a function of what the item is rather than of who has watched it — a design that buys two properties and, just as importantly, has a price worth naming.

A two-tower model is two separate networks — one reads the user, one reads the item — whose outputs are compared by a single dot product, written <·,·>, which for unit-length vectors is the cosine of the angle between them:

score(u, i)  =  <f(u; user features), g(i; ITEM CONTENT features)>

That structure buys two properties, and they are separate wins that people habitually run together:

Property 1 — g is a function of item features, not an item id. Title text embedding, thumbnail embedding, channel id, category, duration, language, upload time. A video uploaded ninety seconds ago has all of those, so g(i) is meaningful at zero interactions. This is the actual reason to prefer two-tower over MF, and it has nothing to do with neural networks being more expressive.

Property 2 — g depends only on i, so it can be precomputed.

800 M items × 128-d fp16  =  205 GB of item embeddings
built once per index cycle, loaded into an ANN index
per request: ONE forward pass of f(u), then ONE ANN query

Where d = 128 comes from

Every memory number in this chapter is linear in d, the embedding width, so it is worth knowing that 128 is a decision rather than a house default.

The sibling chapters land elsewhere, for good reasons. The objective argues for d = 32 over 5 M listings, because the within-market signal is genuinely low-dimensional and a larger d there mostly buys hubness — the pathology where a few vectors end up near everything and turn up in every result list.

Here the corpus is 160x larger, and the tower has to separate 800 M items across every topic a global catalog contains. An ablation — rerunning the same system with exactly one thing changed — settles it.

The table below has one row per candidate d. Index bytes is 800e6 × d × 2 (two bytes per fp16 number). Recall@2,000 is the share of held-out satisfied watches that appear somewhere in the 2,000 candidates retrieval returns; it is the only quality number retrieval is allowed to be judged on, because retrieval’s job is to not lose the good item, not to order it.

d      index bytes (800 M, fp16)     recall@2,000
 32          51.2 GB                     0.71
 64         102.4 GB                     0.79
128         204.8 GB                     0.83
256         409.6 GB                     0.835

Look at the last two rows: memory doubles, recall moves by 0.005. Recall stops moving between 128 and 256 and memory does not, so 128 is the last doubling that pays.

Quote the table rather than the number. A reader who takes d = 128 from here into a 5 M-item catalog has bought 4x the memory for nothing; a reader who takes d = 32 into 800 M items has capped retrieval recall at 0.71 and will spend six months blaming the ranker.

The alternative it beat, and the price it paid

Contrast a model that mixes user and item features early — write it score = h(u, i). Nothing factors, so h must be evaluated once per candidate, and you are back at Framing and the arithmetic that forces two stages’s 11.9 million A100s. The tower separation is not an architectural taste; it is the only structure that makes the item side precomputable.

It does have a price, and candidates rarely name it. Name it correctly, though, because the version everybody repeats is false.

The false version: the two towers cannot see each other until the dot product, so the retrieval model structurally cannot express “this user watches 40-second videos on cellular but 20-minute videos on wifi.”

That is wrong on its face. A dot product is a sum of user-feature × item-feature products, so it is nothing but interaction terms. Two dimensions are enough to carry that exact example, with each tower seeing only its own side:

def user_tower(conn):
    """The user side of a 2-d two-tower model. It never sees the item."""
    return (1.0, 0.0) if conn == "cellular" else (0.0, 1.0)


def item_tower(duration_s):
    """The item side. It never sees the user, and it is precomputable."""
    short = 1.0 if duration_s <= 120 else 0.0
    return (short, 1.0 - short)


def two_tower_score(conn, duration_s):
    return sum(u * i for u, i in zip(user_tower(conn), item_tower(duration_s)))


CATALOG = {"40s clip": 40.0, "20min video": 1200.0}
for _conn in ("cellular", "wifi"):
    _scores = {name: two_tower_score(_conn, d) for name, d in CATALOG.items()}
    print("%-8s %s  winner = %s" % (_conn, _scores, max(_scores, key=_scores.get)))

# the interaction the usual telling calls impossible, decided correctly, twice
assert two_tower_score("cellular", 40.0) > two_tower_score("cellular", 1200.0)
assert two_tower_score("wifi", 1200.0) > two_tower_score("wifi", 40.0)

cellular {'40s clip': 1.0, '20min video': 0.0}  winner = 40s clip
wifi     {'40s clip': 0.0, '20min video': 1.0}  winner = 20min video

So the towers get it right in both contexts, from two dimensions, with neither tower ever seeing the other side.

The real limit is rank, not structure

The two towers do interact. What is capped is how many independent interactions they can carry, and the right word for that cap is rank.

Picture the thing the model is trying to represent as a big table: one row per user type, one column per item type, and in each cell how much that user type likes that item type. That table is the interaction matrix. Its rank is the number of independent patterns you need to reconstruct it — a table you can write as “one user-side number times one item-side number” is rank 1; a table that needs five such products summed is rank 5.

A d-dimensional dot product is a sum of d products of a user-side number and an item-side number. So it can express exactly the interaction matrices of rank d or less, and nothing more.

• “Short videos on cellular, long videos on wifi” is rank 1. It fits at d = 128 with 126 dimensions to spare — which the code above just demonstrated at d = 2.
• A random 500x500 interaction matrix keeps only 63% of its singular-value energy in its top 128 directions. (Singular values are the sizes of a matrix’s independent patterns, biggest first; “energy in the top 128” is the share of the total the biggest 128 account for.) So 37% of that matrix is unrepresentable at d = 128, no matter how well the towers are trained.

You buy sublinear retrieval by capping interaction at rank d, and you buy the rest back in the ranker. That is the two-stage design in one sentence.

Stating it as rank rather than as structure is also what gives you a dial. “Structurally impossible” has no dial. d has one, and it is the ablation table above.

Training the towers: sampled softmax and the logQ correction

What the retrieval model takes as input and how it is trained comes down to one loss and one correction. Without the correction, the model silently learns something you did not ask for. The correction, logQ, gets only a sentence in Stage 6 training, so it is derived here in full, with numbers.

Positives, negatives, and the split

A positive is a training example the model should score high; a negative is one it should score low.

Here a positive is a (user, item) pair where the watch cleared the satisfied-watch threshold — not a click. That choice is load-bearing. Training retrieval on clicks imports The watch time trap’s clickbait failure into the candidate pool, where no downstream stage can remove it, because the ranker can only reorder what retrieval hands it.

Negatives come from two places:

• The other items in the same training batch. Reusing the batch is what makes the loss affordable at all — you get B - 1 negatives for free from items you already loaded.
• A small mined pool of near-misses: items the model currently scores high that are not, in fact, positives. The infonce version where the vanishing is quantitative prices why one mined negative is worth thousands of random ones.

Split temporally, always. Train on days 1-28, validate on day 29, test on day 30. A random split lets tomorrow’s co-watch edges leak into today’s training pair, and the leaked model looks excellent right up until it is served.

The loss you want, and the one you can afford

The loss is a softmax — the function that turns a set of raw scores into probabilities by exponentiating each one and dividing by the sum of all of them — taken over the whole corpus:

L(u, i+)  =  -log [ exp s(u, i+) / SUM over all 800 M items j of exp s(u, j) ]

Nobody can compute that. The denominator is a sum over all 800 million items, once per training example.

The obvious shortcut is to sum over a sample of items instead. But that shortcut is broken as written. Sample B negatives from some distribution Q and sum exp s(u, j) over just those, and you get a biased estimate of the true sum — biased meaning wrong on average, not merely noisy. The reason is that Q is not uniform: an item that gets sampled twice as often contributes twice as much to the sample sum as it should.

Importance sampling is the standard repair for exactly this situation. When you estimate a total from a non-uniform sample, divide each sampled term by its own sampling probability, so that a rarely-drawn item counts for all the items like it that were not drawn.

Here the terms being summed are exp s(u, j), so dividing by Q(j) inside the exponential is the same as subtracting log Q(j) from the score. That subtraction is the whole correction.

sampled softmax with logQ correction, applied to EVERY logit in the batch:

    logit'(u, j)  =  <f(u), g(j)>  -  log Q(j)

Q(j)  =  P(item j appears in a batch)  ~=  B · p_j
         where p_j is item j's share of the training stream and B is the
         batch size (8,192 here), valid while B·p_j << 1

Two items, one batch, real numbers

The table below runs the correction on two items that differ only in popularity. Both get an identical raw dot product of 2.00, so any difference in the bottom row comes purely from the correction.

Take a head item at p = 6.1e-6 of the training stream. Uniform share would be 1/800e6 = 1.25e-9, so this item is about 4,900x uniform. And a tail item at p = 6.1e-9, about 5x uniform. Batch size B = 8,192.

Read the last row carefully; the direction is easy to get backwards.

The tail negative is drawn once for every 0.05 / 5e-5 = 1,000 draws of the head negative. So when it is drawn, it has to stand in for the roughly 1,000 items like it that were never sampled. That is why its logit — the raw score that goes into the softmax — comes out larger.

The head negative goes the other way: it is down-weighted so that the partition function, the softmax’s denominator, does not count it a thousand times over.

The correction is not about being kind to unpopular items. It is about making a sampled sum an unbiased estimate of an unsampled one.

What skipping it costs

Skip the correction and the head item’s appearances as somebody else’s negative are over-counted by exactly that 1,000x ratio. At convergence the model learns s(u,i) - log p_i instead of s(u,i).

Size that handicap:

log(0.05) - log(5e-5)  =  6.91 nats
exp(6.91)              =  1,000x in softmax probability

A nat is the natural-logarithm unit of log-odds, the counterpart of a bit. So at an identical dot product, the popular item is ranked as though it were a thousand times less relevant. That is not a small distortion you can regularize past — it is larger than the range the towers use for relevance in the first place.

Some teams want a popularity penalty, which is why this is a decision rather than a bug. Be explicit about which one you are doing, because “we use in-batch negatives” silently ships a popularity penalty of 6.9 nats that nobody chose and nobody can tune. With the correction in place, popularity debiasing becomes a separate, deliberate term you can dial (Popularity bias amplification’s residualization is where it belongs).

Three mechanical details

• The subtraction lives inside the training softmax and nowhere else. The served score is the plain dot product <f(u), g(i)>. Applying - log Q at serving re-introduces a popularity boost of exactly the magnitude you just removed, with the sign flipped.
• Estimate Q with a decayed counter, and pick the half-life on purpose. A decayed counter is a running count in which older observations are progressively discounted; the half-life is how long it takes an event to count for half as much. An exact fp32 count per item — fp32 being 32-bit floating point, four bytes — costs 800e6 × 4 B = 3.2 GB, which fits in memory, so no approximate sketch data structure is needed. Half-life too long and a newly-viral item carries a stale-low Q, so it is under-corrected as a negative and over-promoted. Too short and tail counts are single-digit, so log Q is noise. A one-day half-life against a daily index cycle is the usual landing spot.
• logQ does not fix tail recall, and conflating the two is the common error (Stage 6 training makes the same point). A tail item’s problem is that it appears in too few positive pairs to have learned an embedding at all. logQ corrects the score scale along the popularity axis; the fresh pool and exploration (Position bias and the feedback loop) are what buy positives.

The retrieval model, assembled

Six questions, asked of every model in this chapter.

• What is the model? Two networks — a user tower and an item tower — each ending in a 128-dimensional embedding, compared by one dot product.
• What does it take as input? The user tower reads the user’s history, context and device. The item tower reads only content features — title text, thumbnail, channel, category, duration, language, upload time — and deliberately not the item’s id or its interaction counts.
• Where do the labels come from? Logged watches. A positive is a (user, item) pair whose watch cleared the satisfied-watch threshold; negatives are the other items in the same batch plus a small mined pool. The labels are free, and the choice of which watch counts is the most consequential product decision in this section.
• How is it trained? Sampled softmax with the logQ correction applied to every logit in the batch, split temporally across days 1-28 / 29 / 30.
• How is it served? The item tower runs offline once per index cycle over all 800 M items, and its output goes into an ANN index. The user tower runs once per request.
• How do you know it works? Recall@2,000 against held-out satisfied watches — the ablation table above is exactly that measurement — plus corpus coverage, because a retrieval model that never proposes 99.5% of the corpus is failing in a way recall alone will not show.

What must be true for the two-tower design to hold

1. Item content features are genuinely predictive of who will watch it. That is what makes a zero-interaction embedding meaningful. It is false for content whose appeal is entirely social.
2. The user-item interaction structure that matters has rank at most d = 128. That is all a dot product of two independently computed vectors can span. This assumption starts failing well before the 500x500 case above — which is why the ranker exists.
3. The popularity distribution is stable enough for a decayed counter to estimate Q. A newly viral item violates exactly this.

Multiple sources, and why the quota is where policy lives

The two-tower model just derived is one retrieval source, not the retrieval system. A serious feed runs six of them, and the quota — how many candidates each source is allowed to contribute — is where the product’s diversity policy actually lives.

Here are the six. The “Returns” column is that source’s quota — how many candidates it is allowed to put into the pool — and the six quotas add to 2,000. Read the last column as the reason no single source can be the whole system.

Dedupe before you truncate

Each source over-fetches — asks for more than its quota. Then three things happen to the union, in this order:

1. Dedupe: collapse it so each video appears once, no matter how many sources found it.
2. Cap: enforce a per-channel limit so no single channel dominates.
3. Truncate: cut each source back to its quota.

That order is load-bearing. The cheapest way to see why is to run the wrong order and count what it loses. Take just two of the six sources:

two-tower over-fetches 900   (quota 600)
co-watch  over-fetches 600   (quota 400)
140 videos appear in both

dedupe FIRST, then truncate:
    co-watch drops its 140 duplicates -> 460 unique, truncate to 400
    two-tower truncates to 600
    pool contribution                   =  1,000     <- both quotas filled

truncate FIRST, then dedupe:
    two-tower -> 600, co-watch -> 400
    of the 140 overlaps, 140 × (600/900) × (400/600)  =  62 survive both cuts
    pool contribution  =  600 + 400 - 62  =  938

The second block’s middle line is the one to follow: after truncation, a given overlapping video survives on the two-tower side with probability 600/900 and on the co-watch side with probability 400/600, so about 140 × 0.67 × 0.67 = 62 are still duplicated at dedupe time — and each of those 62 costs a slot.

Sixty-two slots evaporate and nothing logs it. The pool is simply smaller than the quotas say, and the quota table still adds to 2,000 on the wiki page.

Across all six sources the same roughly 7% leak turns a 2,000-candidate pool into about 1,860. It lands hardest on the pair with the widest overlap — two-tower and co-watch — and those are the two narrowest sources. So the ordering bug shifts the pool’s composition toward fresh and trending as well as shrinking it. Dedupe first.

Why six and not one

Every source has a different failure mode, so a single source’s bias becomes the system’s bias with nothing to counteract it. Look down the “Fails when” column: co-watch narrows, trending amplifies popularity, social only works for connected users. Running them together means no single failure owns the pool.

There is a second reason, operational and underrated: each source is independently debuggable, independently A/B-able, and independently killable when one of them starts returning garbage.

The per-source quota is where the diversity policy actually lives. A diversity penalty bolted onto re-ranking can only reorder what retrieval already returned; if co-watch supplied 90% of the pool, no re-ranker can diversify it. Setting co-watch to 400 of 2,000 is a policy decision, made once, in a place you can read.

3. Ranking

With the candidates in hand, the expensive model takes over — the one that scores 300 candidates and decides the slate. What it takes as input, why it predicts eight things instead of one, and how eight probabilities become one number are all consequences of a single premise: any objective you can write down will be gamed.

The diagram below is the whole system on one page; every box in it is derived somewhere in this chapter. For now, two features matter: the request splits into two user representations at the top, and a dotted arrow at the bottom feeds the logs back into the ranker.

flowchart TD
    U(["Request"]) --> RT["Real-time user state<br/>last 20 actions · mean-pooled<br/>0 FLOPs · updated in ms"]
    U --> LT[("Long-term user tower<br/>recomputed nightly<br/>500 M × 128-d")]

    LT --> SRC["6 retrieval sources<br/>ANN + co-watch + fresh<br/>+ subs + trending + social"]
    RT --> SRC
    SRC --> POOL["~2,000 candidates<br/>dedupe · per-channel cap"]
    POOL --> PRE["Pre-rank · 128-d in<br/>MLP 128/128/64 · 0.05 MFLOP<br/>no new feature fetch<br/>2,000 -> 300"]
    PRE --> RANK["Full ranker · 1,500-d in<br/>7 feature families<br/>shared embeddings -> MMoE<br/>1024/512/256"]
    RT --> RANK
    RANK --> HEADS["8 heads<br/>click · watch-frac · like<br/>share · subscribe<br/>not-interested · report<br/>survey satisfaction"]
    HEADS --> VAL["Value model<br/>weights fit to a 28-day<br/>out-of-band outcome"]
    VAL --> RR["Slate re-rank<br/>MMR diversity · freshness<br/>integrity demotions from ch 05"]
    RR --> OUT(["20 slots<br/>1 in 20 reserved for exploration"])
    OUT --> LOG[("Logs: impression + position<br/>+ propensity + candidate set")]
    LOG -.->|"IPW-weighted"| RANK

    style LT fill:#1d3557,color:#fff
    style SRC fill:#1d3557,color:#fff
    style PRE fill:#40916c,color:#fff
    style RANK fill:#2d6a4f,color:#fff
    style VAL fill:#2d6a4f,color:#fff
    style OUT fill:#bc6c25,color:#fff
    style LOG fill:#9d0208,color:#fff

Colours are the key published under Framing and the arithmetic that forces two stages’s diagram: navy is read from something built offline, light green is the guard stage, green is the learned model that produces the score, orange is the slot the score does not fill, red is the edge you cannot undo.

Following one request down the diagram

1. Two user representations are built in parallel, on completely different clocks. The real-time state is just the mean of the last 20 watched items’ embeddings. The long-term user tower is recomputed nightly for all 500 M users. Cold start explains why this split exists.
2. Both feed the six retrieval sources, and the union of what those return is the ~2,000-candidate pool.
3. The pre-ranker cuts 2,000 to 300 without fetching anything new.
4. The full ranker scores those 300 over 1,500 input dimensions, through an MMoE trunk (derived below).
5. Eight heads each predict a different outcome, and a small value model collapses the eight into one score.
6. A slate re-rank picks the final 20. It applies MMR — maximal marginal relevance, which builds the list one item at a time and penalizes each candidate by how similar it is to what is already on the list, so the slate does not fill with twenty versions of the same video — plus freshness and the integrity demotions from ch 05. One of the 20 slots is reserved for exploration.
7. Everything shown is logged with its position and propensity, and that log is what trains the ranker on the next cycle — IPW-weighted to undo the bias the system’s own choices introduced (Position bias and the feedback loop).

That dotted arrow from the log back to the ranker is the most dangerous edge in the diagram. Position bias and the feedback loop and Metrics and why offline and online disagree are both about it.

What the 1,500 input dimensions actually are

Framing and the arithmetic that forces two stages priced the ranker at 1,500 input dimensions. Every FLOP number in the chapter descends from that figure — and the fleet size in Scale and cost descends from its byte count — so the concatenation deserves to be spelled out.

It has seven families. The Dims column is what each family contributes to the 1,500. The two columns nobody reads and everybody should are Refresh — how often that family’s values change, which decides whether it can be precomputed — and Trap, the specific way each family goes wrong.

Four of the terms in that table are worth a line each. Mean-pooled means simply averaged: take the embeddings of the last 20 items and average them into one vector, which costs nothing and updates instantly. ASR is automatic speech recognition — the transcript of the video’s audio, embedded. CTR is click-through rate, the share of times a shown item is clicked. And hashing the tail of a categorical feature means mapping its millions of rare values into a fixed number of buckets rather than giving each its own row in the embedding table, which is what keeps 20 M channel ids from dominating memory.

Three things follow from the table that do not follow from the number alone.

1. This is where the interaction rank Candidate generation gave up comes back.

The tower separation never forbade “40-second videos on cellular, 20-minute videos on wifi” — that one is rank 1 and the dot product carries it. What it capped was the number of such terms, at d = 128. The ranker has no such cap.

Here connection type (family 2) and duration (family 6) sit in the same concatenation, feeding a trunk — the shared body of the network, below the per-task heads — that is nothing but interaction terms. The 40-dimension cross family is the small part. The real cross-learning is the trunk, which is why the ranker needs 1,500 dims and depth and the retrieval tower does not.

2. Families 4-7 are the per-candidate fetch, and they are the serving cost.

(768 + 128 + 32 + 40) dims × 2 B/dim  =  1,936 B per candidate
1,936 B × 300 candidates              =  0.58 MB of scattered reads/request

That 0.58 MB is what the request actually moves, against 1.32 GFLOP of arithmetic. Scale and cost shows the bytes winning.

3. Families 1-3 are fetched once per request, not once per candidate.

128 + 72 + 332 = 532 of the 1,500 dims do not scale with the candidate count at all. That is why the ranker can afford to be wide on the user side: widening the user side is free per candidate, and widening the item side is not.

One populated row

A table of dimension counts is not an input. Below is what a single (user, candidate) pair actually looks like on the way into the trunk.

The block is split in two. The top half is request-scoped — fetched once and shared by all 300 candidates. The bottom half is candidate-scoped, fetched 300 times. The two annotated fields, connection_type and duration_s, are the ones the rank argument turns on.

request r_91f4c2  ·  user u_4471903 (locale en-GB, Pixel 7, 19:40 local, wifi)
  ---- fetched ONCE per request, 532 of the 1,500 dims ----------------------
  long_term_user_emb      128-d, ||v|| = 1.00, recomputed 03:12 UTC (16.4 h stale)
  locale/device/hour/dow  "en-GB" / "android-hi" / 19 / Thu
  connection_type         "wifi"                      <- family 2
  last_20_mean_pool       128-d over 20 items, newest 4 min old
  last_query_emb          128-d, "sourdough starter"
  session_counters        dwell 512 s · skips 3 · items seen 27

  ---- fetched PER candidate, 968 dims, 1,936 B ----------------------------
  candidate v_88301755   "Sourdough: the 6-minute version"  channel c_5590
  title / thumb / ASR     256 + 256 + 256-d
  channel_id (hashed) 64 · category "food" 32 · language "en" 32
  duration_s              360                         <- family 6
  age_h 41 · residual_views +1.8 sd · residual_ctr +0.9 sd · cadence 2.1/wk
  source                  co-watch (rank 7 of 400, source score 0.71)
  user_channel_affinity   0.34 · language_match 1 · co_watch_score 0.62

connection_type and duration_s are now sitting in the same concatenation with no rank ceiling above them. In retrieval those two could only meet through a rank-128 bottleneck; here they meet directly.

Why multi-task, stated as a threat model

The ranker predicts eight things rather than one, and the argument for it is a security argument rather than an accuracy one.

A single objective is a specification, and every specification you can write down is gameable by the ranker and by creators. Predict clicks and you get thumbnails that promise things the video does not contain. Predict watch time and you get The watch time trap. Predict likes and you get engagement-bait requests for likes.

The defence is not a better single objective. It is a set of heads chosen so that no one lever moves all of them the same way.

Read the table below as a roster of defences rather than a list of predictions. Every positive head is paired with something that falls when the item is bad, and the survey head is the only one a creator can neither observe nor A/B against.

The base rate column is how often each event happens at all. Notice the spread — 0.58 at the top and 0.00006 at the bottom — because that four-order-of-magnitude range is what breaks the obvious way of combining the heads two subsections from now.

The cheapest large quality win in most recommenders is adding a negative head. The reason is redundancy: all the positive heads correlate with each other, so each new one adds little information the others did not already carry. The negative signal is nearly orthogonal to all of them — orthogonal meaning it carries information none of the others do.

Measured here: adding the not-interested head with a large weight moved offline AUC by 0.002 and moved 28-day retention by 0.9 points. (AUC is area under the ROC curve — the probability the model scores a random positive above a random negative, where 0.5 is chance and 1.0 is perfect.)

A 0.002 AUC move next to a 0.9-point retention move is the whole argument in one line, and Metrics and why offline and online disagree is about why those two numbers disagree so often.

Shared bottom is not enough — MMoE, and the gradient argument

Predicting eight things from one network creates a new problem: the tasks fight.

A shared-bottom model is the obvious multi-task design — one trunk, eight heads — and its weakness is that tasks whose gradients point in opposite directions in that shared trunk destroy each other’s representation. Clickbait maximizes P(click) and minimizes P(survey satisfaction); those two heads want the trunk to encode “thumbnail promises drama” with opposite signs, so whichever task has more data simply wins.

MMoE (multi-gate mixture of experts) replaces the single trunk with a bank of several small trunks, called experts, and gives each task its own softmax gate — a learned set of weights summing to 1 that decides how much of each expert that task reads. Conflicting tasks learn to route to different experts and stop fighting over the same parameters.

Here is the same ranker trained both ways. The column to read is the difference, and the row to read first is click AUC.

shared-bottom          MMoE (8 experts, per-task gates)
click AUC     0.781        0.782      unchanged
satisfaction  0.712        0.761      +4.9 points
share AUC     0.694        0.729      +3.5 points
params        +0%          +18%

The tasks that gain are exactly the ones that conflicted with the dominant task. Click AUC does not move, because click was already winning the fight in the shared trunk — it had nothing to gain from a truce. Satisfaction and share were the losers, and they are the two that jump.

The price of the truce is the last row: 18% more parameters.

Combining the heads into one score

Eight predictions have to become one ordering. The blend that does it turns on four details, each a place where the obvious implementation is wrong in a way no dashboard shows.

The score is a weighted sum, where p_k is head k’s prediction, phi_k is a transform applied to it and w_k is its weight:

score  =  sum_k  w_k · phi_k(p_k)

Four details decide whether that works. The first is about phi_k, the second is about the watch-time head specifically, the third is about w_k, and the fourth is about what the sign of the finished score means.

Detail 1 — transform before weighting

Base rates span 0.58 to 0.00006, a ratio of 0.58 / 6e-5 = 9,700. Four orders of magnitude.

A linear blend of raw probabilities cannot survive that. P(share) moves by at most 0.0008 between a terrible item and a perfect one, so for the share head to matter against the click head, w_share has to be numerically absurd — and at that size it also amplifies the share head’s noise.

The fix is phi_k = log. Use log(p_k), or logits, which puts every head on a comparable scale of relative lift rather than absolute probability.

Why log space fixes commensurability, in three lines. Combining where the value judgement lives makes the same repair from the other direction — it keeps the linear blend and reads each weight against its head’s base rate. Log space does that job additively and for free. Write p̄_k for head k’s base rate:

log p_k  =  log(p_k / p̄_k)  +  log p̄_k
            ^^^^^^^^^^^^^^     ^^^^^^^^
            the item's LIFT    a constant, same for every candidate

Sum that over the eight heads and the second term collapses to one number — sum_k w_k · log p̄_k = 7.44 at the base rates and weights below. It is added identically to every candidate, so it cancels in the ordering.

Weighting log-probabilities is lift-weighting. That is why the log blend here does not commit the commensurability error ch 10 diagnoses, and why the code block below asserts the identity instead of asking you to believe it.

Detail 2 — watch time needs its own transform

Raw seconds has a heavy tail: a distribution where rare enormous values are common enough to dominate an average. A single 3-hour outlier can swamp a batch.

log1p(seconds), meaning log(1 + seconds), compresses it. And the head predicts watch fraction rather than seconds in the first place; seconds are reconstructed at the end, only for reporting.

Detail 3 — where the weights come from

Three options, and they are not equally honest.

1. Hand-tuned by A/B sweeps. Eight weights cannot be grid searched: even 4 values each is 65,536 arms, an arm being one variant an experiment has to allocate traffic to. So the practice is one-at-a-time sensitivity followed by a small simplex search — a derivative-free method that crawls downhill by reflecting a cluster of trial points — at about six weeks per full pass, and the result encodes whatever the team believed six weeks ago.
2. The value model. Collect an out-of-band target — one measured outside the recommendation loop, here 200 k survey responses plus 28-day retention plus reported regret — and fit w so the combined score predicts it. Because the target is collected outside the recommendation loop and resolves 28 days later, the ranker cannot hack it within a session. Cost: enormous label latency, so it is refit quarterly and the fast loop runs against the fixed w.
3. A bandit over weight vectors at cohort level. Fast, and it optimizes whatever short-horizon metric you feed it — which is the problem the value model exists to solve. Useful for a handful of weights, dangerous as the primary mechanism.

Ship 2, use 1 to bootstrap, use 3 only for weights whose short-horizon and long-horizon effects are known to agree.

Detail 4 — the integrity term, and why “multiplier” is the wrong word

This one decides whether the integrity demotion works at all (The integrity coupling), and it turns entirely on the sign of the combined score.

score = sum_k w_k · log(p_k) is a sum of signed multiples of log-probabilities. Log-probabilities are negative, so a positive weight contributes a negative number and a negative weight contributes a positive one. Which side wins is a fact about the item, not about the design.

Work it on two items, using the weights in the code below:

weakly-engaging item, combined score            =  -24.19
item at the head table's own base rates         =   +8.15

The base-rate item comes out positive because the two negative heads enter as -w · log(p) on rare events, and rare events have large negative logs:

not-interested   -2.0 × log(0.0021)    =  -2.0 × -6.17  =  +12.33
report           -4.0 × log(0.00006)   =  -4.0 × -9.72  =  +38.89

Now try to demote with a multiplier — an integrity score in (0, 1], applied as score × integrity. At integrity 0.5:

weak item        -24.19  ->  -12.10     PROMOTED
base-rate item    +8.15  ->   +4.08     demoted

The failure is non-monotonicity, not a sign flip. That is worse than a sign flip, because the reordering runs in exactly the wrong direction: weak items rise relative to ordinary ones under a mechanism built to suppress them. And there is no item-independent sentence you can put in a runbook to describe what the multiplier does, because it does different things to different items.

The integrity term belongs in the same log space as everything else: score + log(integrity). Adding a log is multiplying in the space the score is a log of, so this is the multiplication people meant. It is monotone decreasing for every item, whatever the sign of its sum.

A “multiplier” is only a multiplier in the space the score actually lives in.

The blend, as running code

The block below implements everything above and then executes it on real numbers. Three things to look for: combined_score sums over the heads and looks each weight up (not the reverse, which would crash on the duration weight); the integrity term is added, not multiplied; and the asserts at the bottom demonstrate the multiplier bug on both a weak item and an ordinary one, because asserting it on the weak item alone is a test that passes because the fixture was chosen to make it pass.

Two more functions ride along here because Position bias and the feedback loop quotes their output: examination_curve interpolates the measured position-bias knots to all 20 slots, and ipw_weight turns a position into an inverse-propensity weight with a floor.

import math

EPS = 1e-6
DURATION_KEY = "duration"


def combined_score(heads, weights, duration_s, integrity):
    """Blend calibrated head probabilities into one ranking score.

    Three things are load-bearing and all three are easy to get wrong:

    1. log-space. Base rates run from 0.58 (satisfaction) to 0.00006
       (report). A linear blend of raw probabilities cannot let the share
       head matter at any weight that is not numerically absurd.
    2. the duration term SATURATES. Linear in duration is a duration prior;
       predicting watch fraction and adding log1p(minutes) restores a
       preference for longer content without letting a 3-hour stream win
       on arithmetic alone. `duration` is a weight with no head behind it,
       so the sum runs over the HEADS and looks each weight up -- iterating
       the weights and indexing the heads raises KeyError on it, under every
       calling convention consistent with the head table above.
    3. integrity is ADDITIVE IN LOG SPACE, because `s` has no fixed sign.
       `s` is a weighted sum of log-probabilities, so it is usually negative,
       and `s * integrity` with integrity 0.5 maps -24.19 to -12.10 -- an
       integrity demotion that PROMOTES the item up the slate. Adding
       log(integrity) is exactly multiplying the underlying product of
       probabilities by it, and it is monotone decreasing for every `s`.
    """
    stray = set(weights) - set(heads) - {DURATION_KEY}
    if stray:
        raise KeyError("weights with no head: %s" % sorted(stray))
    s = sum(weights.get(k, 0.0) * math.log(max(p, EPS)) for k, p in heads.items())
    s += weights.get(DURATION_KEY, 0.0) * math.log1p(duration_s / 60.0)
    return s + math.log(max(integrity, EPS))     # a demotion, never a filter


EXAMINATION = {1: 0.81, 2: 0.62, 3: 0.49, 5: 0.34, 8: 0.23, 12: 0.16, 20: 0.09}


def examination_curve(measured=EXAMINATION, slots=20):
    """Linear interpolation of the measured knots over every slot.

    Section 5 prices the weight clip "over all 20 positions", which requires
    a propensity at all 20; the randomized swap only measures seven of them.
    Without the interpolation, `examination.get(position)` returns None for
    13 of 20 positions and IPW silently drops two thirds of every slate --
    and drops them non-uniformly, which is a bias, not a sample.
    """
    ks = sorted(measured)
    out = {}
    for j in range(1, slots + 1):
        if j in measured:
            out[j] = measured[j]
            continue
        lo = max(k for k in ks if k < j)
        hi = min(k for k in ks if k > j)
        out[j] = measured[lo] + (j - lo) / (hi - lo) * (measured[hi] - measured[lo])
    return out


CURVE = examination_curve()
PROPENSITY_FLOOR = min(EXAMINATION.values())      # 0.09, the slate's minimum


def ipw_weight(position, examination=CURVE, floor=PROPENSITY_FLOOR):
    """Inverse propensity weight with a propensity FLOOR, not a weight clip.

    A weight clip is the reflex and it does nothing here. With 20 slots and
    examination(20) = 0.09, weights span 1.23 to 11.1, so the usual clip at
    M = 20 never fires; M = 8 buys 1.2x variance for 3.5% bias, and M = 5
    buys 2.2x for 18%. There is no tail to trim.

    The floor is the slate's own smallest propensity, so it BINDS by
    construction: every position on this slate is at or above it, and a
    logged propensity below it is an event from another surface or a
    corrupted log. Those are dropped, not up-weighted, because a single
    1e-4 propensity contributes more gradient than 10,000 honest events.
    A floor of 0.05 is decorative -- nothing on a 20-slot slate reaches it,
    so it is a parameter that never fires, which is the same failure as the
    M = 20 clip below. Pair this with self-normalized IPS at the estimator
    level.
    """
    p = examination.get(position)
    if p is None or p < floor:
        return 0.0                    # drop; never rescue with a clip
    return 1.0 / p


# --- the head table and the clip table, executed ------------------------
BASE = {"click": 0.061, "watch_frac": 0.34, "like": 0.011, "share": 0.0008,
        "subscribe": 0.0004, "not_interested": 0.0021, "report": 0.00006,
        "satisfaction": 0.58}
WEIGHTS = {"click": 1.0, "watch_frac": 1.0, "like": 0.3, "share": 2.0,
           "subscribe": 3.0, "not_interested": -2.0, "report": -4.0,
           "satisfaction": 1.5, "duration": 0.4}
WEAK = {"click": 0.010, "watch_frac": 0.15, "like": 0.001, "share": 0.00005,
        "subscribe": 0.00002, "not_interested": 0.020, "report": 0.0005,
        "satisfaction": 0.30}

# `duration` is a weight with no head -- a loop that iterates the weights and
# indexes the heads raises KeyError on it, so the sum must run over the heads
assert isinstance(combined_score(BASE, WEIGHTS, 300.0, 1.0), float)
try:
    combined_score(BASE, dict(WEIGHTS, mystery=1.0), 300.0, 1.0)
except KeyError:
    pass
else:
    raise AssertionError("a weight naming no head must not be ignored")

# --- the multiplier bug is NON-MONOTONE, which is worse than a sign flip ----
# `s` has no fixed sign: it is about -24 on a weak item and +8.15 on an item at
# the head table's own base rates, because -w*log(p) on the rare NEGATIVE heads
# contributes +12.33 (not-interested) and +38.89 (report). So `s * integrity`
# moves weak items UP and ordinary items DOWN, at the same integrity score.
print("%-6s %10s %12s %12s" % ("item", "clean", "x 0.5", "+ log(0.5)"))
for _name, _h, _d in (("BASE", BASE, 300.0), ("WEAK", WEAK, 45.0)):
    _clean = combined_score(_h, WEIGHTS, _d, 1.0)
    print("%-6s %10.4f %12.4f %12.4f"
          % (_name, _clean, _clean * 0.5, combined_score(_h, WEIGHTS, _d, 0.5)))
    assert abs(combined_score(_h, WEIGHTS, _d, 0.5)
               - (_clean + math.log(0.5))) < 1e-12       # log space: always DOWN
assert combined_score(BASE, WEIGHTS, 300.0, 1.0) > 0.0   # NOT negative on most items
assert combined_score(WEAK, WEIGHTS, 45.0, 1.0) < 0.0
assert combined_score(WEAK, WEIGHTS, 45.0, 1.0) * 0.5 > combined_score(
    WEAK, WEIGHTS, 45.0, 1.0)                            # multiplier PROMOTES the weak
assert combined_score(BASE, WEIGHTS, 300.0, 1.0) * 0.5 < combined_score(
    BASE, WEIGHTS, 300.0, 1.0)                           # and DEMOTES the ordinary
for _h in (BASE, WEAK):
    for _d in (5.0, 45.0, 3600.0):
        assert (combined_score(_h, WEIGHTS, _d, 0.5)
                < combined_score(_h, WEIGHTS, _d, 1.0))  # log space, every item

# --- log-space weighting IS lift-weighting: the base-rate level cancels -----
_LEVEL = sum(WEIGHTS[k] * math.log(BASE[k]) for k in BASE)
print("\nsum_k w_k log(pbar_k) = %.4f, an item-independent constant" % _LEVEL)


def lift_score(heads, weights, duration_s, integrity, base=BASE):
    """The same score written as weighted LOG-LIFT plus a constant."""
    s = sum(weights.get(k, 0.0) * math.log(max(p, EPS) / base[k])
            for k, p in heads.items())
    s += weights.get(DURATION_KEY, 0.0) * math.log1p(duration_s / 60.0)
    return s + _LEVEL + math.log(max(integrity, EPS))


# --- one request, one slate, printed --------------------------------------
SLATE = [
    ("tutorial, 6 min", 360.0, 1.00,
     {"click": 0.074, "watch_frac": 0.62, "like": 0.021, "share": 0.0019,
      "subscribe": 0.0013, "not_interested": 0.0009, "report": 0.00003,
      "satisfaction": 0.71}),
    ("clickbait, 40 s", 40.0, 1.00,
     {"click": 0.152, "watch_frac": 0.31, "like": 0.008, "share": 0.0006,
      "subscribe": 0.0002, "not_interested": 0.0094, "report": 0.00021,
      "satisfaction": 0.33}),
    ("livestream, 3 h", 10800.0, 1.00,
     {"click": 0.048, "watch_frac": 0.06, "like": 0.009, "share": 0.0005,
      "subscribe": 0.0007, "not_interested": 0.0018, "report": 0.00005,
      "satisfaction": 0.52}),
]
_ranked = sorted(((combined_score(h, WEIGHTS, d, i), n, d, h["click"])
                  for n, d, i, h in SLATE), reverse=True)
print("\n%-4s %-17s %8s %8s %9s" % ("rank", "candidate", "dur_s", "p_click", "score"))
for _r, (_s, _n, _d, _c) in enumerate(_ranked, 1):
    print("%-4d %-17s %8.0f %8.3f %9.4f" % (_r, _n, _d, _c, _s))
assert _ranked[0][1] == "tutorial, 6 min"      # highest click rate does NOT win
assert _ranked[-1][1] == "clickbait, 40 s"     # p_click 0.152 and last, by design
for _n, _d, _i, _h in SLATE:                   # the lift form is the same number
    assert abs(lift_score(_h, WEIGHTS, _d, _i)
               - combined_score(_h, WEIGHTS, _d, _i)) < 1e-9

# every slot has a propensity, and it matches the two endpoints section 5 quotes
print("\ninterpolated examination, every slot:")
print("  " + " ".join("%d:%.3f" % (j, CURVE[j]) for j in range(1, 21)))
print("  IPW weights span %.2f (pos 1) to %.2f (pos 20)"
      % (ipw_weight(1), ipw_weight(20)))
assert len(CURVE) == 20 and all(ipw_weight(j) > 0.0 for j in range(1, 21))
assert abs(ipw_weight(1) - 1.0 / 0.81) < 1e-12
assert abs(ipw_weight(20) - 1.0 / 0.09) < 1e-12
assert abs(CURVE[4] - 0.415) < 1e-12


def _clipped(m):
    return [j for j in range(1, 21) if 1.0 / CURVE[j] > m]


# the interpolated curve reproduces section 5's clip table exactly
for _m in (20, 10, 8, 5):
    print("  M = %2d clips positions %s" % (_m, _clipped(_m) or "nothing -- never fires"))
assert _clipped(20) == []
assert _clipped(10) == [19, 20]
assert _clipped(8) == [17, 18, 19, 20]
assert _clipped(5) == list(range(10, 21))

# the floor binds on anything a 20-slot slate cannot produce, and only that
assert ipw_weight(20, {20: 0.05}) == 0.0
assert ipw_weight(20, {20: 0.09}) == 1.0 / 0.09
assert ipw_weight(21) == 0.0

Running it prints the four things the prose above quotes — the multiplier bug, the cancelling constant, one slate ordered end to end, and the interpolated propensity curve Position bias and the feedback loop needs:

item        clean        x 0.5   + log(0.5)
BASE       8.1533       4.0767       7.4602
WEAK     -24.1954     -12.0977     -24.8885

sum_k w_k log(pbar_k) = 7.4366, an item-independent constant

rank candidate            dur_s  p_click     score
1    tutorial, 6 min        360    0.074   19.2394
2    livestream, 3 h      10800    0.048    9.0942
3    clickbait, 40 s         40    0.152   -3.1433

interpolated examination, every slot:
  1:0.810 2:0.620 3:0.490 4:0.415 5:0.340 6:0.303 7:0.267 8:0.230 9:0.213
  10:0.195 11:0.177 12:0.160 13:0.151 14:0.143 15:0.134 16:0.125 17:0.116
  18:0.107 19:0.099 20:0.090
  IPW weights span 1.23 (pos 1) to 11.11 (pos 20)
  M = 20 clips positions nothing -- never fires
  M = 10 clips positions [19, 20]
  M =  8 clips positions [17, 18, 19, 20]
  M =  5 clips positions [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]

The slate is the key result: the clickbait candidate has more than twice the click probability of the winner and finishes last.

Why: its not_interested at 0.0094 and report at 0.00021 are 0.0094/0.0021 = 4.5x and 0.00021/0.00006 = 3.5x their base rates, and those two heads enter at weights of -2 and -4. That is The watch time trap’s threat model producing an ordering rather than describing one.

The livestream is the other instructive row. Its 3 hours buy it 0.4 × log1p(10800/60) = 0.4 × 5.20 = 2.08 from the duration term. That is real, and it is nowhere near enough to overcome a 0.06 watch fraction — which is exactly what “saturating” was supposed to mean.

Where the labels come from, and when they arrive

Labels look like bookkeeping, but they contain one of the chapter’s sharpest traps.

Every head is trained on the same logged impression rows. But each head’s label arrives on its own clock — and training on a fixed 24-hour window turns “has not happened yet” into “did not happen,” in a way that is not uniform across items.

Training weights are not scoring weights

The loss is a sum of per-head losses over impressions: binary cross-entropy — the standard loss for a yes/no prediction — for the seven binary heads, and squared error in logit space for watch fraction.

Each of those per-head losses carries a training weight, chosen to balance gradient magnitude across heads whose base rates span four orders of magnitude.

Those are not the w_k from the combine above, and conflating the two is the most common way a head goes quietly dead.

• A training weight decides how much a head influences the shared trunk.
• A scoring weight decides how much its prediction influences the slate.

A head can be well-trained and weighted to zero at serving, or badly trained and weighted heavily. Only the second one shows up in a slate diff, which is why the first one survives for months.

Every head reads the same row and a different clock

The table below is the label-arrival schedule. Censored in the fourth column means the event genuinely happened, but had not yet happened when the training window closed — so the row went into training labelled negative.

The three bolded rows are where the damage is. Compare the last column against the base rates in the head table above to see how much each head’s apparent rate is deflated.

How many labels the 24-hour window destroys

Scale and cost prescribes daily incremental training on the last 24 hours. A censored positive is not a missing row — it is a row labelled negative.

Count them. The platform serves 400e9 impressions a day (Scale and cost), so multiply that by each head’s true base rate and then by its censoring rate:

share      400e9 × 0.0008   × 0.22  =  70.4 M/day  labelled negative
subscribe  400e9 × 0.0004   × 0.31  =  49.6 M/day  labelled negative
report     400e9 × 0.00006  × 0.44  =  10.6 M/day  labelled negative

Those are exactly the heads at base rates 4e-4 and 6e-5 whose weights Ranking argues hardest for.

The survivable half and the dangerous half

Part of that damage is uniform across items, and that part is survivable.

A head trained toward 1 - 0.31 = 0.69x the true subscribe rate is miscalibrated low — its stated probabilities do not match observed frequencies. Metrics and why offline and online disagree argues calibration matters because the heads combine arithmetically. But a constant offset shifts every item equally, so it does not reorder the slate.

The part that reorders the slate is that the censoring rate is a function of the item. A subscribe following a 90-second clip lands inside the session; one following a 25-minute video usually does not. So short videos get their subscribes counted and long videos do not:

subscribe censored by a 24 h window
    videos under  2 min      9 %
    videos over  20 min     47 %

spurious advantage to short videos, in score points, at w_subscribe = 3.0
    3.0 × [ log(0.91) - log(0.53) ]   =   3.0 × 0.541   =   1.62

the DELIBERATE saturating duration term across the same range
    0.4 × [ log1p(20) - log1p(1) ]    =   0.4 × 2.351   =   0.94

Read those two figures against each other. The label window hands short videos 1.62 score points. The duration term, which was deliberately chosen, is worth 0.94 across the same range and points the other way.

The accidental prior is stronger than the deliberate one. That is The watch time trap’s trap re-entering through the data pipeline, after Ranking spent a whole subsection removing it from the objective.

The fix, and two fixes that look equivalent and are not

The fix is per-head label maturity windows: hold each head’s rows until that head’s window closes. So the daily job trains click / watch / like / not-interested on T-1, share and subscribe on T-3, and report on T-7.

Sibling Training runs the same discipline (24 h for reshare and comment, 1 h for click and dwell), and the general form is 2d delayed labels and the maturity correction.

Two alternatives look equivalent and are not:

• One global window at the slowest head, 7 days. It is unbiased. It also costs the click head — 24.4 billion positives a day, the one head where freshness genuinely pays — a week of staleness, in order to protect a head with 13 million.
• Drop the unmatured rows instead of holding them. This is the tempting one, and it reproduces the bias exactly: the rows you drop are the long-video rows. You have thrown away sample size and kept the confound. Only holding a row until its own window closes is unbiased, and it costs storage rather than accuracy.

Serving is the same code path or none of this holds. Train on features logged as served (Trainingserving skew in features), and split temporally — days 1-28 train, day 29 validate, day 30 test. Never a random split: every counter feature in the item-scalar family bridges the two halves, so a random split lets the future leak into the past.

The ranker and the value model, assembled

• What are the models? Two. The ranker is a wide network over 1,500 input dimensions, with an MMoE trunk of 8 experts and per-task gates feeding 8 heads. The value model is a tiny regression whose parameters are the 8 scoring weights.
• What do they take as input? The ranker reads the seven feature families above — 968 dimensions fetched per candidate, 532 once per request. The value model reads the 8 head predictions and a duration term.
• Where do the labels come from? The ranker’s labels are logged user actions, each on its own clock, each held until its own maturity window closes. The value model’s target is deliberately not logged actions but an out-of-band 28-day outcome — 200 k surveys, retention, reported regret — precisely so it cannot be gamed from inside a session.
• How are they trained? The ranker retrains daily on the last day’s matured rows, with a weekly full refresh, IPW-weighted and temporally split. The value model is refit quarterly, because its target takes 28 days to resolve.
• How are they served? The ranker scores 300 candidates in 16 ms of a 75 ms path. The value model is eight multiplications and an addition, and the integrity demotion is added in log space in the same step.
• How do you know they work? Per-head AUC and calibration offline; a doubly-robust off-policy estimate on the uniform-random slice; D7/D28 return rate in a permanent holdout online (Metrics and why offline and online disagree).

What must be true for the ranking stage to hold

1. Every head is calibrated. The heads are combined arithmetically, so an overconfident head silently changes the weights you set.
2. No single creator lever moves all eight heads the same way. That is the entire threat model, and it would be false if the survey head were observable to creators.
3. 0.58 MB of per-candidate features can be fetched in about 24 ms. This is what sizes the fleet in Scale and cost. Halve the bytes per candidate and the fleet halves; halve the FLOPs and nothing happens.

4. The watch-time trap

The ranking design above looks the way it does because of watch time. It is the metric every recommender is asked to maximize, and it fails in three separate ways — one about the arithmetic of a product, one about content length, and one about the difference between a session and a lifetime. The third is the dangerous one because every short experiment endorses it.

Mechanism 1 — the product decomposes and one factor is easy to game

The first failure is arithmetic. Expected watch time is a product of two factors, and they are not equally hard to move:

E[watch_time]  =  P(click) × E[watch | click]

item A, honest       P(click) 0.040   E[watch|click] 180 s   ->  7.20 s
item B, clickbait    P(click) 0.140   E[watch|click]  62 s   ->  8.68 s

B wins on expected watch time while being worse on every per-viewer measure, because inflating the click factor is cheap — it takes a thumbnail and a title — while raising retained attention is expensive. Any objective that multiplies a cheap factor by an expensive one will be optimized through the cheap factor.

Mechanism 2 — pure watch time is a duration prior wearing a disguise

The second failure is that total watch time rewards length rather than quality, and it does so silently — a prior here meaning a standing thumb on the scale that the model never had to justify:

3-hour livestream, watched  6%   =  10,800 s × 0.06  =  648 s
6-minute tutorial, watched 90%   =     360 s × 0.90  =  324 s

The livestream wins by 2x while satisfying the user far less. Ranking on total watch time is therefore a systematic bet on long content, and creators respond to it within weeks by padding.

The obvious fix — rank on watch fraction — inverts the bias: a 15-second clip watched 100% beats everything. Neither term is right alone; you need both, which is why the head predicts fraction and the score carries a separate, saturating duration term. Saturating, not linear, so that “longer” stops paying above a few minutes.

Mechanism 3 — the sign flip across sessions

The third failure is the one that makes this a trap rather than a bug: the relationship between watch time and satisfaction reverses depending on the timescale you measure it over. D7 and D28 below mean the share of users who come back within 7 and within 28 days.

                              20-40 min      >90 min
                              sessions       sessions
survey satisfaction (1-5)        4.1            2.7
D7 return rate                  0.71           0.63
"I regret this session" rate    0.04           0.19

Within a session, watch time and satisfaction are positively correlated. Across sessions, they are negatively correlated. A ranker optimized on a within-session objective therefore climbs a metric that is anticorrelated with the outcome the business actually has — and every short A/B confirms the climb, because a two-week experiment cannot see D28.

The design responds in three parts, and each one of them appears somewhere else in this chapter as a component:

• Multi-objective with explicit satisfaction terms (Ranking), so the score contains something that falls when the session goes bad.
• A value model whose target resolves 28 days out, so the weights are fit against the long horizon even though the ranker runs against the short one.
• A permanent long-term holdout — a holdout being a slice of users deliberately kept off every experiment, here 0.5% on a frozen policy, never reassigned — because the across-session effect is only visible by comparing populations over months. Without the holdout, the sign flip is undetectable by construction, and the system will optimize itself into the corner one confidently-shipped A/B at a time.

5. Position bias and the feedback loop

The system’s training data is produced by the system itself. That single fact is why training on raw clicks makes a model of your previous model, why a standard correction exists, why — unusually — the textbook refinement to that correction is worthless here, and why the loop only ever lets you learn about a small fraction of the corpus.

The decomposition, and why naive training is circular

A click is not evidence of relevance on its own, because the user has to look at a slot before they can click it. Everything that follows hinges on splitting those two effects.

P(click | item r shown at position j)  =  examination(j) × relevance(r)

Examination is the probability the user looked at position j at all. Relevance is the probability they would click if they did look. A logged click confounds the two, and the model has no way to tell them apart from the log alone.

Examination can be measured honestly, though. Randomly swap the contents of positions on a small slice of traffic — 0.5% here — so that what sits in a slot is unrelated to how good it is. Then any remaining difference in click rate across slots is pure position effect.

Here is what that measurement returns. Note that only seven of the twenty positions are measured — that gap turns into a real bug two subsections down.

Position 1 is looked at nine times as often as position 20. That is the size of the confound.

Training on raw clicks fits the product, examination(j) × relevance(r). And j was assigned by the previous model, so the new model learns “items the old model liked are good,” which is a fact about the old model. Iterate that and you get a system that is extremely confident about a shrinking set of items.

Inverse propensity weighting

The standard fix is to undo the bias by weighting — though the refinement everybody adds on top of the weighting turns out, on a 20-slot slate, to buy nothing.

Inverse propensity weighting (IPW) works by weighting each logged example by the inverse of the probability it had of being observed. The propensity here is the examination probability, so rare-to-be-seen events count for more, exactly in proportion to how rarely they were seen:

weighted_loss  =  sum over logged events  loss( click / examination(j) )
                                                  ^^^^^^^^^^^^^^^^^^^^
                       the weight rides on the CLICK, not on the row

E[corrected label for item r shown at position j]
    =  examination(j) × relevance(r) × (1 / examination(j))   <- it clicked
     + (1 - examination(j) × relevance(r)) × 0                <- it did not
    =  relevance(r)                                   <- position cancels

The second half of that block is the proof, in two lines. A logged impression at position j is a click with probability e_j · r, in which case the corrected label is 1/e_j; otherwise the label is 0. The expectation is e_j · r × (1/e_j) = r. Position cancels.

Where the 1/e_j sits, because the obvious placement is a no-op

Almost everyone writes the correction as sum over logged events (1/e_j) · loss(r) — the weight multiplying every logged impression at position j, clicks and non-clicks alike.

That does nothing. 1/e_j is then a positive constant multiplying that position’s entire objective, and scaling an objective by a positive constant cannot move where it is minimized. You converge to e_j · relevance(r) — exactly the biased quantity IPW exists to remove.

The estimator that works is the Horvitz-Thompson one written above, where the click contributes 1/e_j against unit impression mass. Said plainly: the correction weights the positive term only.

The code below fits all three schemes in closed form and prints where each converges. Note the middle column: it is identical to “no correction at all.”

def logloss_argmin(pos_mass, neg_mass):
    """argmin over q of  -pos*log(q) - neg*log(1-q)  is pos / (pos + neg)."""
    return pos_mass / (pos_mass + neg_mass)


def masses(e, r, scheme):
    """Expected (positive, negative) loss mass from ONE logged impression.

    A logged impression at position j is a click with probability e*r, so an
    UNWEIGHTED fit sees positive mass e*r against unit total and converges to
    e*r -- the position bias, baked in. Putting 1/e on every logged row scales
    both masses by the same 1/e, which is why it changes nothing.
    """
    click = e * r
    if scheme == "unweighted":
        return click, 1.0 - click
    if scheme == "weight_every_event":          # the formula as it is usually written
        return click / e, (1.0 - click) / e
    if scheme == "weight_the_click":            # Horvitz-Thompson: label = click / e
        return click / e, 1.0 - click / e
    raise ValueError(scheme)


RELEVANCE = 0.30
print("relevance r = %.2f\n" % RELEVANCE)
print("pos   e     unweighted   1/e on ALL rows   1/e on the CLICK")
for _j, _e in ((1, 0.81), (8, 0.23), (20, 0.09)):    # three knots of the table above
    _q = [logloss_argmin(*masses(_e, RELEVANCE, s)) for s in
          ("unweighted", "weight_every_event", "weight_the_click")]
    print(" %2d  %.2f      %.4f           %.4f            %.4f"
          % (_j, _e, _q[0], _q[1], _q[2]))
    # the reflexive placement is algebraically identical to no weighting at all
    assert abs(_q[1] - _q[0]) < 1e-12
    assert abs(_q[0] - _e * RELEVANCE) < 1e-12      # recovers examination x relevance
    assert abs(_q[2] - RELEVANCE) < 1e-12           # recovers relevance, at every j

relevance r = 0.30

pos   e     unweighted   1/e on ALL rows   1/e on the CLICK
  1  0.81      0.2430           0.2430            0.3000
  8  0.23      0.0690           0.0690            0.3000
 20  0.09      0.0270           0.0270            0.3000

The middle column is the bug: it is the left column, exactly, at every position. A correction that is algebraically a constant factor on the objective it corrects is decoration, and it is the kind of decoration that survives review because the formula it is written as looks like the textbook one. The right column is what Metrics and why offline and online disagree’s off-policy estimators and the IPW-weighted retraining in the Ranking diagram are both actually doing.

Price the weight clip before importing it

IPW weights can get large, and a large weight means one logged event dominating the gradient. The standard reflex is to clip them:

w  =  min( 1/examination(j),  M )

Capping at M trades a computable bias for a reduction in variance. That is a genuine trade in general. Price it on this slate before importing it, because here the answer is not the textbook one.

20 slots, examination(20) = 0.09  ->  the largest weight the system can
                                      produce is 1/0.09 = 11.1
weights therefore span 1.23 (position 1) to 11.1 — a range of 9x

interpolate examination over all 20 positions, uniform relevance:
    M = 20    clips nothing. The operation never fires.
    M = 10    clips positions 19-20    bias 0.6 %    variance  1.03x
    M =  8    clips positions 17-20    bias 3.5 %    variance  1.2x
    M =  5    clips positions 10-20    bias 18 %     variance  2.2x

A clip is worth exactly what the tail of the weight distribution is worth, and a 20-slot slate has no tail. The whole weight range is 9x. There is no M that buys a meaningful variance reduction without a bias large enough to change conclusions.

M = 20 is the worst of the four options, because it looks like a control and clips nothing. A parameter that never binds is a parameter that will be cited in a design review and audited by nobody.

“Interpolate over all 20 positions” is load-bearing

The randomized swap measures seven of the twenty positions. A propensity table keyed only on the measured positions is missing thirteen of them, and an IPW routine that drops events with no logged propensity therefore discards 13/20 = 65% of every slate.

The volume is not even the worst part. The survivors are positions 1, 2, 3, 5, 8, 12 and 20 — so the discard is a function of position, which is precisely the confound IPW exists to remove. You would have built a second position bias while removing the first.

Linear interpolation between the measured knots fills the table. It is that interpolation that produces the clip boundaries above, and the code block in Ranking asserts all four of them.

What to use instead of a clip

Clipping is a technique for unbounded propensities: logged bandits with 1e-4 action probabilities, or deep result pages where examination decays two more orders of magnitude. Neither describes a 20-slot slate.

Two controls belong here instead, and both are structural rather than numeric.

• Log a propensity floor. Refuse to train on any event whose propensity is unknown or below the floor, rather than up-weighting it enormously.
• Use self-normalized IPS. IPS is inverse propensity scoring — the same weighting used as an estimator rather than as a training weight. The self-normalized form divides by the sum of the weights instead of by the count, which controls variance without introducing a bias you then have to apologize for.

Set that floor to the slate’s own minimum propensity, 0.09, and not to a round number. A floor of 0.05 cannot fire, because nothing a 20-slot slate produces goes below 0.09 — it is the same dead parameter as the M = 20 clip, sitting in the config looking like a control.

At 0.09 the floor binds by construction. Every legitimate slate position clears it, and anything under it is an event from another surface or a corrupted log — which is exactly the population the floor is for.

The honest version of “we use IPW” is showing that you priced the clip. Here, pricing it says not to.

Where the propensities come from

All of the above depends on knowing the propensities, and there are three ways to get them.

Two terms first. Confounded means a third factor — the model’s own opinion of the item — drives both the position and the click, so the measurement conflates them. EM is expectation-maximization, an algorithm that infers hidden quantities by alternating between guessing them and refitting the model.

Pay for the randomized swap. It is the only unconfounded source, it is 0.5% of traffic, and every downstream correction depends on it being right.

The feedback loop, quantified

Position bias is one half of the circularity. The other half is that most of the corpus is never shown at all — and the number on how much of the catalog the system can ever learn about is smaller than anyone guesses:

800 M eligible videos
videos receiving >= 100 impressions/day        4.1 M      =  0.51 %
videos receiving >= 1,800 impressions/day      0.9 M      =  0.11 %

Ninety-nine and a half percent of the corpus generates no training signal, so the “800 M item corpus” is really a 4 M item corpus with a very expensive index attached. An item outside that set has an embedding built only from content features, is never impressed, never gets interaction data, and never improves — a closed loop with no way in.

Exploration is the way in, and its cost is visible while its benefit is not

The only way into a closed loop is to deliberately show things the model does not yet rate. How much of that is needed can be derived rather than guessed — and the derivation matters, because every two-week experiment will vote to stop doing it.

Keep 1 of every 20 slots reserved for the exploration pool, ranked with a Thompson or UCB bonus over the posterior — the current probability distribution over an item’s true engagement rate, which is wide when the item has little data and narrow when it has a lot (Thompson sampling).

How much exploration is actually needed

Do not guess a percentage. The number falls out of how much data one item needs before its estimate is worth ranking on.

Say you want to estimate a click-through rate near 0.05 to within 20% relative — a 95% confidence interval whose half-width is a fifth of the estimate. se is the standard error, the expected size of the estimate’s wobble, and 1.96 is the standard-normal multiplier for 95% coverage.

want  1.96 · se(p)  =  0.2 · p  =  0.010
      se            =  0.010 / 1.96          =  0.0051
      n             =  p(1-p) / se^2
                    =  0.05 × 0.95 / 2.60e-5
                    =  0.0475 / 2.60e-5      =  1,827 impressions

So one new video needs about 1,800 impressions before its CTR estimate is worth trusting. Now scale that to the daily upload cohort and divide by the impressions the platform actually serves:

graduating the daily upload cohort
    500,000 new videos/day × 1,827   =  914 M exploration impressions/day

total impressions/day
    20e9 requests × 20 slots         =  400e9

exploration share needed             =  914e6 / 400e9  =  0.23 %

A slot reservation of 1 in 20 is 5%, and the derived need is 0.23%. So the binding question is not “can we afford exploration” but “are we spending it on the right items.” Spend it on the fresh pool and on items whose posterior is wide, not uniformly.

That leaves a 5 / 0.23 = 22x gap between what is reserved and what is derived. That is not a rounding error and it should not be left standing. Two honest readings, with different consequences:

• The reservation is a cap, not a quota. A reserved slot is only spent when the exploration pool actually holds an item that needs it, so the realized share is whatever the uncertainty rule spends — possibly far under 5%. If that is the design, then the number to report and to price in an A/B is realized share, and if nobody logs it, the “-0.4% engagement” result below is uninterpretable: you do not know which policy it measured.
• Or the 0.23% is undercounted, in which case say what it is missing. It covers only first-estimate precision for the new-upload cohort. It does not cover new users (Cold start), re-estimation of items whose posterior has widened under drift — which prices out at a further 0.012%/day, so not this — or maintaining the propensity support that Metrics and why offline and online disagree’s off-policy estimators need on the tail. Only the last of those is plausibly large, and nobody has ever measured it.

Either way the discipline is the same: a slot reservation is a ceiling and a derived requirement is a floor, and a system that reports the ceiling while spending an unlogged amount cannot tell you what exploration cost. Log realized exploration share per request, alongside the propensity.

Two instruments, two answers

Now measure the same intervention two ways. The clocks are different: exploration’s cost lands inside a two-week window and its benefit does not.

2-week A/B, exploration on vs off:
    engagement          -0.4 %      significant, negative
    ship decision                   kill it

90-day persistent holdout:
    corpus coverage     0.51 % -> 2.8 %
    creator D30 retention        +11 %
    engagement                   +1.2 %      significant, positive

In any two-week experiment exploration is only ever a cost, because the benefit is a supply-side effect that takes months to arrive. Most teams kill exploration in an A/B and never learn what it was worth. The persistent holdout is the only instrument that can see it, which makes the holdout an infrastructure decision rather than an experiment.

6. Cold start

Exploration budgets one flavour of ignorance; cold start is the general problem of serving something the system has no history for, and it comes in three flavours here — a new user, a new item, and a user whose stored representation is simply out of date. The three have genuinely different fixes, and the third is where most candidates reach for the wrong objection.

New user

A user at their first request has no history, so the long-term tower has nothing to encode. What is available at request 1 is only locale, language, device, time of day, referral source, and whatever onboarding collected — and the measurable result is that asking beats modelling:

new-user session-2 return rate:
    popularity-by-locale only                          0.42
    + 3 explicit topic picks at onboarding             0.57
    + Thompson bandit over 40 topic clusters,
      updated per interaction                          0.61

Look at the size of the first jump: 0.42 to 0.57 for three taps, then 0.57 to 0.61 for a whole bandit on top.

Three onboarding taps are worth more than any model you can build for a user with zero history, because they are the only source of information that exists.

The bandit then narrows. Each of the 40 topic clusters is an arm — one option the bandit can choose — and each carries a Beta posterior, the standard probability distribution over an unknown success rate, updated after every success or failure.

Useful discrimination arrives after roughly 15 interactions, which is one session. But only because the arms are not independent, which is the key point. Fifteen observations across 40 arms is 15/40 = 0.4 pulls per arm, which discriminates nothing on its own.

What makes it work is that the 40 clusters share a hierarchical prior over their embedding neighbourhood: nearby topics are assumed to behave similarly, so a pull on one arm also updates its neighbours. State this explicitly: a flat Beta-per-arm bandit at 40 arms needs closer to 600 interactions. Mechanics in Contextual bandits the practical middle ground.

New item

A new item is half-solved by construction and half not, and separating the halves is the point.

Solved: the content-feature tower gives a usable embedding at t = 0 (Candidate generation), so a video uploaded ninety seconds ago is retrievable.

Not solved: it gives no reliable engagement estimate. That is the 1,827-impression number above, and no amount of content understanding substitutes for it.

So the design adds a fresh pool with three properties:

• an impression floor: every new item gets a guaranteed budget of impressions, spread across diverse user cohorts rather than concentrated;
• a graduation criterion: leave the pool when the CTR posterior half-width falls below 20% relative, or at 72 hours, whichever comes first;
• cohort-spread, because 1,827 impressions all shown to one interest cluster estimates that cluster’s CTR, not the item’s.

Stale user representations, and splitting by update frequency

The third cold start is not about missing data but about stale data, and it is the one where the obvious objection to the obvious fix is wrong.

A user who just watched six pasta videos should see cooking content in the next request. If the user tower is recomputed nightly, they will not.

The reflex is to recompute the tower per request. Price that before rejecting it, because the objection everybody reaches for does not survive contact with the arithmetic:

the FLOP argument, and it FAILS:
    509,000 req/s × 0.3 MFLOP per user-tower pass  =  0.15 TFLOP/s
                                                   =  0.001 A100
    against the ranker's 4.5. The arithmetic is free, exactly as §9 says.

“Too expensive” is not the objection, and a candidate who reaches for it is guessing. There are two real objections, and the second is decisive.

Objection 1 — the cost is the input, not the pass. The long-term tower reads an aggregate over the user’s whole history: hundreds of counters plus a ~2,000-event watch sequence, about 8 KB. That is a batch-shaped read.

509,000 req/s × 8 KB  =  4.2 GB/s of RANDOM reads
                         against a 500 M-row store
lands as               ~6 ms  inside an 8 ms user-side fetch budget (§9)

It shows up in the latency table, not the GPU bill. That is the general shape of every “why not just recompute it” question in this chapter.

Objection 2 — the benefit is close to zero, because you would be refreshing the one component that is definitionally insensitive to the update.

The long-term tower is an average over months. Six new pasta videos against a 2,000-event history move a mean-pooled embedding by 6 / 2,006 = 0.3%.

So you would pay 6 ms of p50 latency to move the vector by a third of a percent — in the direction of a signal that the last-20-actions pool already represents at full strength and zero cost.

Split by refresh cost

The table below is the resolution. One row per component of the user representation, ordered by how expensive that component is to refresh.

Split the user representation by how expensive each part is to refresh:

The mean-pooled recent-actions vector feeds both a short-term retrieval source and the ranker directly. The expensive representation updates slowly and the cheap one updates instantly, which is the correct decomposition whenever refresh cost is non-uniform — and it is the answer to “your recommendations do not react to what I just watched.”

What must be true for these cold-start fixes to hold

1. Users will answer three onboarding taps. That is a product decision that can be withdrawn, and it takes 0.57 - 0.42 = 15 points of session-2 return with it.
2. An item’s CTR is stable enough that 1,827 impressions estimate something durable. False for content whose appeal is a news cycle.
3. Long-term taste really does move slowly relative to a session. That is what makes a nightly tower acceptable. On a surface where users switch context every few minutes, the nightly component would be nearly worthless and the whole budget should go to the real-time pool.

7. Metrics, and why offline and online disagree

“How do you know it works” has an offline answer and an online answer, and in this system specifically they disagree in sign about three times in five. The disagreement is structural rather than statistical, which means no amount of extra data fixes it and one particular slice of traffic does.

Offline

Four offline measurements, each with the failure it hides. NDCG is normalized discounted cumulative gain, which scores an ordering while discounting positions further down the page, and a doubly-robust estimator combines a model of the reward with propensity weighting so that it stays correct if either of the two is right:

Calibration is not optional here and it is for a structural reason. In a single-model classifier you can miscalibrate and fix it with a threshold. Here, eight heads are multiplied by weights and summed; if the click head is overconfident by 15% and the share head is underconfident by 10%, the effective weights are not the weights you set. See Calibration what it means and when it matters.

Online

Six online measurements, one primary and five guardrails. Two of the names need glossing: the Gini coefficient of impression share is a single number between 0 and 1 saying how unequally impressions are distributed across the corpus, where 0 is perfectly even and 1 is one item taking everything; and topic entropy, measured in nats, is how spread out a user’s viewing is across topics, where a lower number means a narrower diet.

They disagree, and here is why

The two tables above are supposed to agree with each other and do not. Three mechanisms drive the disagreement, and the first is not a bug in anyone’s pipeline but a property of how the data was collected. Start with its size.

Five real launches. Read each row by comparing the sign of column 2 against the sign of column 3.

launch   offline AUC delta   online engagement delta   shipped?
  A          +0.004                 +0.9 %              yes
  B          +0.011                 -0.3 %              no
  C          -0.002                 +1.4 %              yes
  D          +0.007                 +0.1 %              no
  E          +0.009                 -1.1 %              no

Sign disagreement in three of five — B, C and E. And a sharper count on top of that: the offline winner was the online loser twice, B and E.

Those are two different counts of two different things, and conflating them is how a system-health number gets quietly halved. B and E are cases where offline actively pointed the wrong way; C is a case where offline said “no” and online said “yes.”

Three mechanisms produce this, and the first is the deep one.

1. Offline evaluation is scored on data logged by the old policy. A new model’s favorite items were, by construction, rarely shown, so they have no labels. The evaluator either drops them (and scores the model only on the overlap with the old policy) or treats them as negatives. Either way, the model that ranks most like production scores best offline. Offline AUC systematically rewards similarity to the logging policy, which is exactly the property a candidate model is trying not to have.

2. Offline metrics are per-request; online outcomes are per-session and per-user. Crowding out, repetition across requests, and slate-level diversity have no representation in a per-request AUC.

3. The correction that is applied at serving may not be applied in eval. If the ranker is trained IPW-weighted but evaluated on raw logged clicks, the evaluation carries the position bias the training removed.

Three fixes follow, in order of how much they buy:

• A permanent uniform-random slice. 0.1% of requests serve a uniformly random sample from the candidate pool. Size it: 0.001 × 20e9 = 20 M requests/day, and 20 M × 20 slots = 400 M unbiased impressions/day — enough to evaluate any ranker’s top-20 with real statistical power, at a cost of 0.1% of engagement. This is the highest-value 0.1% of traffic in the system, because it is the only data that is not a function of the model being evaluated.
• Doubly-robust off-policy estimation on that slice plus the propensity-logged main traffic (Off policy evaluation and why bandits get it and rl does not).
• Log the candidate set and the propensity, always. Retrofitting propensities is impossible; a log without them is permanently un-debiasable.

8. Failure modes

Six failure modes account for how this system degrades in production. Four of them are feedback loops — the system’s output becomes its own input — and the useful habit is to read each as a gain: if one pass through the loop multiplies some concentration by a factor above 1, the failure is not a risk but a certainty on a schedule.

8.1 Filter bubbles

A filter bubble is what happens when a user’s recommendations narrow until they no longer see anything outside a shrinking band, and the mechanism is a loop with gain slightly above 1.

Mechanism, as a contraction. The ranker scores highest on items similar to past engagement. Those get shown. Engagement accrues to them. The user’s history narrows. The ranker’s estimate of the user’s interests, derived from that history, narrows. Each pass multiplies the concentration by a factor slightly above 1.

Here is the contraction measured. Every row starts at 3.1 nats and runs the same twelve weeks; only the mitigation differs, so read the right-hand column as “where the user ends up.”

topic entropy of a user's 30-day watch history, 12-week cohort:
    no diversity constraint        3.1 nats  ->  1.9 nats
    per-source quotas only         3.1       ->  2.6
    quotas + MMR slate re-rank     3.1       ->  2.9

Quotas alone recover 0.7 nats of the 1.2 lost; MMR on top recovers another 0.3.

The quota does most of the work and the re-ranker does the rest, in that order, because a re-ranker can only reorder what retrieval already returned. If co-watch supplied 90% of the pool there is nothing to diversify toward.

Report topic entropy of served slates and of user histories as a standing guardrail. Slate entropy alone is gameable — you can serve a diverse slate that the user never clicks and the history keeps narrowing anyway.

8.2 Popularity bias amplification

The second loop is about items rather than users: popularity is a feature and an outcome, which makes it a positive feedback loop with gain above 1 — showing a popular item makes it more popular, which makes the feature larger, which shows it more.

impression-share Gini across the corpus:
    month 0    0.71
    month 6    0.83     unmitigated
    month 6    0.74     with logQ correction + popularity residualization
                        + per-channel slate caps

The mitigation that matters most is residualizing popularity features: feed the model how much an item beat its own expectation, rather than its raw total.

instead of      raw view count
feed            views - E[views | age, category, channel size]
                ^^^^^   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                actual  what a video of that age, category and
                        channel size would typically get

The raw feature lets the model learn “popular things get clicked,” which is true and useless — it is a fact about the past, and feeding it back closes the loop. The residual lets it learn “this performed better than its cohort,” which is information the model did not already have.

8.3 The feedback loop makes offline eval optimistic

The third loop runs through your evaluation rather than through your users, which is what makes it hard to notice. Metrics and why offline and online disagree derives it in full; the compact statement is: your offline evaluation set is a sample drawn by your current model, so it flatters models that resemble your current model, and the flattery grows every time you ship. The uniform-random slice is the only exit.

8.4 Clickbait and engagement bait

The fourth loop closes outside your company entirely.

It is the The watch time trap mechanism seen from the supply side: creators A/B their own thumbnails against your ranker, at a scale you cannot match. That makes it the same arms-race structure as The multimodal problem and the arms race.

Four controls hold the line:

• the satisfaction head carrying a real weight;
• the not-interested head;
• a feature measuring how well the title and thumbnail match the actual content;
• post-click retention, used as a per-video quality prior that decays slowly.

8.5 Training/serving skew in features

Training/serving skew is when a feature is computed one way while training and a different way while serving, so the model sees inputs at serving that it never saw while learning. It is the classic recommender outage, and it is silent:

feature: user_watch_count_7d
    training  computed from a batch table, complete through midnight
    serving   computed from a streaming store with a 40-minute lag

result    the serving value is systematically lower than the training value
          for exactly the users who are active RIGHT NOW -- i.e. everyone
          in the request
          AUC offline 0.79, online engagement -2.1 %

The skew is correlated with the thing you are predicting, which is what makes it lethal rather than noisy. The mechanisms and fixes are in Feature stores and trainingserving skew; the recsys-specific discipline is to log features as served and train on those logs, so training and serving are the same code path by construction.

8.6 The integrity coupling

The last failure is where ranking meets safety, and it contains a sign error that ships as working code.

Borderline content is material that is not against policy but sits close to the line. It is reliably engaging, so a ranker with no integrity input will find it and promote it. The demotion tier from ch 05 is what stops that.

The integrity score belongs in two places, and they do different jobs.

• As a feature in the ranker. This is what generalizes: the model learns that borderline-adjacent recommendations lower satisfaction, and that transfers to items the classifier has never scored.
• As a graded demotion in the re-rank. This is what is enforceable: it is a number in the score you can audit and roll back, and unlike a feature it does not depend on the trunk having learned anything.

A post-hoc filter is a third thing and it is strictly weaker than both. It is a hard threshold, so it is all-or-nothing on a continuous score; it teaches the model nothing; and it silently changes the slate composition below the cut with no record of what it removed.

“Multiplier” is the wrong word, and the wrong word ships as the wrong code

The combined score is a weighted sum of signed multiples of log-probabilities, so its sign is a fact about the item, not about the design. It is -24.19 on a weakly-engaging item and +8.15 on an item at the head table’s own base rates, because the negative heads enter as -w · log(p) on rare events.

Multiply by an integrity score in (0, 1] and the effect goes both ways:

weak item        -24.19 × 0.5  =  -12.10     UP the slate
base-rate item    +8.15 × 0.5  =   +4.08     down the slate

The bug is non-monotonicity, and it is worse than a uniform sign error. At one integrity score it promotes weak items and demotes ordinary ones, so a flagged item gains rank against the very population it was supposed to lose rank against — and there is no single sentence about direction that holds.

The demotion has to be score + log(integrity). Adding a log is multiplication in the space the score is a log of, and it is monotone decreasing for every item regardless of sign.

Ranking derives it, and its code block asserts it on both a weak item and a base-rate one. Asserting it on the weak item alone is a test that passes because the fixture was chosen to make it pass.

Summary

Every failure above reduces to a mechanism that causes it, a measurement that would reveal it, and a control that holds it — and being able to reproduce this table from memory is a fair test of the whole chapter:

9. Scale and cost

Pricing the system takes three passes — the arithmetic floor, the latency budget, and the fleet you actually rent — and the gap between the first and the third is the whole point. The FLOP number tells you which design is impossible. The byte count tells you what the possible one costs.

The arithmetic floor

Start with FLOPs, because they are the easiest to compute and the least binding.

Two terms in the block below. HNSW is the graph-based vector index the retrieval side runs on, and M is how many neighbour links each vector keeps — the per-vector memory cost of those links is derived in Hnsw memory per vector derived, which is what “RAG §4.3” points at.

Read the block in four parts: index memory, user embeddings, the two model fleets, and then the dollar figure at the bottom.

retrieval index    800 M × 128-d fp16                    =  205 GB
                   HNSW graph overhead at M=32, see RAG §4.3   ≈  +205 GB
                   sharded 16 ways, 3 replicas            =  ~1.3 TB resident

user embeddings    500 M × 128-d fp16                     =  128 GB
                   refreshed nightly: 500 M forward passes
                   at 0.3 MFLOP each = 150 TFLOP = 1 GPU-second   negligible

ranking fleet      300 items × 4.39 MFLOP × 509 k/s = 671 TFLOP/s
                   at 150 TFLOP/s effective                =  4.5 A100
pre-rank fleet     2,000 × 0.05 MFLOP × 509 k/s = 51 TFLOP/s  =  0.3 A100
                                                              -----------
                                                        FLOP floor: ~4.8 A100

this is a FLOOR, not a bill. Serving is memory- and fetch-bound once the
candidate set is small, so the real fleet is 1-2 orders above it; the number
that matters here is the RATIO to 11.9 M, not the absolute.

cost at the floor  4.8 × 24 × $2.00                    =  $230/day
per request        $230 / 20e9                         =  $1.2e-8

Where the 80 ms goes

Framing and the arithmetic that forces two stages names 100 ms end to end and ~80 ms for ML, and calls latency the binding constraint three times. Spend that budget line by line and the interesting result appears: half of it goes to moving bytes rather than doing arithmetic. KV below is a key-value store, and hydration means fetching the stored features and payloads for a set of ids:

request parse + auth + user feature fetch (KV)           8 ms
user tower forward pass, 0.3 MFLOP                       1 ms
6 retrieval sources, fanned out IN PARALLEL             14 ms   <- slowest, not the sum
dedupe + per-channel cap + seen-filter                   3 ms
pre-rank, 2,000 -> 300, 0.05 MFLOP each                  5 ms
feature hydration for 300 candidates, 0.58 MB           24 ms
full ranker, 300 candidates, 1.32 GFLOP                 16 ms
value combine + integrity demotion + MMR slate           4 ms
                                                       ------
                                                        75 ms  ML path, p50
                                                       ~96 ms  p95 with queueing

payload hydration (titles, thumbnail urls) + serialize   20 ms
                                                       ------
                                                        95 ms  end to end, p50

Read the convention before copying the number. This 100 ms is the server-side recommendation call measured at the gateway; the client round trip is outside it. That is why the ML share is 75% here and 20% in Scale and cost — that budget carries 60 ms of content-payload hydration and 45 ms of serialization and network inside its 269 ms. The two numbers are not comparable and a candidate who quotes “80% for ML” without the boundary will be corrected.

Two things in that table are the whole story.

Fetch is 38 of the 75 ms — the retrieval fan-out plus feature hydration — and neither is arithmetic. This is the demonstration behind the sentence Framing and the arithmetic that forces two stages and Scale and cost both assert: once the candidate set is small, FLOPs stop binding. Shaving the full ranker from 16 ms to 12 ms buys 5% of the path; batching the 300-candidate hydration into one multi-get per shard instead of per-candidate buys three times that.

Even the 16 ms of “ranker” is mostly not ranking. The dense arithmetic for 300 candidates is 1.32 GFLOP / 150 TFLOP/s = 8.8 microseconds — the wall clock is 1,800x that, and the difference is embedding gathers, kernel launch overhead, and a batch of 300 that does not come close to filling an A100. Batching across concurrent requests recovers most of it, which is why the fleet lands at ~20x the FLOP floor rather than 1,800x.

Why the real fleet is ~100 nodes and not 4.8

The actual machines come to 22 times the arithmetic floor, for a reason that inverts the usual intuition about where to optimize.

The FLOP floor above omits retrieval (4.6 MFLOP/request, Framing and the arithmetic that forces two stages) because it rounds to zero. What it also omits is the reason the fleet exists at all — the fleet is sized by how many bytes have to be online and how many copies survive a shard loss, and the arithmetic rides along free.

ANN index          410 GB (vectors + HNSW graph)
                   16 shards × 3 replicas                       48 nodes

ranker item features (families 4-7 of §3)
                   800 M × 1,936 B  =  1.55 TB
                   20 shards at 80 GB × 3 replicas              60 nodes
                                                              ---------
                                                               108 nodes

FLOP floor                                                     4.8 A100
ratio                                                              22x

Then check what those nodes are doing with their arithmetic:

509,000 req/s over 60 ranker nodes  =  8,483 req/s per node
                                    ×  1.32 GFLOP  =  11.2 TFLOP/s
against 150 TFLOP/s effective       =  7.4 % utilization

The ranking fleet runs at 7% arithmetic utilization because it was sized to hold 1.55 TB, and that one number is the entire “the FLOP floor is not the bill” argument made concrete. It is also the design lever: anything that shrinks bytes per candidate — int8 embeddings, meaning one byte per number instead of two, hashing the channel-id tail, dropping the ASR embedding if it does not earn its 512 bytes — moves the fleet, and anything that shrinks FLOPs does not.

Sizing the training data

The last thing to size is the training data, and it is the only line in the section that surprises people upward.

The block below does three things: counts the raw daily impression log, applies the row-retention rule that shrinks it, and restates the per-head maturity windows from Ranking that govern when each row is allowed to be used.

20e9 requests/day × 20 impressions  =  400e9 impressions/day
logged features, ~600 B/impression   =  240 TB/day raw

positive := a click, base rate 0.061 (§3) -- this is the ROW-RETENTION rule
for the ranker's training set, not a head's label; the towers' positive is
a satisfied watch (§2), and each head reads its own event (§3)
    positives    400e9 × 0.061        =  24.4e9  ->  14.6 TB   kept in full
    negatives    375.6e9 / 20         =  18.8e9  ->  11.3 TB   sampled 20:1
                                                     --------
                                                      ~26 TB/day retained
daily incremental training + a weekly full refresh, on PER-HEAD maturity
windows (§3): click/watch/like/not-interested at T-1, share and subscribe
at T-3, report at T-7. A single 24 h window censors 31 % of subscribes
and 44 % of reports into the negative class, and it censors them by video
duration, which is the §4 trap re-entering through the pipeline.

Note what keeping only one negative in twenty does to the class mix:

before sampling   24.4e9 pos : 375.6e9 neg   =  1 : 15.4
after  sampling   24.4e9 pos :  18.8e9 neg   =  1.3 : 1
                  24.4 / (24.4 + 18.8)       =  56 % positive

Negative downsampling is mandatory at this volume, and it breaks calibration — by a factor you can read straight off that ratio. That matters because Metrics and why offline and online disagree established that the heads are combined arithmetically, so a miscalibrated head silently rescales its own weight.

Correct the logits analytically for the sampling rate rather than re-calibrating empirically. The analytic correction is exact and free (Resampling and the calibration it breaks).

10. Alternatives considered and rejected

Every design has a shadow: the seventeen reasonable things you could build instead, why each is genuinely attractive, and the specific number that rules it out.

Read the middle column first: if an alternative does not look tempting, its rejection is not worth studying. RL in the last row is reinforcement learning, which optimizes a sequence of decisions against a long-horizon reward rather than scoring each one independently.

11. Interviewer pushback

The chapter ends as dialogue. Each question below is one an interviewer actually asks, the italic line names what it is really testing, and the answer is the shortest complete version of the relevant derivation — which is also a fair test of whether you could now explain this system to someone else.

“Why two stages? Just rank everything with your best model.” Testing: whether you can produce the gap or only name the pattern. Because it is six and a half orders of magnitude, and that number is forced. A request can afford to score about 300 of 800 M items, and log10(800e6/300) is 6.4.

Here is the pricing. A real ranker over 1,500 input dimensions through 1024/512/256 is about 4.39 MFLOP per user-item pair. Times 800 M items is 3.51 PFLOP for one request, which is 23 seconds on an A100 against an 80 ms budget. At 509 k requests a second that is 1.79 ZFLOP/s, or 11.9 million A100s. Nothing about quantization or batching touches that.

So retrieval has to be an operation whose cost does not scale with the corpus — a precomputed index and an ANN query.

Ranking 300 candidates is 1.32 GFLOP per request, only about 4.5 A100s of arithmetic. I would say explicitly that the real ranking fleet is one to two orders larger than that, because once the candidate set is small the binding constraint is embedding memory and feature fetch rather than FLOPs. The FLOP number tells you which stage is impossible, not what the possible one costs.

The same argument one level down gives you the pre-ranker: 88x cheaper per item, cutting the expensive stage’s input 6.7x. It costs 0.10 GFLOP a request and saves 7.46, so it pays for itself about 75 times over.

“Why two towers rather than one model over the pair?” Testing: whether you know what the separation buys and what it costs. Two things, and they are separate wins that people habitually run together.

First, the item tower is a function of item features — title embedding, thumbnail, channel, duration — not an item id. So a video uploaded ninety seconds ago gets a meaningful embedding. That is the real fix for item cold start, and it is why matrix factorization fails: MF’s item parameter is defined only by its interactions, so with zero interactions the gradient is exactly zero and the vector stays at its random init forever.

Second, because g depends only on i, 800 M item embeddings can be computed once per index cycle and loaded into an ANN index — 205 GB. A request is then one user forward pass and one ANN query. A model that mixes user and item features early does not factorize and has to be evaluated per candidate, which is single-stage in disguise.

The price is real, but it is a rank price rather than a structural one, and I would correct the usual phrasing here. A dot product is a sum of user-feature times item-feature terms, so “short videos on cellular, long videos on wifi” is rank 1 and two dimensions express it exactly.

What the separation actually costs is everything past rank d. A random 500x500 interaction matrix has only 63% of its singular-value energy inside the top 128 directions, so the other 37% cannot be expressed at all. That remainder has to live in the ranker. You buy sublinear retrieval by capping interaction at rank d, and buy the rest back downstream.

“Watch time is the metric leadership cares about. Optimize it.” Testing: the central trap of this problem. I would not, and I can give three mechanisms.

First, expected watch time factors as P(click) × E[watch | click]. Inflating the click factor is cheap — a thumbnail — while raising retained attention is expensive. So a clickbait item at 0.14 click and 62 seconds beats an honest item at 0.04 and 180 seconds, 8.68 against 7.20, while being worse on every per-viewer measure.

Second, total watch time is a duration prior in disguise: a 3-hour stream watched 6% is 648 seconds and beats a 6-minute tutorial watched 90% at 324, so creators pad. Ranking on fraction inverts it and 15-second clips win. You need a fraction head plus a separate saturating duration term.

Third and worst, the sign flips. Within a session, watch time and satisfaction correlate positively. But sessions over 90 minutes score 2.7 on satisfaction against 4.1 for 20-40 minute sessions, and return 8 points less often at D7. So a two-week A/B will always confirm the climb.

The response is multi-objective with explicit satisfaction and not-interested heads, weights fit by a value model against a 28-day out-of-band target, and a permanent 0.5% frozen holdout — because without the holdout the sign flip is undetectable by construction.

“Where do the head weights come from? That sounds hand-wavy.” Testing: whether you have thought about the meta-objective. It is the least principled part of the system and worth being honest about. Eight weights cannot be grid searched — four values each is 4^8 = 65,536 arms — so the default in most teams is one-at-a-time A/B sensitivity plus a small simplex search, about six weeks per pass. The result encodes what the team believed six weeks ago.

The better answer is a value model: collect an out-of-band target — 200 k surveys, 28-day retention, reported regret — and fit the weights so the combined score predicts it. That target is collected outside the recommendation loop and resolves 28 days later, so the ranker cannot hack it inside a session.

Two details matter mechanically. Transform before weighting, because base rates span 0.58 to 0.00006 and a linear blend of raw probabilities cannot let the share head matter at any sane weight. And every head has to be calibrated, because the heads are combined arithmetically — an overconfident click head silently changes the weight I set.

“Your logs say position 1 gets clicked. Just train on clicks.” Testing: whether you see the circularity. Clicks factor as examination times relevance, and examination at rank 1 is 0.81 against 0.09 at rank 20 — measured with a randomized position swap on 0.5% of traffic. Position was assigned by my previous model, so training on raw clicks fits a model of my previous model, and iterating that produces a system extremely confident about a shrinking set of items.

The fix is inverse propensity weighting, and I would be precise about where the weight goes. The corrected label is click / examination(j), so the 1/e rides on the click and not on the row. Put it on every logged row instead and it becomes a constant factor on that position’s whole objective — it cancels, and you converge to examination × relevance exactly as if you had done nothing. With it on the click, the position term cancels in expectation and the fit recovers relevance.

The reflex at that point is to clip the weights, and I would price the clip before applying it, because on a 20-slot slate it does not pay. Examination bottoms out at 0.09, so weights span 1.23 to 11.1 and there is no tail to trim. Clipping at M=20 never fires at all; M=8 costs 3.5% of the relevance mass for a 1.2x variance reduction; M=5 costs 18% for 2.2x.

Clipping is for unbounded propensities — logged bandits, deep result pages. Here I would use a propensity floor plus self-normalized IPS instead, and drop events whose propensity was never logged rather than up-weight them.

And I pay for the randomized swap rather than harvesting natural position variation, because natural variation is confounded: the item was low because the model scored it low.

“How much of your corpus does the model actually know about?” Testing: whether you have looked. Half a percent. Out of 800 M eligible videos, about 4.1 M get 100 or more impressions a day and 0.9 M get the 1,827 impressions an item needs before its CTR estimate has a 20% relative confidence interval. So the 800 M corpus is really a 4 M corpus with a very expensive index attached, and the loop is closed: an item outside the set is never impressed, never gets data, never improves. The only way in is exploration, and the required budget is smaller than people expect — 500 k new videos a day at 1,827 impressions each is 914 M impressions, which is 0.23% of the 400 billion impressions served. So the question is not affordability, it is targeting: spend it on the fresh pool and on wide posteriors, not uniformly.

“Exploration costs us 0.4% engagement in the A/B. Kill it.” Testing: whether you understand what an experiment can see. The A/B is measuring correctly and answering the wrong question. Exploration’s cost is immediate and lands inside a two-week window; its benefit is a supply-side effect that takes months — more items with enough data to be rankable, better creator retention, a broader corpus. On a 90-day persistent holdout, coverage goes from 0.51% to 2.8%, creator D30 retention is up 11%, and engagement is up 1.2%. In any two-week experiment exploration is only ever a cost, so most teams kill it in an A/B and never learn what it was worth. That makes the persistent holdout an infrastructure decision rather than an experiment — you either have one before you need it or you cannot answer this class of question at all.

“Offline AUC is up 1.1 points. Ship?” Testing: whether you know why offline and online disagree here specifically. Not on that. Across five real launches I have seen sign disagreement three times — and twice the offline winner was the online loser, which is the sharper of the two counts.

The mechanism is structural rather than noise. Offline evaluation is scored on data logged by the old policy, so a new model’s favorite items were rarely shown and have no labels. The evaluator either drops them or scores them as negatives, and either way the model that ranks most like production scores best. Offline AUC rewards resembling the logging policy, which is exactly the property a candidate model is trying not to have.

On top of that, offline is per-request while the outcome is per-session, so crowding-out and repetition are invisible.

What I would look at instead: a doubly-robust off-policy estimate, and the 0.1% of traffic serving a uniformly random sample from the candidate pool. That is 20 M requests and 400 M unbiased impressions a day, for 0.1% of engagement — the only data in the system that is not a function of the model being evaluated.

“Users complain the feed is a bubble. Fix it.” Testing: whether you fix it where it is caused. It is a contraction with gain slightly above 1: the ranker prefers items similar to past engagement, those get shown, engagement accrues, the history narrows, the estimate narrows. Measured, a 30-day watch-history topic entropy goes 3.1 nats to 1.9 over twelve weeks unmitigated.

The fix has to be upstream of ranking. Adding per-source quotas leaves the twelve-week end state at 2.6 instead of 1.9, and adding MMR slate re-ranking on top takes it to 2.9. The quota does most of the work, because a re-ranker can only reorder what retrieval returned — if co-watch supplied 90% of the pool there is nothing to diversify toward.

And I would report entropy of the user’s history, not just of served slates. Slate entropy alone is gameable: you can serve a diverse slate the user never clicks, and the history keeps narrowing anyway.

“Size the fleet and tell me the cost per request.” Testing: arithmetic under pressure. Ranking is 300 candidates at 4.39 MFLOP, so 1.32 GFLOP per request. At 509 k requests/s peak that is 671 TFLOP/s, and at 150 TFLOP/s effective per A100 that is about 4.5 GPUs. The pre-ranker is 2,000 items at 0.05 MFLOP, 51 TFLOP/s, another 0.3.

So the FLOP floor for serving is under 5 GPUs — which I would flag as a floor rather than a bill, since the real fleet is sized by embedding memory, feature fetch and tail latency, and lands one to two orders higher.

Retrieval is memory-bound rather than compute-bound: 800 M items at 128-d fp16 is 205 GB, roughly double with HNSW graph overhead, sharded 16 ways with 3 replicas — about 1.3 TB resident.

For the real fleet rather than the floor: the ANN index is 48 nodes at 3 replicas. The ranker’s per-candidate item features are 1,936 bytes across 800 M items, so 1.55 TB, which is 20 shards times 3 replicas — another 60. About 108 nodes, 22x the FLOP floor.

At that size the ranking nodes run at 7.4% arithmetic utilization, because they were sized to hold bytes. That is why int8 embeddings move the fleet and a cheaper ranker does not.

Two footnotes. The user tower refresh is genuinely free: 500 M forward passes of 0.3 MFLOP is 150 TFLOP total, about a GPU-second. And training data is the surprise — 400 billion impressions a day at ~600 bytes is 240 TB raw, which is why negatives are downsampled 20 to 1. That breaks calibration, which matters because the heads get combined arithmetically, so I correct the logits analytically for the sampling rate rather than re-calibrating empirically.

“What does the ranker actually take as input? You said 1,500 dimensions.” Testing: whether the architecture number came from a system or from a slide. Seven families, summing to 1,500:

• 128 dims of long-term user embedding — the retrieval tower’s own output, reused;
• 72 dims of request context: locale, device, local hour, weekday, connection;
• 332 dims of real-time session state: mean-pooled embeddings of the last 20 watched items, their channels, the last search query, and a few session counters;
• 768 dims of item content: title, thumbnail and an ASR/audio topic embedding at 256 each;
• 128 dims of item categoricals, mostly a hashed channel id;
• 32 dims of item scalars — and those are residualized counts, views − E[views | age, category, channel size], not raw ones;
• 40 dims of cross and provenance features, including which of the six retrieval sources supplied this candidate and its rank within that source.

Two things follow. This concatenation is where the interaction rank the two towers gave up comes back: connection type and duration are in the same trunk here, with no rank-128 ceiling over them.

And 532 of the 1,500 are fetched once per request rather than once per candidate, so widening the user side is free and widening the item side is not. The item families are 1,936 bytes per candidate, 0.58 MB per request, and that is what actually sizes the fleet.

“Walk me through the 100 ms.” Testing: whether the latency budget is a constraint or a slogan. Eight milliseconds for auth and the user-side feature fetch, one for the user tower forward pass, fourteen for six retrieval sources fanned out in parallel — that is the slowest source, not the sum — three for dedupe and the per-channel cap, five for the pre-ranker taking 2,000 to 300, twenty-four for feature hydration on those 300, sixteen for the full ranker, four for the value combine, integrity demotion and MMR. Seventy-five milliseconds of ML path at p50, about 96 at p95 with queueing, plus twenty for payload hydration and serialization: 95 end to end.

I would flag the convention, because it is where these numbers get misquoted. That 100 ms is server-side at the gateway and excludes the client round trip, which is why my ML share is 75% and chapter 10’s is 20%.

The interesting part is that 38 of the 75 milliseconds is fetch and none of it is arithmetic. Even the ranker’s 16 ms is mostly not ranking: the dense math for 300 candidates is 8.8 microseconds, so the wall clock is 1,800 times the arithmetic. The gap is embedding gathers and a batch of 300 that cannot fill an A100.

“You train daily on the last 24 hours. What breaks?” Testing: whether label latency is a phrase or a computed bias. The three heads I argued hardest for. A click label lands in under thirty seconds, but a share takes hours, a subscribe hours to days, and a report days — so a 24-hour window censors 22% of shares, 31% of subscribes and 44% of reports. A censored positive is not a missing row, it is a row labelled negative: 49.6 million subscribes and 10.6 million reports a day training the model to say no.

The uniform part of that is survivable. It deflates the base rate and miscalibrates the head low, and a constant offset does not reorder a slate.

The part that reorders the slate is that censoring is a function of the item. A subscribe after a 90-second clip lands in-session; one after a 25-minute video does not. So censoring runs 9% on short videos and 47% on long ones. At a subscribe weight of 3.0 that is 3.0 × [log(0.91) − log(0.53)] = 1.62 score points handed to short videos, against a deliberate saturating duration term worth 0.4 × [log1p(20) − log1p(1)] = 0.94 across the same range.

The label window is a stronger duration prior than the duration term, and nobody chose it. It is the watch-time trap coming back in through the pipeline after I removed it from the objective.

The fix is per-head maturity windows: hold each head’s rows until its own window closes — T−1 for click, T−3 for share and subscribe, T−7 for report. Not one global 7-day window, which costs the click head — 24 billion positives a day — a week of staleness to protect a head with 13 million. And not dropping the unmatured rows, which is the tempting one and reproduces the bias exactly, because the rows you drop are the long-video rows.

“You said sampled softmax with a logQ correction. Show me it doing something.” Testing: whether the formula was memorized or understood. Take a head item at 6.1e-6 of the training stream and a tail item at 6.1e-9, and a batch size of 8,192. The head item appears as an in-batch negative with probability about 0.05 — one batch in twenty — and the tail item at 5e-5, one batch in twenty thousand. So log Q is −3.00 against −9.90, and at an identical raw dot product of 2.0 the corrected logits are 5.00 and 11.90.

The direction is the part people get backwards. The tail negative is up-weighted because it has to stand in for the thousand items like it that were never sampled; the head negative is down-weighted so the partition function does not count it a thousand times over. It is importance sampling on the denominator, not charity toward unpopular videos.

Skip it and at convergence you learn s(u,i) − log p_i, so the head item carries a 6.9-nat handicap — a factor of 1,000 in softmax probability at an identical dot product — that nobody chose and nobody can tune.

Three mechanical points I would add unprompted:

• the subtraction happens on the logits inside the training softmax and never on the served score, or you re-introduce exactly the popularity boost you removed;
• Q comes from a decayed counter, 3.2 GB for an exact fp32 table over 800 M items, and the half-life is a real parameter, because too long under-corrects a newly viral item;
• logQ does not fix tail recall, which is a different failure — too few positive pairs — with a different control.

“Why not just use an LLM to rank? It would understand the content.” Testing: whether you price ideas before liking them. Six orders of magnitude. Three hundred candidates at roughly 600 tokens of context each is 180,000 tokens of prefill per request. At 8 B parameters that is 2 × 8e9 × 1.8e5 = 2.9 PFLOP per request, against the ranker’s 1.32 GFLOP — 2.2 million times the arithmetic, or $500 M a day against a $230/day FLOP floor. And the task is producing an ordering, which a 2 M-parameter MLP does in 4.39 MFLOP.

Where an LLM does earn its cost is upstream and offline: generating content-understanding features for the item tower — topic, style, claim structure, thumbnail-to-content consistency — computed once per video at upload rather than once per user per request. That is 500 k inferences a day instead of 300 × 20e9 = 6 trillion, and it puts the semantic understanding exactly where the item tower needs it for cold start.

Next: 07 — Event Recommendation.