10 — Personalized News Feed

This chapter designs the ranked feed of a social network end to end: what the score should contain, where each term in it comes from, how a post gets from one author’s write into two billion people’s candidate sets, and which of the resulting failures are bugs versus properties. Three results carry the design: maximizing engagement is a specific and knowable mistake rather than a conservative default; the weights in a multi-objective score cannot be read as written unless you divide by base rates first; and the most important effect of a feed ranker is invisible to the experiment that ships it. Each is backed by arithmetic you can redo.

The one-line shape: a viewer’s identity and session context go in, an ordered list of about 25 posts comes out, drawn from roughly 3,000 candidates that the viewer’s own social graph and a set of recommendation indexes produced.

The standard recipe — retrieve a pool of candidates, rank them, done — is correct and it is not the answer, because three things about a social feed break it. (ml-system-design/06 derives that two-stage recipe in full; the summary you need here is that a cheap model narrows millions of items to a few thousand and an expensive model orders those.)

The content is produced by the people you are ranking for. A video catalog is exogenous — determined outside the system, so it exists whether or not you rank it. Feed inventory is created, in response to what the ranker rewarded last week, by users who can see their own metrics. Your ranking function is an input to next week’s candidate distribution.

The candidate set is a graph query, not a catalog query. There is no global pool of “posts” you retrieve from. There is your pool: your connections, your groups, your follows. It is per-user, it is small, and it expires.

The objective is contested. Nobody disagrees about what a good video recommendation is. People disagree, sincerely and permanently, about what a good feed is, and that disagreement shows up as a term in the score.

Two ideas from earlier chapters are used repeatedly and are restated here so nothing else is required reading. The two-stage retrieval pattern is the one just described: a cheap model cuts a large pool to a manageable one, then an expensive model orders the survivors — ml-system-design/06 derives it, and ml-system-design/01 gives the general framework for taking a system-design question from a one-line prompt to a served architecture. The watch-time trap is the observation that optimizing a system for how long people engage with it selects for content that holds attention rather than content that is worth attention; The watch time trap derives it, and Whose feed is it below shows the feed’s version is strictly worse.

0. The model roster, the assumptions, and the deliberate absences

Before any model is derived, it helps to have the whole roster in view: every model named, what each stage takes on faith, and what this design refuses to model at all.

Nothing here is derived; every row points at the section that derives it. Use it as an index and return to it when a later section names a component you have forgotten.

0.0 The whole system in one paragraph

A viewer opens the app. The system collects a few thousand candidates — posts that are at least eligible to be shown — from four places: posts already delivered into this viewer’s private inbox, posts from very large accounts held in memory, posts recommended by search-style indexes, and posts from groups they belong to. A cheap model cuts those few thousand to 600. An expensive model then scores each of those 600 on eleven separate questions (“will they click? will they comment? will they hide it? would they say it was worth their time?”). A configuration file of weights collapses those eleven numbers into one score per post. A penalty pushes down anything that looks close to a policy violation. A final pass picks 25 posts in order, enforcing rules about variety that no single post’s score can express. That is the whole pipeline, end to end.

Seven words appear throughout and are worth fixing now.

• Candidate — a post that has been retrieved and is eligible to be shown, before anything has scored it.
• Retrieval — the step that produces candidates. Cheap, high-volume, allowed to be sloppy.
• Ranker — a model that scores candidates so they can be sorted. The “light” ranker is cheap and runs on thousands; the “heavy” ranker is expensive and runs on hundreds.
• Head — one output of a model that has several outputs. A model with eleven heads makes eleven predictions from one shared computation.
• Embedding — a fixed-length list of numbers standing in for something that is not numeric (a post, an author, a topic), learned so that similar things land near each other.
• Impression — one post shown to one person once. The atomic unit of everything logged in this chapter.
• Lift — a probability expressed as a multiple of its own average. A post whose comment probability is three times the typical post’s has a comment lift of 3. Combining where the value judgement lives shows why the whole score is built out of lifts rather than raw probabilities.

0.1 Every model in the system

Of the eleven rows below, eight are learned models, one is a two-parameter statistical fit, and two are fixed rules; saying which is which is part of the design. “Offline” means the work runs on a schedule, before any request; “online” means inside the request, while the viewer waits; “at write” means once per post, when it is created.

Two measurements recur in the fifth column. AUC is the area under the ROC curve, which is the probability that the model gives a randomly chosen positive example a higher score than a randomly chosen negative one — 0.5 is a coin flip and 1.0 is perfect. Calibration means the predicted probabilities are literally true in the long run: among impressions the model calls 3% likely, 3% actually convert.

Most of the rows are learned models; rows 9, 10 and 11 are not, and those three are where this chapter spends most of its argument.

0.2 What each stage assumes, and which assumptions carry weight

An assumption is load-bearing when the design changes shape if it is false, and merely convenient when a wrong value only moves a number.

0.3 What this system deliberately does not model

• It does not model long-term user welfare. There is no per-item label for it and there never will be, so the weight vector is set by people and audited by slow experiments (Combining where the value judgement lives, Property 4). The chapter’s position is that pretending otherwise is the failure, not the honesty.
• It does not model the slate. The score is computed one post at a time, so “three posts from the same author in a row” is inexpressible in it; that is handled by a separate re-selection pass rather than another weight (Diversity the constraint no pointwise score can express).
• It does not model the supply side’s response. Creators adapting to the ranker is detected and measured, never predicted (The dashboard that says ship and the system that got worse).
• It does not estimate per-post engagement rates to decision-grade precision, because The freshness trap your best features are missing when they matter most shows that would cost 4.75 times the platform’s entire daily inventory. The cold-start path is built to work without them.
• There is no reinforcement-learning policy over the session, even though the objective really is sequential; Alternatives considered and rejected gives the reason.
• There is no large language model on the read path. Alternatives considered and rejected prices it at $3.2 M/day and confines such models to offline labelling.
• There is no approximate-nearest-neighbour retrieval for in-network content. The candidate set is ~300 items, and approximating a 300-item scan buys nothing (The inventory is small and that inverts the retrieval problem).

1. Whose feed is it?

What is the system actually being asked to maximize? On measurement, the obvious answer is close to the opposite of the right one, which is why the objective ends up with four blocks.

Three parties have a legitimate claim on the answer and they want different things.

Two terms in that table are worth defining. Dwell is how long a post stays on screen before the viewer scrolls past it. DAU is daily active users, the count of distinct people who open the product on a given day.

The first row is the problem. The thing the viewer wants has no logged event. There is no satisfied field on an impression. So the objective gets written in terms of what does have an event, and every failure in this chapter descends from that substitution.

The watch time trap derives why watch time is a hackable target. The feed’s version is strictly worse for one structural reason: video supply is a catalog that changes slowly, and feed supply is a population that adapts in days. A watch-time-maximizing video ranker surfaces the worst items in a fixed catalog. An engagement-maximizing feed ranker changes what gets written. Those are different severities of the same bug.

The proxy gap, measured

“Engagement is a bad proxy for value” is a claim that needs a table behind it.

Take a week of impressions, bucket by content type, and put the engagement rate next to a survey question asked of a sampled subset — “was this post worth your time?”, scored top-2-box of a 5-point scale, meaning the share of answers landing in the top two of the five options.

The third column is what the platform can log for free; the fourth is what it had to ask for. The bottom two rows are where the disagreement is sharpest.

These eight types are 75.5% of inventory; the rest is a long tail of small types. Every inventory-weighted mean below is taken over these eight rows, renormalized — the weighted any-interaction rate is 0.077 and the weighted satisfaction rate is 44%, and those two numbers get used repeatedly.

Rank-order by engagement and rank-order by satisfaction are nearly reversed at both ends. Put a number on that, since “nearly reversed” invites pushback.

The number is the Spearman correlation: the ordinary correlation computed on the two rankings rather than the raw values, so it measures only whether the orders agree. Rank the eight rows by engagement (1 = highest) and again by satisfaction, take each row’s difference d between its two ranks, and square it:

engagement ranks   5  3  7  8  6  2  1  4       (in table order, top row first)
satisfaction ranks 1  2  4  3  5  7  8  6
d                  4  1  3  5  1 -5 -7 -2
d²                16  1  9 25  1 25 49  4       sum d² = 130

rho  =  1  -  6 · sum d² / ( n(n² - 1) )        n = 8, so n(n²-1) = 8 · 63 = 504
     =  1  -  6 · 130 / 504
     =  1  -  780 / 504
     =  1  -  1.548
     =  -0.55

Zero would mean the two orderings are unrelated; +1 would mean they agree exactly. -0.55 means they substantially disagree — the more engaging half of the table is, on average, the less satisfying half.

The two highest-engagement categories are the two lowest-satisfaction categories. They are only 8% of inventory today because the current ranker holds them down. Remove the holds and they grow, because they are cheap to produce and they win the competition for slots.

That is the whole argument for the design in Features: if the score contains only engagement terms, the optimum of the score is the bottom two rows of that table. Not as a slippery-slope worry — as arithmetic on the numbers you already have.

What the objective has to contain

The score has four blocks, each of which exists because the ones above it are not enough. Nothing here is final — Combining where the value judgement lives rewrites this expression once and rewrites it correctly; the point for now is the shape.

Notation, all of it: Value(u, p) is “the value of showing post p to user u”. sum_k means “add up over every k”. w_k is the weight on the k-th kind of positive reaction, c_j is the cost of the j-th kind of negative one, and d(p) is a demotion factor between 0 and 1 derived in The integrity interaction from how likely the post is to break the platform’s rules.

Value(u, p)  =  sum_k  w_k · P_k(engagement type k | u, p)      the actions we can log
              + w_s   · S(would you want to see this | u, p)     the thing we had to ask for
              - sum_j c_j · N_j(negative action j | u, p)        hide, report, unfollow
              - d(p)  · Value_positive                           integrity demotion, §7

Line by line: line 1 is everything the viewer’s own behaviour labels for free. Line 2 is the one signal nobody’s behaviour produces, so it has to be elicited by survey. Line 3 subtracts the reactions that mean “do not show me this”. Line 4 scales the whole thing down when the post looks close to a policy violation, where Value_positive is the total of the lines above it.

The second line is what distinguishes a strong answer. Everyone writes the first. Writing the second and defending its data cost (The survey head and why it is worth six orders of magnitude of data cost) is the harder and more important part.

2. Candidate generation: a graph query with an expiry date

Where do the candidate posts come from? For a feed, the answer inverts the usual retrieval design twice over: the pool you retrieve from is too small rather than too large, and it turns over so fast that the standard nightly index build throws away most of the day’s value.

2.1 The inventory is small, and that inverts the retrieval problem

Count how many posts a viewer is actually eligible to see and the answer comes out in the hundreds — which removes approximate retrieval from the in-network path and replaces the retrieval problem with a different one.

Chapter 06 retrieves 10^3 candidates from a 10^9-item catalog. Do the same arithmetic for a feed. In-network below means posts from accounts and groups the viewer has an explicit relationship with; out-of-network means everything else, recommended rather than subscribed to.

The x 3 in the block below is the 72-hour window expressed in days: a post is useful for three days, so three days of posting accumulates.

median user:  200 follows/friends
              each posts   0.20 posts/day
              45 groups,   1.4 posts/day each that pass group-level filters
              useful window: 72 h  =  3 days

in-network inventory  =  200 x 0.20 x 3   +   45 x 1.4 x 3
                      =  120             +   189
                      =  309 posts

Three hundred posts, not a billion. You do not need approximate retrieval over your own network; you can score every eligible post exhaustively.

Check the other end of the distribution before believing that. The p95 user — the one at the 95th percentile of connectivity, more connected than 95% of the population — has 900 follows and 200 groups:

p95 user  =  900 x 0.20 x 3   +   200 x 1.4 x 3
          =  540             +   840
          =  1,380 posts

Still a full scan, and still two orders of magnitude below anything that would justify an index. So the entire ANN apparatus that dominates a video-recommendation design — approximate nearest neighbour, the family of structures that find probably-closest vectors without checking them all — is unnecessary for the in-network half of a feed.

The problem is at the other tail, where a full scan of the whole graph does not fill one screen:

new user, 12 follows, 2 groups:
   12 x 0.20 x 3  +  2 x 1.4 x 3  =  7.2 + 8.4  =  16 posts
session demand: 25 impressions

A new user’s entire social graph produces two-thirds of one session. So the retrieval problem is not “narrow 10^9 to 10^3.” It is fill: for most of the population most of the time, in-network inventory is insufficient and the deficit must come from somewhere else.

The table below splits the population three ways and reads left to right as “how much of this segment’s feed cannot come from their own graph”. The last column is the deficit the recommendation side has to cover.

The mixing ratio is per-user and it is the single most consequential dial in the system, because out-of-network content has a different quality distribution, a different integrity risk profile, and a completely different retrieval architecture. Two sources, one merged list:

• In-network: an exhaustive scan of a per-user inbox — a list of post identifiers maintained for each viewer. No approximation, and therefore no recall loss.
• Out-of-network: approximate-nearest-neighbour retrieval over a global embedding index, plus topic and entity inverted indexes (the search-engine structure that maps each term to the list of posts containing it), plus a small set of hand-tuned sources such as what is trending in your city or new posts in groups adjacent to yours.

2.2 The index turns over 33% per day, and that kills nightly rebuilds

The pool is not just small — it is being replaced at a rate no catalog system prepares you for, fast enough that the standard practice of rebuilding a search index overnight discards nearly half of the value the index exists to deliver.

posts created                 500 M / day
useful window                 72 h
live inventory                500 M x 3   =  1.5 B posts
daily turnover                500 M / 1.5 B  =  33 % / day

Compare a video catalog at ~1%/day. A feed’s searchable pool is replaced 33 times faster than a video catalog’s.

Now price the staleness of a nightly rebuild. The standard practice is to build the approximate-nearest-neighbour index once overnight and serve from that snapshot all day. Snapshot at 04:00, serve until the next one, and ask how much of the day’s content the index has never seen by evening. Posts arrive at 500 M / 24 h = 20.8 M/h:

posts created since snapshot, by 20:00:  16 h x 20.8 M/h  =  333 M   (22 % of live inventory)

Why 22% understates it

That 22% is a count of posts. What you actually lose is engagement, and the missing posts are the newest ones — which, by the decay curve in Recency derived rather than assumed, is where almost all the engagement is.

Two definitions before the arithmetic. The hazard lambda is the rate at which a post’s chance of being interacted with decays per hour: an interaction rate of exp(-lambda t) falls by the same fraction every hour that passes. And if a post’s interaction rate at age t is exp(-lambda t), then the total engagement it collects between two ages is the area under that curve between them — which is what an integral computes.

So: total engagement in the first 16 hours of life, divided by total engagement over the whole 72-hour window. With the fitted lambda = 0.029 /h from Fit the decay do not pick it, and using integral_0^T exp(-lambda t) dt = (1 - e^(-lambda·T)) / lambda:

lambda · 16  =  0.029 x 16  =  0.464
lambda · 72  =  0.029 x 72  =  2.088

integral_0^16  exp(-0.029 t) dt  =  (1 - e^-0.464) / 0.029  =  (1 - 0.6287) / 0.029  =  12.80
integral_0^72  exp(-0.029 t) dt  =  (1 - e^-2.088) / 0.029  =  (1 - 0.1239) / 0.029  =  30.21

share  =  12.80 / 30.21  =  42.4 %

A nightly index rebuild makes 42% of the day’s available engagement invisible to out-of-network retrieval, even though it is only missing 22% of the posts.

What that forces

The index has to accept incremental inserts at sub-minute lag. That means an HNSW graph — hierarchical navigable small world, the standard approximate-nearest-neighbour structure, a layered graph you walk downhill toward the query — that is mutated in place rather than built once.

Mutation is not free. It brings tombstone accounting, where a deleted item is marked dead rather than physically removed because unpicking it from the graph is expensive, and it brings the gradual recall loss that accumulates as those dead entries pile up (Hnsw memory per vector derived). That cost is not optional; it is what the 33% turnover buys you out of.

3. Recency, derived rather than assumed

Most feed designs hand-pick a recency penalty. Measure it instead and two surprises fall out: a single global number is wrong by a factor of 46 across content types, and once the right one is derived it should mostly not be applied as a multiplier at all. Behind both sits a harder problem — a feed’s best features are missing exactly when they matter most.

3.1 Fit the decay, do not pick it

The decay of engagement with post age can be fitted from data — and the answer turns out to be not one number but one per content type.

Start with the confound, because the naive version of this measurement is wrong. If you just plot interaction rate against post age across all posts, you measure two things at once: old posts really do get less engagement, and the ranker shows good posts sooner, so old posts are disproportionately the ones the ranker did not like.

Separate them by holding predicted quality fixed. Bucket the impressions on the ranker’s own score decile — a tenth of the sorted population, so every post in one bucket was judged roughly equally good — and measure age against interaction rate inside one bucket. Then any remaining slope is age, because quality is constant by construction.

The middle column is the mid-point of each age band, which is what gets used as the t in the fit.

age (h)     mid   any-interaction rate
 0 - 1      0.5          0.062
 1 - 3      2.0          0.054
 3 - 6      4.5          0.041
 6 - 12     9.0          0.033
12 - 24    18.0          0.026
24 - 48    36.0          0.017
48 - 72    60.0          0.011

Now fit an exponential through the two endpoints. If the rate follows rate(t) = rate_0 · exp(-lambda t), then the ratio of two rates is exp(-lambda · Δt), so taking a logarithm turns the fit into one division:

first point   t = 0.5 h,  rate 0.062
last point    t = 60.0 h, rate 0.011
time span     60.0 - 0.5  =  59.5 h

ln(0.062 / 0.011)  =  ln 5.636  =  1.729     the log of the fall over that span
lambda             =  1.729 / 59.5  =  0.0291 / h
half-life          =  ln 2 / lambda  =  0.693 / 0.0291  =  23.8 h

That gives a half-life of about 24 hours — the time it takes engagement to fall by half. (The half-life is ln 2 / lambda because exp(-lambda · t) = 0.5 exactly when lambda · t = ln 2.)

That single number is wrong for most of your inventory, and knowing why is the point of the section. Refit it per content type. Each row below is the same two-point fit, run on that type’s own measurements: a rate at 1 hour, a rate span hours later, and the lambda and half-life those two points imply.

Work one row so the table is checkable. Breaking news: ln(0.090 / 0.012) = ln 7.5 = 2.015, over a 12-hour span, so lambda = 2.015 / 12 = 0.168 /h and the half-life is 0.693 / 0.168 = 4.1 h.

Breaking news decays 70 times faster than an evergreen how-to. One global half-life cannot serve both.

3.2 The exchange rate, which is what you actually argue about

A fitted decay rate is still not a quantity a design review can hold an opinion about. What is, is the exchange rate it implies: how much better an old post has to be to beat a new one.

If the final score is quality · exp(-lambda · t), then an older post at quality q_old ties a fresh post at q_new exactly when

q_old · exp(-lambda t)  =  q_new
q_old / q_new           =  exp(lambda t)

That ratio is the freshness/quality exchange rate, and it is the only honest way to discuss “is the feed too recency-biased.” lambda is an abstraction nobody has an intuition about. “This post has to be twice as good to keep its slot” is a sentence a product manager can disagree with.

Substitute t = 24 h into exp(lambda · t) for three of the fitted lambdas. The middle step is the multiplication most write-ups skip:

global lambda = 0.0291:      0.0291 x 24 = 0.698   ->  exp(0.698) =  2.01 x better
news   lambda = 0.168:       0.168  x 24 = 4.03    ->  exp(4.03)  = 56.2 x better
life-event lambda = 0.0087:  0.0087 x 24 = 0.209   ->  exp(0.209) =  1.23 x better

Read those as sentences. Under the global rate, a day-old post has to be twice as good as a fresh one to hold its slot. Under the news rate, 56 times as good. Under the life-event rate, 23% better is enough.

One global half-life applied to both ends of that table is off by a factor of 46 — 56.2 / 1.23 = 45.7. In practice it buries a friend’s engagement announcement from yesterday under a fresh aggregator meme, and it keeps yesterday’s breaking news alive past the point of being wrong. Both are complaints real users file, and both are the same modelling error.

The code below runs every fit in Fit the decay do not pick it and every exchange rate above, and asserts the numbers rather than asking you to trust the table.

import math


def fit_lambda(rate_early, t_early, rate_late, t_late):
    """Two-point exponential fit for the engagement hazard, per content type.

    Returns (lambda_per_hour, half_life_hours). Fit this on impressions that
    have been bucketed by the ranker's own score decile, otherwise the curve
    is confounded by the fact that better posts are also seen sooner.
    """
    lam = math.log(rate_early / rate_late) / (t_late - t_early)
    return lam, math.log(2.0) / lam


def freshness_exchange_rate(lam, age_hours):
    """How much better an `age_hours`-old post must be to tie a fresh one.

    This is the number to argue about in a design review -- not lambda, and
    not the half-life. At lam=0.0291 a one-day-old post needs 2.0x; at the
    breaking-news lam=0.168 it needs 56.2x. Same decay family, 46x apart.
    """
    return math.exp(lam * age_hours)


# --- 3.1's per-type fits and 3.2's exchange rates, executed --------------
FITS = {                       # name: (rate at 1 h, rate after span, span h)
    "breaking news": (0.090, 0.012, 12),
    "topical":       (0.058, 0.014, 48),
    "photo, friend": (0.101, 0.041, 48),
    "life event":    (0.140, 0.075, 72),
    "evergreen":     (0.031, 0.026, 72),
}
for _name, (_r0, _r1, _span) in FITS.items():
    _lam, _hl = fit_lambda(_r0, 1.0, _r1, 1.0 + _span)
    print("%-14s lambda %.4f /h   half-life %6.1f h   a 24 h-old post must be %6.2fx better"
          % (_name, _lam, _hl, freshness_exchange_rate(_lam, 24)))

_lam_news, _hl_news = fit_lambda(0.090, 1.0, 0.012, 13.0)
_lam_ever, _hl_ever = fit_lambda(0.031, 1.0, 0.026, 73.0)
assert abs(_lam_news - 0.168) < 5e-4 and abs(_hl_news - 4.1) < 0.05
assert abs(_lam_ever - 0.0024) < 5e-5 and abs(_hl_ever - 284) < 1.0

_lam_g, _hl_g = fit_lambda(0.062, 0.5, 0.011, 60.0)      # the global two-point fit
print("global fit:    lambda %.4f /h   half-life %6.1f h   24 h exchange rate %.2fx"
      % (_lam_g, _hl_g, freshness_exchange_rate(_lam_g, 24)))
assert abs(_lam_g - 0.0291) < 5e-5 and abs(_hl_g - 23.8) < 0.1
assert abs(freshness_exchange_rate(_lam_g, 24) - 2.01) < 5e-3
assert abs(freshness_exchange_rate(0.168, 24) - 56.3) < 0.2
assert abs(freshness_exchange_rate(0.0087, 24) - 1.23) < 5e-3
# the 46x is a spread in the EXCHANGE RATE, which is what one global lambda hides
print("exchange-rate spread, news vs life event: %.1fx"
      % (freshness_exchange_rate(0.168, 24) / freshness_exchange_rate(0.0087, 24)))
assert abs(freshness_exchange_rate(0.168, 24)
           / freshness_exchange_rate(0.0087, 24) - 45.7) < 0.5

3.3 Do not multiply the decay onto the score

Having gone to the trouble of measuring the decay, mostly do not apply it as a multiplier — the right place for it is in the model’s inputs, not on top of the model’s output. Feed age_seconds, log(age), and age_bucket x content_type to the ranker as features and let it learn the interaction, because:

1. A hand-applied multiplier double-counts — the model already saw age and already discounted.
2. The right lambda is a function of content type, author, viewer, and time of day, which is exactly the sort of high-order interaction a model fits and a human does not.
3. A multiplier is unconstrained at the tail: exp(-0.168 · 72) = 5.6e-6 zeroes an entire content class.

Keep an explicit decay for exactly one job: breaking ties in the retrieval stage, where you have no model score yet and you have to cut a heavy user’s ~3,000 in-network posts plus everything three out-of-network indexes returned down to the ~3,000 candidates the light ranker will score (Architecture). There the multiplier is cheap, monotone, and its errors are recoverable downstream.

3.4 The freshness trap: your best features are missing when they matter most

At the young end of the decay curve sits a structural bind: the model’s strongest inputs are unavailable precisely when the post is most valuable, and the obvious fix turns out to be not merely expensive but arithmetically impossible.

A five-minute-old post has no engagement counts. Every count-derived feature — post CTR (click-through rate, the share of impressions that got clicked), early like velocity, comment-to-impression ratio, reshare depth — is null, meaning simply absent, exactly in the window where Fit the decay do not pick it says the post is worth the most.

The obvious answer is “explore: show it to some people and measure.” Price it before believing it.

se below is the standard error: the typical size of the gap between an estimate made from n samples and the truth. For a rate p estimated from n impressions, se = sqrt(p(1-p)/n), which rearranges to n = p(1-p)/se² — the number of impressions you need to buy a given precision. Set se = 0.02 around a typical interaction rate of p = 0.05, so se² = 0.0004:

target: estimate a post's interaction rate to +/- 0.02 absolute, around p = 0.05
n  =  p(1-p) / se^2  =  0.05 x 0.95 / 0.0004  =  0.0475 / 0.0004  =  119 impressions

500 M posts/day x 119  =  59.5 B exploration impressions/day
total feed impressions  =  200 B/day
                                       ------
                          59.5 / 200  =  30 % of ALL inventory

Thirty percent of everything the platform can show, spent on measurement rather than on serving. And it does not even buy a usable number: +/- 0.02 around p = 0.05 is a 40% relative error, which cannot separate a 0.05 post from a 0.06 post — exactly the distinction a ranking decision turns on.

So ask for four times the precision, +/- 0.005. Precision costs quadratically, so se shrinking 4x makes n grow 16x:

se = 0.005,  se^2 = 0.000025
n  =  0.0475 / 0.000025  =  1,900 impressions
500 M x 1,900  =  950 B impressions  =  950 / 200  =  4.75 x the entire daily inventory

Measuring per-post engagement rates to decision-grade precision is arithmetically impossible, by a factor of five, before you have spent a single impression on anything else. So the design is forced:

• The cold-start path uses only author-level, content-level, and viewer-author features — no post-level counts. Train it as a separate head or with count features masked, so the model does not learn to depend on something that will be absent.

• Exploration is a budget, not a policy: a fixed 3-6% of slots, spent on the posts where the exploration is decision-relevant rather than on all of them. The allocation rule is Thompson sampling — keep a probability distribution over each author’s true quality and pick each option in proportion to how likely it is to be the best, so uncertain options get tried without being forced (ml 08).

  The general asymmetry — exploration’s cost is visible and its benefit is not — is Exploration is the way in and its cost is visible while its benefit is not. The feed-specific part is the impossibility just derived: because no affordable budget reaches decision-grade precision, the budget can never be “enough”, so it has to be spent by an explicit rule rather than sized to a target precision.

• Count features enter with an explicit maturity gate. Rather than showing the model a raw rate k/n computed from a handful of impressions, show it a shrunk estimate (k + alpha·prior) / (n + alpha). When n is small the alpha·prior term dominates and the estimate sits at the prior; as n grows the observed data takes over and the estimate slides toward k/n. The model therefore sees a smooth transition instead of a jump from nothing to a value (ml-system-design/01).

4. Features

What does the ranker get to look at? The features group into families by where each comes from — and in two of those families, getting the engineering wrong causes architectural bugs rather than accuracy losses.

The seven families

Seven families, and the column to read first is the last one. Every family carries a characteristic failure, and three of those failures get whole sections later. “Signal” is how much predictive power the family carries; “cost” is what it takes to compute at serving time.

Two things in that table cause architectural bugs rather than accuracy losses, so they get their own subsections.

Where the content encoders run

Content embeddings are computed at write time, never at read time. The arithmetic is one line each way.

At write: 500 M posts a day, spread over 86,400 seconds, is a steady 500e6 / 86400 = 5,787 posts/s. At a few hundred milliseconds of encoder per post, that is an ordinary batch job.

At read: the feed serves 200 B impressions a day, so running the same encoders on the read path is 200 B encoder calls — a factor of 400 more work, and all of it inside a request the viewer is waiting on.

This is the most common architectural error in a feed design: putting a multimodal encoder behind a 400 ms feed request. Encode once when the post is created, store the vector on the post, and read it back.

Position bias, and the two fixes

Position bias is the fact that an item shown higher on the page gets clicked more regardless of quality. A model trained naively on logged clicks therefore learns “was shown first” rather than “is good”, and the bias compounds because the model’s own output decides tomorrow’s positions.

Two standard fixes, both derived in Position bias and the feedback loop and not re-derived here:

• A bias tower: a small side-network fed only the position. The main network cannot use position, so it is forced to explain everything the bias tower cannot.
• Inverse propensity weighting (IPW): weight each logged event by one over its propensity — the probability that the logging system would have shown that item in that slot. Events the old policy was unlikely to produce count for more, which un-does the old policy’s preferences.

Two twists are specific to feeds.

The position curve is steeper than a search results page’s, because a feed is an infinite scroll and most sessions end before position 20:

position         1     3     5    10    15    20
relative rate  1.00  0.71  0.58  0.38  0.28  0.22

By position 10 a post gets 38% of the clicks it would get at position 1. But that number mixes two effects — people look less carefully further down, and many sessions simply ended before reaching slot 10. “Position” and “session survived long enough to see it” are entangled, and separating them is feed-specific work.

The propensities exist only if the ranker recorded them. So: log the candidate set and the score distribution, not just the served slate. Without that log, every off-policy estimate in Offline — any attempt to estimate how a different ranker would have performed, using data the current one produced — is unavailable after the fact, and no amount of later work recovers it.

One row, populated

The seven families above are a schema — a list of what kinds of values exist. Here is one actual row filled in, so you can see what a single (viewer, post) pair looks like to the model.

The post is the engagement bait that Engagement bait amplification scores at Value = 0.447, seen by a median viewer 3.2 hours after it was written. The seven family names run down the left; the values are the features.

Every count-derived line in the post counts family is the numerator of a head probability in Engagement bait amplification’s table. That is the point of showing the input next to the output — the eleven probabilities are not free-standing numbers, they are this row divided through by its own impression count.

viewer x author edge   interactions_7d 0          interactions_30d 1
                       interactions_90d 3         profile_visits_90d 0
                       messages_ever 0            tie_strength 0.04
                       reciprocity 0.00           (author is an OON recommendation)

viewer                 topic_affinity[trivia] 0.11   topic_affinity[news] 0.38
                       session_depth 12           device mobile
                       connectivity 4g            local_hour 22
                       long_term_topic_entropy 2.9 nats

author                 hist_interaction_rate 0.121   follower_count 84,000
                       posts_per_day 6.2          integrity_history 2 demotions / 90 d
                       account_age_days 410

content                text_emb[256] (at write)   lang en        has_media false
                       topic trivia_prompt        entities none  link_domain none
                       solicitation_terms 3       ("LIKE", "COMMENT", "SHARE")
                       p_violating 0.04           bait_score 0.91

post counts            age_hours 3.2              impressions 41,900
                       likes 8,003                comments 3,687   reshares 1,299
                       hides 377                  early_ctr 0.061
                       maturity gate  n = 41,900  ->  shrinkage weight 0.96

context                position_slot 7            surface main_feed
                       session_depth 12           hours_since_last_session 5.1
                       logged propensity 0.031    (for the IPW in section 10.1)

group / source         source OON_topic_index     group_quality n/a

Three things this row shows that the family table cannot. The strongest family is empty: viewer x author edge is “highest signal” and every one of its values here is zero or near it, because this is an out-of-network recommendation — which is exactly the 30-50% of a median feed The inventory is small and that inverts the retrieval problem says has to come from somewhere other than the graph. The post is mature, at 41,900 impressions, so the count features are present and the maturity gate is nearly fully open at 0.96; the same row at age_hours = 0.1 has that whole family null and is the The freshness trap your best features are missing when they matter most problem. And the propensity is logged, 0.031, because without it Offline’s off-policy estimate of this decision is unavailable after the fact.

5. Multi-task ranking

This is the centre of the design. The ranker is one network predicting eleven different reactions, and the harder half of the problem is not building it but turning eleven probabilities into one number. That combining step is where the product’s value judgement is written down, and where most feed designs go wrong.

5.1 The heads

Start with what the model predicts — eleven things — and why one network predicts all of them rather than eleven networks predicting one each.

The architecture is a shared bottom — one network body that all tasks use, so the expensive representation is computed once — with a small per-task head or tower on top of it, one per prediction.

Eleven heads in four groups is typical. The grouping is the design: cheap positives are easy to get and easy to fake, costly positives take more intent from the viewer, negatives are the viewer saying “stop”, and the last group has exactly one member because there is exactly one signal nobody’s behaviour produces.

Count the table: 3 + 3 + 4 + 1 = 11, and those eleven names are the eleven rows of Combining where the value judgement lives’s base-rate table, the eleven keys of the BASE dictionary in Demotion is how you spend a classifier that is too weak to remove with, and the eleven probabilities Engagement bait amplification scores a real post on. A head that is not in all four places is not a head; it is a metric somebody wanted. Profile visit and link click-through are the obvious candidates for a twelfth and a thirteenth, and they are deliberately absent: neither has a measured base rate here, so neither can be given a share weight, and Combining where the value judgement lives is about to show that a head without a base rate is a head whose contribution nobody can read.

Why one model with eleven heads rather than eleven models. The features are identical, the sparse embedding tables — the big lookup tables that turn identifiers like author or topic into vectors — are 95% of the memory, and serving eleven models means eleven lookups of the same rows. One shared bottom means one lookup. That is an engineering argument, and it is sufficient on its own.

There is a modelling argument too, and it is that a plain shared bottom stops being enough once two tasks want to pull the shared layers in opposite directions. The standard resolution is a mixture of experts (MoE): several parallel sub-networks, with a small per-task gate deciding how much of each expert that task uses, so conflicting tasks can quietly stop sharing the parts they disagree about. That derivation is in Shared bottom is not enough mmoe and the gradient argument and is unchanged here. The feed-specific note is which tasks conflict: p(comment) and p(hide) peak on the same divisive content, so they are not merely different, they are anti-correlated on the slice that matters. Report per-head AUC against single-task baselines; if any head is more than ~0.005 below its single-task model, the sharing is costing you.

5.2 Combining: where the value judgement lives

Eleven calibrated probabilities now have to become one number. Four properties of the combining expression govern how, and the second of them decides whether the weight vector you review is the weight vector the ranker obeys.

Start with the obvious expression: multiply each head’s probability by a weight and add them all up, subtracting the negatives. w_ names a positive weight, c_ a cost on a negative head.

Value  =  w_click·p_click + w_like·p_like + w_comment·p_comment + w_reshare·p_reshare
        + w_dwell·p_dwell + w_survey·p_survey
        - c_hide·p_hide - c_seefewer·p_seefewer - c_unfollow·p_unfollow - c_report·p_report

That expression is what almost everyone writes, and it is broken in a way that no amount of tuning the weights repairs. Four properties explain why and fix it; Property 2 is the central one.

Property 1: it is linear, so calibration is not optional

In a pure ranking system, a monotone distortion of a score — any transformation that preserves the order, like squaring a positive number — is harmless, because only the order is used. Nothing downstream reads the score’s value.

Here the heads are added, which destroys that safety. If p_comment is systematically 1.4x overconfident, the effective weight on commenting is 1.4 · w_comment, and nobody wrote that down. In a linear multi-task value model, miscalibration is a silent, undocumented edit to your value judgement.

So every head is calibrated independently, and the reliability diagrams — plots of predicted probability against observed frequency, which lie on the diagonal exactly when a model is calibrated — are a launch gate rather than a diagnostic (Calibration what it means and when it matters).

Then note what calibration does not buy. Calibration makes each head’s number true. It does not make two heads’ numbers comparable. Those are different problems, and the second one is Property 2.

Property 2: the heads live on base rates 3,000x apart, so raw weights are not the judgement they look like

A head’s base rate p̄ (read: “p-bar”) is how often that outcome occurs across all served impressions. They are not close to each other: 44% of rated posts draw a top-2 survey answer, while 0.015% of impressions draw a report. That is a factor of 0.44 / 0.00015 = 2,933, call it 3,000.

Here is the diagnostic that exposes the problem, and it is one multiplication. For a typical post — one sitting at every head’s base rate — head k contributes w_k · p̄_k to the score. So write each head’s base rate next to its weight, multiply, and read the resulting column.

Worked, for the top row: the survey head has base rate 0.44 and weight 0.70, so it contributes 0.70 x 0.44 = 0.308. The whole positive block sums to 0.336, so the survey head alone is 0.308 / 0.336 = 91.7% of it.

Three things fall out of that table, and each of them contradicts what the weight column appears to say.

One. The survey head is not one term among seven, it is 91.7% of the score. A weight vector that reads like a balanced compromise is, per impression, a survey-only ranker with rounding error attached.

Two. The comment weight is not 3.7x the like weight, it is 0.70x it. Reading the raw column, 0.55 against 0.15 looks like a strong preference for costly interactions — a comment declared 3.7 times as valuable as a like. Read the contribution column instead: commenting moves a typical post’s score by 0.0022 and liking moves it by 0.0032. Likes are 5.25x more common (0.021 / 0.004), which swamps the 3.7x weight advantage. The preference is inverted relative to its author’s intent.

Three. The entire four-head negative block is 5.0% of the positive block. report: 12.0 is the largest number in the config and the third-smallest term in the score — only reshare and unfollow, tied at 0.00160, are smaller — because reports run at 1.5 per 10,000 impressions.

Why calibration cannot fix this

None of this is miscalibration. Every number in the p̄ column is the calibrated truth — being the long-run frequency is exactly what makes it the population base rate.

The defect is elsewhere. A probability of something that happens 44% of the time is being added to a probability of something that happens 0.015% of the time, and nothing in the expression converts between them.

That is a commensurability failure: the terms are not measured in comparable units. Adding a temperature to a distance is not wrong arithmetic, it is meaningless arithmetic, and this is the same category of mistake. It is invisible to every calibration diagnostic you own, and it is the reason the weight vector cannot be reviewed as written — the reviewer reads w and the ranker obeys w · p̄.

The fix: weight lifts, not probabilities

A lift — the prediction divided by its own base rate — puts the base rate in both the numerator and the denominator, so it has no units. That is precisely what makes lifts addable across heads that live on wildly different scales.

Divide each head by its own base rate before weighting:

Value  =  sum_k  w_k · ( p_k / p̄_k )   -   sum_j  c_j · ( n_j / n̄_j )

Follow what that does to the typical post. At a post sitting at every head’s base rate, every ratio p_k / p̄_k equals 1, so term k contributes exactly w_k. The weight vector now is the vector of value shares. It sums to the score of an average post; each entry is the fraction of that score the head carries; and a reviewer who reads the config is reading the value judgement directly.

Normalize the positive weights to sum to 1.000 and the negative ones to 0.400. The base-rate post therefore scores 1.000 - 0.400 = 0.600, which becomes the reference every other score in this chapter is quoted against.

The right-hand column below is the same judgement translated back into the old raw-probability units, computed as w / p̄. It is there to make one point, stated under the table.

The right-hand column is what you would have had to write by hand to express that judgement in the original form, and nobody writes report: -1600. The base rates were always in the weights. They were just never written down, so they were never reviewed.

Property 3: a share constraint holds at the mean and says nothing about the tail

Normalizing fixes what an average post’s score is made of. But ranking is not decided by average posts — it is decided by the extremes, and a share constraint says nothing about those.

Work the extreme case. Engagement bait amplification’s engagement bait draws comments at 22 times the base rate, and its survey score is a quarter of the base rate. Two terms, two multiplications:

comment  share 0.150,  lift 22.0   ->  0.150 x 22.0  =  3.300
survey   share 0.350,  lift  0.25  ->  0.350 x 0.25  =  0.0875

The comment term alone contributes 3.30 to a score whose average is 0.600, and the survey head — nominally the largest share in the vector — has 0.0875 to argue with. It loses by a factor of 38.

So normalization alone does not demote the bait at all. It moves it from slot 1 to slot 1 (Engagement bait amplification recomputes the whole score: 8.9x the average post, still first).

The fix is to cap how far any one lift can run. Each lift is clipped:

lift_k(p)  =  min( p / p̄_k , L_k )        L_k  =  p99(p_k) / p̄_k

L_k is measured, not chosen: it is the head’s own 99th-percentile prediction divided by its base rate, which is the point where the head’s outputs stop resembling the data it was fitted on. It lands between 2.8 and 3.4 across the eleven heads, so a single L = 3 is used below for exposition. Above its own 99th percentile a head’s output is far more likely to be solicitation or a calibration tail than genuine quality, which is the same claim Engagement bait amplification’s third defense makes conditionally and the clip makes unconditionally. Clipping is what makes the share vector mean something out in the tail where ranking happens, and it leaves Property 1 intact: below the cap, a head that is 1.4x overconfident still edits its own weight by 1.4.

Property 4: the weights cannot be learned, and the reason is a missing label

To fit w you would need a per-item label for the thing Value is supposed to approximate — long-term user welfare. There is no such label and there never will be. Nobody can annotate an impression with “this made the viewer’s life better”.

So w is set by humans and checked by long-running experiments against retention and survey aggregates. That is a very low-bandwidth channel, and the bandwidth is worth counting.

A holdout is a group of users deliberately kept on an unchanged system for a long period, so the accumulated effect of everything shipped to everyone else can be measured against them. One holdout takes 8-12 weeks to produce a trustworthy read, so call it 5 read-outs per year per holdout. You can run about four holdouts at once before the groups start contaminating each other and before you run out of people to staff them:

long-term holdout read-out                    8 - 12 weeks  ->  ~5 read-outs/year each
independent holdouts runnable in parallel     ~4  (before contamination and headcount bite)
                                              ------
evaluations of w per year                     4 x 5  =  20, optimistically

Twenty evaluations a year is the entire optimization budget for w.

Put that against the size of the vector. A coordinate sweep is one pass in which you move a single weight at a time and measure the result, so a sweep of n weights costs n evaluations. With 10 weights, 20 evaluations buys two sweeps. With 40 weights you cannot complete even one — which means the extra 30 weights are being set by opinion and defended by the fact that nobody can measure them.

The dimension of the value vector is bounded by your experiment throughput, not by your modelling ability. So keep it at 8-12 terms, keep it in a config file, keep it reviewed by name, and treat every proposed new term as a claim on a scarce quarterly resource.

So how are the weights actually set?

Not by fitting, and not by argument alone. They are set by a five-step procedure: four steps ask a human for a judgement they can actually hold, and one step spends the scarce experiment budget from Property 4.

1. Elicit the shares, not the weights. The question put to the people who own the objective is “of the value a post can add, what fraction should come from the fact that a person told us it was worth their time?” The answer is a number between 0 and 1 and it is w_survey. Repeat for each head. This is a question about product intent; the raw-weight version of the same question (should w_survey be 0.7 or 0.8?) is unanswerable because nobody can hold p̄ in their head.
2. Audit the shares by converting them back to exchange rates. w_comment / w_like in raw units is 37.5 / 3.33 = 11.25: the shares assert that one comment is worth 11.25 likes, and one reshare 19.5. If the room thinks a comment is worth about three likes, the shares are wrong and the exchange rate is where that shows up. The share is the input and the exchange rate is the check, because people have reliable intuitions about the second and none about the first.
3. Set the negative block as a fraction of the positive block, in one number. Here 0.400: an average post loses 40% of what it earns. Then split it by the same exchange-rate audit — a report costs 8x a hide, an unfollow ~3x. The consequence is worth stating because it is the intended shape: a post at the lift cap on report alone loses 0.240 x 3 = 0.72, which is 72% of the entire positive block. The negative heads are dormant at the base rate and dominant in the tail, which is exactly what you want and exactly what report: 12.0 failed to deliver.
4. Move at most two coordinates per read-out, against retention and the survey aggregate, using the ~20 read-outs a year from Property 4. Everything you did not move is set by argument; the config should say so, per line, with a date. A weight nobody has ever moved in an experiment is an opinion with a number next to it, and labelling it honestly is cheaper than defending it later.
5. Treat re-measuring the base rates as a weight change. p̄ drifts — a successful integrity launch cuts p̄_report, a UI change moves p̄_dwell10. Re-baselining the denominators silently re-scales every raw weight, so the base rates are frozen for the life of an experiment, recomputed on a stated cadence, and the re-baseline goes through the same review as an edit to w. This is the failure mode that only appears in the normalized form, and it is worth it: in the raw form the same drift was happening continuously and nobody could see it.

Every term in the vector has a degenerate optimum

The reason the score is a blend is not diplomacy. Each single term, maximized alone, has a well-defined argmax — the input that maximizes it — and every one of those argmaxes is a broken product. That is the mechanical argument for multi-objective, and it is the one to give when someone proposes dropping a term to simplify.

The last row is the one worth stating explicitly. It is why the negative terms are a block subtracted from a positive block rather than independent filters, and it is the formal reason guardrail metrics are handled the way What to do about it given that handles them — as independent blockers on a launch, never as terms you can trade a good engagement number against. The blend is not a compromise imposed on the model. It is a set of mutual constraints among terms whose individual optima are each unshippable.

5.3 The survey head, and why it is worth six orders of magnitude of data cost

One head’s labels are not free, and its budget has to survive a cost review. Two arguments carry it — one about how many labels a small tower actually needs, and one about the value of measuring the right quantity badly versus the wrong quantity perfectly.

The data cost, stated honestly.

engagement labels   200 B impressions/day, every one labeled by the user's own behavior
                    ~2 x 10^11 labels/day, marginal cost 0

survey labels       sample 1 session in 20,000
                    1 in 20,000 x 8 B sessions/day        =  400,000 surveys/day offered
                    response rate 30 %                    =  120,000 completed
                    3 posts rated per survey              =  360,000 item-level labels/day

ratio               2 x 10^11 / 3.6 x 10^5  =  5.6 x 10^5

Roughly six orders of magnitude fewer labels, and they cost real user attention rather than nothing. The reflex response is that a head trained on 0.0002% of the data cannot be worth its slot. Both halves of that reflex are wrong.

Argument 1: the volume is sufficient, because it is a head and not a model

The survey head does not have to learn what a post is. That job was already done by the shared bottom, which 2 x 10^11 engagement labels paid for. The survey head only has to learn a readout on top of a representation it gets for free.

Count its parameters. It is a two-layer tower over the 256-number shared representation, so 256 inputs into 128 hidden units plus 128 bias terms:

parameters  =  256 x 128  +  128  =  32,768 + 128  =  32,896

A common rule of thumb is 10-100 examples per parameter for stable estimation, which puts the requirement at:

10  x 32,896  =    329,000 examples
100 x 32,896  =  3,290,000 examples

And the supply:

360,000 labels/day x 365  =  131,400,000 labels/year  ->  ~130 M

130 M / 3.29 M  =   40x  the high end of the requirement
130 M / 0.329 M =  400x  the low end

Between 40 and 400 times the requirement for a head of this size — and hopelessly short for a model trained from scratch, which would need to learn the representation too. The whole feasibility argument is that the representation is free and only the readout is expensive.

Argument 2: a noisy estimate of the right target beats a precise estimate of the wrong one

This is the argument that decides the question.

Set up. Model engagement and true satisfaction as standardized variables — rescaled to have mean 0 and standard deviation 1, so that correlation is the only property left to talk about. Call them E and S, and let corr(E, S) = rho = 0.35.

Where rho comes from, and where it does not. That 0.35 is measured at the item level, on the survey-labeled impressions themselves. That is the only place it can be measured, and it is emphatically not Whose feed is it’s table, where the rank correlation across content types is -0.55.

Both numbers are real and they point opposite ways. Within a single content type, a more-engaging post is somewhat more satisfying (+0.35). Across content types, the ordering reverses (-0.55). A relationship that holds inside every group and flips when the groups are pooled is Simpson’s paradox, and it is the whole reason the two signals are not interchangeable. It is also why rho has to be computed from the survey sample rather than read off aggregate rows.

Define true value. Suppose the thing you actually want is a blend — 30% engagement, 70% satisfaction. (That split is a stated judgement, not a measurement; the conclusion below is not very sensitive to it.)

V  =  0.3 E  +  0.7 S

Its variance, using Var(aX + bY) = a²Var(X) + b²Var(Y) + 2ab·Cov(X,Y), with both variances equal to 1 because the variables are standardized:

Var(V)  =  0.3²  +  0.7²  +  2(0.3)(0.7)(0.35)
        =  0.09  +  0.49  +  0.147
        =  0.727                                   sd(V) = sqrt(0.727) = 0.8526

Baseline: rank by engagement alone. How well does E track V? Correlation is covariance over the product of standard deviations, and sd(E) = 1:

Cov(E, V)   =  Cov(E, 0.3E + 0.7S)
            =  0.3·Var(E)  +  0.7·Cov(E, S)
            =  0.3         +  0.7 x 0.35
            =  0.545

corr(E, V)  =  0.545 / (1 x 0.8526)  =  0.639

That 0.639 is the number to beat. It is what a perfectly measured engagement signal buys you.

Now add the survey head. Rank by score = 0.3 E + 0.7 S_hat, where S_hat is the head’s prediction of satisfaction. Its reliability r = corr(S_hat, S) says how well that prediction tracks true satisfaction; the rest of S_hat is independent noise. r = 1 is a perfect satisfaction oracle, r = 0 is useless. Being standardized, Cov(S_hat, S) = r and Cov(S_hat, E) = r·rho.

Cov(score, V)  =  Cov(0.3E + 0.7Ŝ,  0.3E + 0.7S)
               =  0.09·Var(E) + 0.21·Cov(E,S) + 0.21·Cov(Ŝ,E) + 0.49·Cov(Ŝ,S)
               =  0.09        + 0.21·rho      + 0.21·r·rho    + 0.49·r
               =  0.09 + 0.0735 + 0.0735·r + 0.49·r
               =  0.1635  +  0.5635 r

Var(score)     =  0.09·Var(E) + 0.49·Var(Ŝ) + 2(0.3)(0.7)·Cov(E,Ŝ)
               =  0.09        + 0.49        + 0.42·r·rho
               =  0.58  +  0.147 r

corr(score, V) =  Cov(score, V) / ( sd(score) x sd(V) )
               =  (0.1635 + 0.5635 r) / ( sqrt(0.58 + 0.147 r) x 0.8526 )

Substitute values of r and find where it crosses 0.639. One row worked, at r = 0.49: the numerator is 0.1635 + 0.5635 x 0.49 = 0.4396, the denominator is sqrt(0.58 + 0.147 x 0.49) x 0.8526 = sqrt(0.652) x 0.8526 = 0.8075 x 0.8526 = 0.6885, and 0.4396 / 0.6885 = 0.639.

 r     corr(score, V)
0.30       0.494
0.40       0.571
0.45       0.609
0.49       0.639      <- exactly the pure-engagement baseline
0.55       0.683
0.65       0.756

Breakeven at r ≈ 0.49. A survey head only has to correlate about 0.5 with true satisfaction to beat a perfectly measured engagement signal, and a head fit on 130 M labels over a shared representation lands at r ≈ 0.55-0.65 in practice — comfortably past the line.

Why the 10^11 engagement labels do not close the gap. They buy nothing, because p_click was already estimated far more precisely than any threshold that changes a ranking decision. Additional data on a signal you have already saturated has zero marginal value; the first data on a signal you have never measured has enormous marginal value. That asymmetry is the entire argument, and it is why the survey budget survives a cost review.

Three practical notes that get asked about:

• Ask pairwise, not Likert, where you can. A Likert question asks for a rating on a fixed scale — “rate this 1 to 5” — and different populations use such scales differently, some clustering at the ends and some in the middle. “Which of these two would you rather have seen?” has no such scale-use bias and produces a cleaner training target. Likert top-2-box is fine as a reported metric and noisy as a label.
• Response bias is real and correctable. Responders skew heavy-user, older, and higher-satisfaction. Reweight each response by one over that user’s modelled probability of responding — the same inverse-propensity idea used for position bias in Features — and report both weighted and unweighted numbers so the size of the correction is visible.
• The survey head cannot be the only defense. It is a smooth signal; the integrity system (The integrity interaction) handles the discrete one. A post that is 3% more satisfying and a post that violates policy are not points on the same axis.

5.4 Diversity: the constraint no pointwise score can express

One property of a good feed cannot be a weight in the score at all, however the score is built — and the pass that enforces it instead turns out to be the only thing bounding a runaway derived four sections later.

Value is computed one post at a time — it is pointwise. “Three posts from the same author in a row” is not a property of any post in that run; it is a property of the slate, the whole ordered page of 25 posts, and a pointwise argmax has nowhere to put it. That is the whole reason diversity is a re-selection pass over the scored list rather than another term in w. It is also why the pass is where The feedback loop formalized’s runaway is actually stopped, which makes its parameters part of the objective whether or not anyone reviews them as such.

The pass fills the 25 slots greedily, meaning it picks the best available post for slot 1, then the best for slot 2 given what slot 1 took, and so on, never revisiting an earlier choice. At each slot it takes

argmax_p  [ Value(p)  -  sum_c  lambda_c · violation_c(p | slate so far) ]

Reading that expression: argmax_p means “the post p that makes the following quantity largest”. violation_c counts how far p pushes the partial slate past constraint c, and lambda_c is how much that costs. So a post with a high Value can still lose its slot if it violates enough constraints.

Greedy is not optimal. The optimal slate is a constrained assignment problem; the greedy version lands within a few percent of it on the pointwise objective, and it is what fits in the 6 ms Scale and cost allots to combine + demotion + diversity for 600 candidates.

Five constraints, and the column that matters is the last one: what each costs in forgone Value, measured rather than assumed. “Hard” means the constraint can never be violated; “soft” means it enters as a penalty the score can overcome.

Eleven percent of pointwise value, and that number is the point of the table. A diversity pass with no measured cost is a pass nobody can argue with, and one whose cost is 11% is a pass that has to justify itself against the 4.9% the ads interleave alone spends. Compute it the same way you compute the demotion’s cost in Demotion is how you spend a classifier that is too weak to remove with: log the counterfactual slate — the page the unconstrained argmax would have produced, recorded but never shown — and report the difference.

The author cap is the only thing in the system that bounds The feedback loop formalized’s fixed point, and it bounds it by construction rather than by degree. A hard cap of k_a per window pins impression share at s_v <= 3/25 = 0.12 for every author regardless of affinity, so the self-reinforcing right-hand side of a_v ∝ r_v · exp(beta · a_v) saturates and the runaway branch has nowhere to go. It does not slow the runaway; it removes it. But 12% of a user’s feed from one author is a generous ceiling, and a system sitting against that ceiling is one whose feature definition is broken — which is why The feedback loop formalized’s fix is in the feature store and this one is a backstop with a number on it.

6. Training

Training the ranker comes down to six decisions, each with a reason attached. Two pieces of vocabulary recur: binary cross-entropy is the standard loss for a yes/no prediction, and a logit is the raw unbounded score a model produces before it is squashed into a probability.

Two rows carry non-obvious arithmetic and are worth reading twice — the sampling row, where a 10:1 downsample shifts every logit by exactly ln(10) = 2.303 and that shift is subtracted back so calibration survives; and the split row, where a random rather than temporal split leaks a post’s own engagement counts across the train/eval boundary.

The continual-update loop is where the subtle failure lives. An hourly model trained on the last hour of labels is trained on impressions that the previous hourly model chose. Over a day that is 24 rounds of a policy training on its own output, with no exploration floor to anchor it. The exploration budget from The freshness trap your best features are missing when they matter most is not just for cold-start posts; it is the only source of counterfactual data in a continually-updated ranker, and without it the model’s estimates for anything it currently suppresses are frozen at whatever they were when it started suppressing them.

7. The integrity interaction

The ranker does not operate alone: another system decides whether a post is allowed to exist at all, and the two are coupled in a direction people rarely expect. Fixing the coupling takes a continuous penalty, and the fix carries a more general lesson: demotion is what lets you extract value from a classifier far too inaccurate to delete anything with.

The feed and the harmful-content system (ml-system-design/05) are not two independent services with a filter between them. That other system produces one number per post, p_violating, the estimated probability that the post breaks the platform’s rules; the feed consumes it. They are coupled, and the coupling runs in the direction candidates do not expect.

7.1 An engagement-maximizing ranker is a borderline-content-maximizing ranker

One table is enough to establish that the content a ranker most wants to promote sits immediately below the line at which content is deleted — and that this is a structural consequence of where the line was drawn, not an accident.

Take the integrity classifier’s score p_violating and bucket live inventory by it. As you move down the table toward content the classifier thinks is more likely to break the rules, the engagement column and the satisfaction column move in opposite directions.

Interaction rate rises monotonically — never falling — as the probability that a post violates policy goes up, right up to the removal line. From the cleanest bucket to the dirtiest servable one that is 0.091 / 0.038 = 2.4x more engagement, while satisfaction falls from 44% to 14%.

That is not a coincidence and it is not an artifact of the classifier. Content that provokes is content that gets responded to, and the policy line was drawn around provocation.

The consequence is exact: a ranker that maximizes engagement subject to “not removed” has its optimum pressed against the removal threshold from below. Removal alone cannot fix this, because removal is a step function and the ranker will simply find the highest point that is still on the servable side of the step.

7.2 Demotion, and the arithmetic of what it buys

The cure is to replace the step function with a ramp — and then to check, numerically, that the ramp is steep enough to beat the engagement gradient it is fighting.

The fix is a continuous penalty in the score, defined below the removal threshold. clip(x, 0, 1) means “hold x inside the range 0 to 1”, so the penalty switches on at p = 0.20 and stops deepening at p = 0.85. The 0.65 is the width of that ramp, 0.85 - 0.20:

d(p)   =  1 - 0.9 · clip( (p - 0.20) / 0.65 , 0, 1 )
Value' =  d(p) · Value

d(p) is a multiplier between 1.0 (no penalty) and 0.10 (a tenth of the original score). Work the worst servable case, p = 0.80:

(0.80 - 0.20) / 0.65  =  0.60 / 0.65  =  0.923
d(0.80)  =  1 - 0.9 x 0.923  =  1 - 0.831  =  0.169

The last column below inverts that multiplier: 1 / 0.169 = 5.9, so the post needs 5.9x the Value it had before to end up at the same final score.

A post at p = 0.80 now has to be six times better on everything else to occupy the slot it used to win outright.

Now check the penalty against the force it is fighting, because a demotion curve that is too shallow accomplishes nothing. The engagement gradient from An engagement maximizing ranker is a borderline content maximizing ranker is a factor of 2.4 across the same range; the demotion is a factor of 5.9. 5.9 beats 2.4, so the optimum moves off the boundary. That comparison — penalty slope against the engagement slope it is fighting — is how you decide whether a demotion curve is strong enough, and it is a question with a numeric answer rather than a policy debate.

7.3 Demotion is how you spend a classifier that is too weak to remove with

The demotion curve generalizes into the most portable idea in the chapter: what you can do with a model depends on what a mistake costs, and a cheap-mistake action can use a model that an expensive-mistake action cannot.

Two terms in the table. Precision is the share of the posts the classifier flags that really are violations, so low precision means many innocent posts are caught. Recall is the share of real violations the classifier catches. Raising the threshold buys precision and loses recall.

The two right-hand columns are different answers to the same rows: at a threshold of 0.50 the classifier is wrong two times in three, which is disqualifying for deleting a post and perfectly acceptable for pushing one down.

The asymmetry is in the cost of a false positive, meaning a legitimate post the classifier wrongly flags. Removing one costs the full post, an appeal, and a trust event; call it R. Demoting it by d = 0.45 costs roughly 55% of its impressions and nothing else; call it 0.09 R empirically, from the impression-loss-to-complaint-rate curve. Plugging into the standard threshold identity (Choosing a threshold from the cost matrix), the break-even precision for demotion sits about an order of magnitude lower than for removal, which is exactly why the demotion curve can start at p = 0.20 where precision is 0.14. Demotion is not a softer removal; it is the mechanism that lets a low-precision classifier do useful work at all. The reason one threshold cannot serve both dispositions is Why you cannot pick one threshold; what the feed adds is a third disposition below both of them, and a continuous one.

The whole scoring path in code, with the two places the value judgement is written down. Four things in it carry the argument: the BASE dictionary is the eleven base rates from Combining where the value judgement lives; POSITIVE and NEGATIVE are the value shares that sum to 1.000 and 0.400; lift() does the division-then-clip that makes the heads addable; and the value > 0 guard inside feed_value() is not defensive tidiness — its comment explains a real bug it prevents. The block ends by executing Combining where the value judgement lives’s two tables and asserting every number quoted there.

# Population base rates per served impression, measured weekly (5.2). These
# are the DENOMINATORS of the value model, so re-measuring them re-scales
# every weight -- a re-baseline goes through the same review as a weight edit.
BASE = {"click": 0.052, "like": 0.021, "comment": 0.004, "reshare": 0.002,
        "dwell10": 0.038, "dwell60": 0.014, "survey": 0.44,
        "hide": 0.006, "seefewer": 0.0024, "unfollow": 0.00040,
        "report": 0.00015}

# Weights are VALUE SHARES in lift units: at a post sitting at every base
# rate, term k contributes exactly w_k. Positives sum to 1.000 and negatives
# to 0.400, so the base-rate post scores 0.600 and every entry is readable as
# "the fraction of an average post's value this head carries". Config,
# reviewed by name, bounded in size by the ~20 long-term read-outs per year.
POSITIVE = {"survey": 0.350, "comment": 0.150, "reshare": 0.130,
            "dwell60": 0.120, "click": 0.100, "dwell10": 0.080, "like": 0.070}
NEGATIVE = {"report": 0.240, "unfollow": 0.085, "seefewer": 0.045,
            "hide": 0.030}

LIFT_CAP = 3.0   # p99(p_k) / base_k, measured per head; 2.8-3.4 across the 11


def lift(head, p):
    """Head probability in units of its own base rate, clipped at the p99.

    The division is what makes heads commensurable: p_survey and p_report
    differ by 2,900x in level, so weights on raw probabilities do not mean
    what they look like (5.2). The clip is a separate job -- a share weight
    constrains a term's contribution at the MEAN and says nothing about a
    post sitting at 22x, which is exactly where engagement bait lives (11.1).
    Neither is calibration, and calibration fixes neither.
    """
    return min(p / BASE[head], LIFT_CAP)


def demotion(p_violating, start=0.20, end=0.85, floor=0.10):
    """Continuous integrity penalty BELOW the removal threshold.

    A step function at the removal line leaves the ranker's optimum pressed
    against the step from below, because engagement is monotone increasing in
    p_violating up to that line. A ramp removes the boundary optimum.
    """
    if p_violating <= start:
        return 1.0
    span = min((p_violating - start) / (end - start), 1.0)
    return 1.0 - (1.0 - floor) * span


def feed_value(heads, p_violating):
    """Combine calibrated head probabilities into a single score.

    `heads` maps head name -> CALIBRATED probability. Calibration is a launch
    gate rather than a diagnostic here: the terms are added, so a head that is
    1.4x overconfident silently multiplies its own weight by 1.4, which is an
    unreviewed edit to the value vector. Calibration is necessary and it is
    not sufficient -- lift() handles the part it cannot reach.
    """
    value = sum(w * lift(k, heads.get(k, 0.0)) for k, w in POSITIVE.items())
    value -= sum(c * lift(k, heads.get(k, 0.0)) for k, c in NEGATIVE.items())

    # The `value > 0` guard is load-bearing, not defensive. demotion() returns
    # d < 1, and multiplying a NEGATIVE value by d shrinks its magnitude --
    # which moves the score UP. Without the guard, a post at p_violating=0.80
    # scoring -0.50 comes out at 0.169 * -0.50 = -0.085 and outranks a clean
    # post at -0.20: the integrity penalty would promote exactly the content
    # it exists to bury, and it would do so hardest on the worst posts.
    # Multiplicative demotion is monotone in p_violating only on the positive
    # half-line. Applying it only there keeps the score non-increasing in
    # p_violating everywhere. (A subtractive penalty needs no guard, but it
    # also does not scale with how much value there is to take away, which is
    # the property that makes the 7.2 table read in quality multiples.)
    return demotion(p_violating) * value if value > 0 else value


# --- 5.2's two tables, executed -----------------------------------------
# The raw-weight form, so the share column can be READ rather than asserted.
RAW = {"survey": 0.70, "dwell10": 0.25, "dwell60": 0.45, "click": 0.10,
       "like": 0.15, "comment": 0.55, "reshare": 0.80,
       "hide": 1.40, "seefewer": 2.10, "report": 12.0, "unfollow": 4.00}

_pos = sum(RAW[k] * BASE[k] for k in POSITIVE)
_neg = sum(RAW[k] * BASE[k] for k in NEGATIVE)
for _k in sorted(POSITIVE, key=lambda h: -RAW[h] * BASE[h]):
    print("  %-8s raw w %5.2f  base %.5f  w*base %.5f  = %5.1f %% of the positive block"
          % (_k, RAW[_k], BASE[_k], RAW[_k] * BASE[_k], 100 * RAW[_k] * BASE[_k] / _pos))
print("positive block %.5f    negative block -%.5f  =  %.1f %% of positive"
      % (_pos, _neg, 100 * _neg / _pos))
assert abs(_pos - 0.33595) < 1e-5
assert abs(_neg - 0.01684) < 1e-5
assert abs(RAW["survey"] * BASE["survey"] / _pos - 0.917) < 5e-4   # not one term of seven
assert abs(_neg / _pos - 0.0501) < 5e-4                            # the whole negative block
assert abs((RAW["comment"] * BASE["comment"])
           / (RAW["like"] * BASE["like"]) - 0.698) < 5e-4          # not 3.7x -- 0.70x

# The ordinal, because summarising a table in a sentence is where these go wrong.
_terms = sorted((round(abs(RAW[k] * BASE[k]), 12), k) for k in RAW)
print("three smallest terms:", [(n, v) for v, n in _terms[:3]])
assert [n for _, n in _terms[:3]] == ["reshare", "unfollow", "report"]

# And the lift form: the weights ARE the value shares, so a base-rate post
# scores exactly 0.600 and each entry is the share of it that head carries.
_base_post = {k: BASE[k] for k in BASE}
print("base-rate post scores %.3f  (positives %.3f, negatives %.3f)"
      % (feed_value(_base_post, 0.0), sum(POSITIVE.values()), sum(NEGATIVE.values())))
assert abs(sum(POSITIVE.values()) - 1.000) < 1e-9
assert abs(sum(NEGATIVE.values()) - 0.400) < 1e-9
assert abs(feed_value(_base_post, 0.0) - 0.600) < 1e-9

Two mechanisms that belong with it:

• Demote the author, not only the post, on repeat. A one-off false positive on a post is noise; a sustained pattern is signal, and the author-level penalty has far better precision because it aggregates.
• Never let demotion be invisible to measurement. Log the counterfactual rank. If you cannot report “posts demoted, impressions lost, and the survey delta on the affected slots,” you cannot tell a working demotion from a broken one.

8. Serving: fanout, and the celebrity arithmetic

Ranking settled, the delivery question remains: how does a post written once reach the candidate sets of millions of people? Two pure designs offer themselves, each fails on a different tail of the same distribution, and the hybrid’s threshold falls out of a memory budget rather than out of taste.

8.1 The two pure designs, priced

Price the two possible answers with arithmetic and both come out unusable — one because of a burst, the other because of a straggler.

Fanout is the general name for the one-to-many delivery step, and the design choice is when to do it. Fanout-on-write (push) does it at post time: when an author posts, append the post id to every follower’s inbox, a per-user list of post ids kept in a key-value store.

posts                        500 M / day
mean followers of a poster   400          (mean >> median 200; the tail is heavy)
inbox writes                 500 M x 400  =  2.0 x 10^11 / day

per second                   2.0e11 / 86,400  =  2.3 M writes/s average
                                                 ~5.8 M/s peak at 2.5x

Sustained, 2.3 M writes/s is a large but ordinary workload for a sharded key-value store — a simple id-to-value database split across many machines by key.

The problem is not the mean. Split the authors into the tiny head of very large accounts and everyone else, and the two halves come out equal:

head accounts:   5,000 accounts x 4 M followers avg x 5 posts/day  =  1.0 x 10^11 writes/day
everyone else:   ~500 M posts x 200 followers (the median)         =  1.0 x 10^11 writes/day
                                                                      ---------------------
                                                     total          =  2.0 x 10^11   (checks)
head accounts are 0.0002 % of 3 B active accounts and 50 % of all inbox writes

Two checks on that split, because the numbers have to hang together.

First, it reproduces the mean the block above assumed. The head writes 25,000 posts a day (5,000 accounts x 5 posts) at 4 M followers each; everyone else writes 500 M posts at 200 followers each:

(2.5e4 x 4e6  +  5e8 x 200) / 5e8  =  (1e11 + 1e11) / 5e8  =  400 followers per post

Second, count the posts rather than the writes. The head’s 25,000 posts are 25,000 / 500 M = 0.005% of the day’s posts, and they carry half the day’s writes. That is the celebrity problem in one line.

And the burst is worse than the average. One post from a 100M-follower account:

writes for one post          100 M
cluster write capacity       ~3 M writes/s
time to deliver              100e6 / 3e6  =  33 seconds of the ENTIRE global write budget

One post consumes 33 seconds of every write the platform can do, and there are thousands of such accounts whose posting is correlated in time because they are all reacting to the same events. That is the celebrity problem, and it is not a tuning issue — fanout-on-write is structurally unable to serve a power-law follower distribution, one in which a handful of accounts have followings orders of magnitude larger than the typical account rather than merely somewhat larger.

Fanout-on-read (pull) is the other pure design: do the work at read time instead. At each feed request, fetch each followed author’s recent posts and merge them.

sessions                     8 B / day
authors merged per session   200
timeline fetches             8e9 x 200  =  1.6 x 10^12 / day  =  18.5 M fetches/s

Eight times the operation count of push (1.6e12 / 2.0e11 = 8), and every one of them is on the critical path — meaning the user is sitting there waiting for it, rather than it happening in the background where nobody notices.

The latency argument is the decisive one, and it turns on a single exponent. Suppose 1% of fetches take longer than 20 ms, and the 200 fetches are independent. Then each fetch is fast with probability 0.99, and all 200 are fast with probability 0.99 multiplied by itself 200 times:

P(no straggler)  =  0.99^200  =  0.134

87% of feed loads would contain at least one 20 ms-plus straggler — a straggler being the one slow response that everything else waits on — and the request’s latency is the maximum over all 200 fetches, not their average. Pure pull is not viable when 200 things must all come back.

8.2 The hybrid, and the threshold derived

Combine the two designs so that each handles the tail the other cannot, and the only open question is where the boundary between them goes — which a memory budget answers.

Push for the long tail, pull for the head. The threshold T on follower count is chosen by one property: the pulled set must be small enough to live in RAM on every feed host, because that is what removes the straggler term.

accounts with > 1 M followers        ~50,000 globally
recent posts kept per account        50
bytes per entry (id, author, ts, type, score hint)   32 B

hot-set size  =  50,000 x 50 x 32 B  =  80 MB

Eighty megabytes. That hot set — the small, frequently-read slice kept close to the code — fits in the memory of every feed server, refreshed by a broadcast stream at sub-second lag. Reading it costs zero network round trips, so the 0.99^n straggler argument evaporates entirely: the merge is a scan over an array already in memory.

The threshold routes each author down one of two paths, and both feed the same merge at read time:

flowchart LR
    A["Author posts"] --> D{"followers x posts/day<br/>over threshold?"}
    D -->|"long tail"| W["Fanout-on-write:<br/>append to each follower inbox"]
    D -->|"head account"| H["Broadcast to head-account<br/>hot set: 80 MB, in process"]
    W --> INB[("Per-viewer inbox<br/>sharded KV")]
    INB --> M["Merge at feed request"]
    H --> M
    M --> R["Candidates for ranking"]

What the hybrid buys. The 50,000 hot-set accounts are no longer pushed to anyone, so all their writes disappear from the push path. Count them in two groups — the 5,000 giants already priced in The two pure designs priced, and the other 45,000:

the 5,000 accounts already priced in 8.1                    1.0e11 writes/day
the other ~45,000 above 1 M, at ~1.1 M followers
   and the ordinary 0.2 posts/day               45,000 x 1.1e6 x 0.2  =  1.0e10
                                                                         --------
                                               writes removed, total  =  1.1e11

writes removed        1.1e11 / 2.0e11                      =  55 %
what is left          2.0e11 - 1.1e11  =  9.0e10 writes over 5e8 posts
                      9.0e10 / 5e8  =  180 followers per pushed post
worst-case burst removed                                   =  100 %  (no head is ever pushed)

The last line but one is the check that the threshold is in the right place: after the hybrid, the mean fanout of a pushed post is 180, against a median follower count of 200. The push path is now serving exactly the accounts it was designed for, and the accounts whose fanout was pathological are gone from it entirely.

Half the write volume and all of the catastrophic bursts, for 80 MB of RAM per host. The reason the trade is this good is a structural property of the follower graph: the accounts that are expensive to push are exactly the accounts that many people follow, which makes their timelines maximally cacheable — the cost of push scales with followers while the cost of pull is amortized across them. That sentence is the answer to the question.

Two details worth volunteering:

• The threshold is not really about followers, it is about followers x posts_per_day. A 200k-follower account posting 40 times a day generates more writes than a 1M-follower account posting twice. Compute the product, sort, and take the top-K until the hot set hits its RAM budget. Follower count is a convenient proxy, not the criterion.
• Inbox lists are capped and given a TTL. A TTL, or time to live, is an expiry after which an entry is discarded automatically. Keep the last 500 ids or 72 hours, whichever binds first, and materialize — actually create and maintain — inboxes only for users active in the last 30 days.

3 B recently-active users x 500 entries x 24 B  =  36 TB
x 3 replicas                                    =  108 TB

An inbox entry is 24 B — post id, author id, timestamp — and not the 32 B of the hot set. The hot set carries a score hint and the inbox does not, because the inbox is a delivery log while the hot set is a pre-ranked cache.

Price the naive version to see what the cap and the TTL are worth. Materialize inboxes for all 5 B registered users rather than the 3 B active ones, and let each list grow to whatever 72 hours of a graph produces — around a thousand entries for the connected half:

5e9 users x 1,000 entries x 24 B  =  120 TB   before replication

120 TB / 36 TB  =  3.3 x

3.3x the storage, for zero benefit, entirely from materializing inboxes nobody reads.

8.3 Architecture

Everything so far assembles into one request path, from the feed request arriving to the ranked feed going out.

The diagram runs top to bottom. The important shape is the funnel in the middle: four candidate sources fan in to ~3,000 candidates, a hard filter and a cheap ranker cut that to 600, an expensive ranker scores those 600, and three post-processing stages turn them into 25 slots. The four coloured boxes are the four places this chapter’s arguments live.

flowchart TD
    REQ(["Feed request"]) --> CTX["Session context<br/>viewer features<br/>seen-set bloom filter"]

    CTX --> INBOX[("Push inbox<br/>sharded KV<br/>500 ids · 72 h TTL")]
    CTX --> HOT["Head-account hot set<br/>50k authors · 80 MB<br/>IN PROCESS · no RPC"]
    CTX --> OON["Out-of-network retrieval<br/>ANN + topic index<br/>incremental insert"]
    CTX --> GRP[("Group / page sources")]

    INBOX --> MERGE["Merge · dedupe<br/>seen filter<br/>~3,000 candidates"]
    HOT --> MERGE
    OON --> MERGE
    GRP --> MERGE

    MERGE --> HARD{"Integrity hard filter<br/>p_violating ≥ 0.85<br/>blocks · mutes · locale"}
    HARD -->|drop| X(["Removed"])
    HARD -->|pass| LIGHT["Light ranker<br/>3,000 -> 600<br/>2-tower dot product"]

    LIGHT --> HYD["Feature hydration<br/>edge · author · content<br/>post counts w/ maturity gate"]
    HYD --> HEAVY["Multi-task ranker<br/>shared MoE bottom<br/>11 heads · calibrated"]

    HEAVY --> VAL["Value combine<br/>shares x clipped lift<br/>1.000 pos · 0.400 neg"]
    VAL --> DEM["Integrity demotion<br/>d p · Value"]
    DEM --> DIV["Diversity pass · greedy<br/>author cap 3/25 · type ≤40%<br/>OON floor by segment · ads 1-in-6<br/>~11% of pointwise value"]
    DIV --> OUT(["Ranked feed"])

    OUT --> LOG[("Impression log<br/>position · candidate set<br/>counterfactual rank")]
    LOG --> TRAIN["Hourly continual update<br/>+ weekly full retrain"]
    TRAIN -.-> HEAVY

    style HOT fill:#bc6c25,color:#fff
    style HEAVY fill:#1d3557,color:#fff
    style VAL fill:#2d6a4f,color:#fff
    style DEM fill:#9d0208,color:#fff

Read it top to bottom as one request. A feed request arrives and the session context is assembled: the viewer’s features plus a seen-set bloom filter, a compact probabilistic structure that answers “have we shown this person this post already?” using a fraction of the memory a real list would need, at the cost of occasional false alarms. Four candidate sources are then queried in parallel. The push inbox is the per-viewer list from The hybrid and the threshold derived, held in a sharded key-value store, capped at 500 ids with a 72-hour time-to-live. The head-account hot set is the 80 MB of the 50,000 largest accounts, read in-process with no RPC — no remote procedure call, no network hop, which is exactly why the straggler arithmetic does not apply to it. The out-of-network retrieval path queries the approximate-nearest-neighbour and topic indexes that accept incremental inserts (The index turns over 33 per day and that kills nightly rebuilds). And the group and page sources supply posts from communities the viewer belongs to rather than accounts they follow, which is the second half of the 309-post inventory counted in The inventory is small and that inverts the retrieval problem.

Those four streams merge into roughly 3,000 candidates, with duplicates removed (dedupe, since the same post can arrive from two sources) and anything the seen filter recognizes dropped. The integrity hard filter then removes what must never be shown at all: posts at p_violating ≥ 0.85, plus accounts the viewer has blocked or muted, plus anything ineligible in their locale — the step-function disposition of An engagement maximizing ranker is a borderline content maximizing ranker, applied before any model spends effort on it. The light ranker cuts 3,000 to 600, feature hydration fetches the actual feature values for those 600 (the word simply means filling in a skeleton record with its data), and the heavy multi-task ranker scores them.

The last three stages are where this chapter’s arguments live. Value combine applies Combining where the value judgement lives: value shares multiplied by clipped lifts, with the positive weights summing to 1.000 and the negative ones to 0.400. Integrity demotion multiplies that by d(p). The diversity pass turns 600 scored posts into 25 ordered slots, and a ranked feed is returned. Everything served is written to the impression log — including position, the candidate set, and the counterfactual rank — which feeds the hourly and weekly retraining that updates the heavy ranker.

Four boxes carry a fill and the fifth deliberately does not. The orange box is the celebrity fix — the hot set, sized by a memory budget rather than by processor time. The navy box is the heavy multi-task ranker, the model every stage above it exists to feed and every stage below it exists to correct. The green box is where the value judgement lives, and the red box is the only thing standing between the green box and An engagement maximizing ranker is a borderline content maximizing ranker. The unfilled box after it is the diversity pass, and it is unfilled because it is not a scoring stage at all: it is a re-selection over the scored list, the only place a slate-level property can be expressed, which is why it costs 11% of pointwise value and why The feedback loop formalized’s runaway terminates there rather than in the objective.

9. Scale and cost

Every capacity number in this chapter and in system-design 11 is derived from the same handful of traffic figures, so fix those first, then spend the request’s 269 ms across its stages — and notice that the resource the ranker is actually constrained by is not the one everyone sizes for.

Five numbers, and everything else in the chapter descends from them. The per-second figures are just the daily ones divided by 86,400 seconds; the 2.5x peak factor is the usual allowance for the fact that traffic is not flat across a day.

DAU                      2 B
sessions/day             8 B          (4 per DAU)
impressions/day          200 B        (25 per session)
feed requests            8 B/day  =  8e9 / 86,400  =  92.6 k/s average
                                  ->  232 k/s peak at 2.5x
posts created            500 M/day  =  5e8 / 86,400  =  5,787/s

Latency budget, p50 at peak

Here is where the 269 milliseconds of a median request go. p50 is the median request and p95 the slow 5%; queueing is time spent waiting for a machine that is busy with someone else’s request, which is why the p95 exceeds the sum of the stages.

The stages appear in the order the Architecture diagram runs them. The line to look for is not any of the model stages.

auth + viewer feature fetch                        15 ms
inbox read, 500 ids, sharded KV                    12 ms
head-account hot-set merge (in process)             1 ms
out-of-network ANN, 3 indexes, top-200 each        25 ms
dedupe + seen-filter (bloom) + integrity hard      12 ms
light ranker, 3,000 -> 600                         18 ms
feature hydration for 600 candidates               40 ms
heavy multi-task ranker, 600 candidates            35 ms
value combine + demotion + diversity                6 ms
content payload hydration (text, media urls, ads)  60 ms
serialization + network                            45 ms
                                                  -------
                                                  269 ms  p50
                                                  ~430 ms p95 with queueing

Feature hydration (40 ms) plus content payload hydration (60 ms) is 100 ms — 100 / 269 = 37% of the budget — and neither of them is machine learning. That is the usual shape and it is the usual surprise.

The consequence is a priority. Optimizing the ranker from 35 ms to 25 ms saves 10 ms, which is 10 / 269 = 3.7%. Batching the payload hydration properly saves around 40 ms, which is 40 / 269 = 15%. Fix the hydration first.

Ranker cost, derived

Now price the arithmetic of scoring 600 candidates per request at peak traffic.

The heavy ranker concatenates its inputs into a 512-number vector and passes it through fully connected layers of width 512 -> 1024 -> 512 -> 256 -> 11, ending at the eleven heads. A fully connected layer from a inputs to b outputs does a x b multiply-accumulates, so the layer widths give the cost directly.

Vocabulary: a MAC is one multiply-accumulate, the fused multiply-and-add that dominates neural network arithmetic and counts as two floating-point operations. MFLOP, GFLOP and TFLOP/s are millions, billions, and trillions-per-second of those operations.

MACs  =  512x1024 + 1024x512 + 512x256 + 256x11
      =  524,288  + 524,288  + 131,072 + 2,816
      =  1,182,464  MACs

         x2 FLOP per MAC  =  2,364,928 FLOP  =  2.36 MFLOP / candidate

per request   600 candidates x 2.36 MFLOP  =  1,418 MFLOP  =  1.42 GFLOP
at peak       232 k/s x 1.42 GFLOP  =  3.30 x 10^14 FLOP/s  =  330 TFLOP/s

accelerators at 150 TFLOP/s effective (small batch, real utilization)
              330 / 150  =  2.2 k  ->  ~3,000 with headroom and redundancy

Three thousand accelerators is the headline number a candidate is expected to produce. The next subsection says why it is the wrong thing to have sized.

The constraint is not FLOPs

Sizing the arithmetic was the easy half, because the arithmetic is not what the fleet is actually limited by. What is, is memory traffic — specifically the embedding lookups, where each of the ~60 sparse features on a candidate (its author id, its topic, its language, and so on) is turned into a vector by reading a row out of a big table.

fp16 below means each number is stored in 16 bits, so two bytes.

sparse features per candidate    60
embedding dim                    64, fp16
bytes per candidate lookup       60 x 64 x 2  =  7,680 B  =  7.68 KB
per request                      600 x 7.68 KB  =  4.6 MB of scattered reads
at peak                          232 k/s x 4.6 MB  =  1.07 TB/s of random access

Those are not sequential reads. They are 36,000 unpredictable row lookups per request, scattered across tables far too big to sit anywhere convenient.

Take the author embedding table alone:

300 M authors x 64 numbers x 2 B  =  38.4 GB

That is too large for one accelerator’s HBM — high-bandwidth memory, the fast memory attached directly to a GPU, of which a single card has tens of gigabytes. So the table shards across machines, and a request’s 600 candidates hit 600 random rows spread across those shards.

If those lookups cross the network, you need 1.07 TB/s x 8 bits = 8.5 Tbit/s of interconnect to keep 330 TFLOP/s of arithmetic fed. That is the real constraint.

The dense math is embarrassingly parallel — it splits across machines with no coordination — while the embedding lookups are a distributed random-access problem. So the design pressure runs the opposite way from the intuition: route each candidate to the shard that already holds its author row, so the lookup is local and only the 11 output scores cross the wire. Getting that inversion right is worth more than any architecture change to the ranker.

Three standard reductions on top of it:

• Hash the long tail of author ids into a shared table, so rare authors share rows and the table stops growing with the account count.
• Quantize the embeddings to int8 — store each number in one byte instead of two. That halves the traffic and costs about 0.001 of AUC.
• Cache the viewer-side embedding once per request rather than once per candidate. The viewer does not change across the 600 candidates.

10. Metrics

Some measurements gate a launch; most merely inform it, and telling the two apart matters more on a feed than anywhere else — because the dashboard that says ship is compatible with a system that got worse, and no experiment you can run will settle it.

10.1 Offline

Plenty can be measured before shipping. On a feed, almost none of it decides anything, and the reason is worth having ready.

The third column is the one that matters. A gate blocks a launch on its own; a diagnostic only tells you where to look. Notice how few gates there are.

Offline metrics on a feed are weaker than in almost any other ML system, and the reason is worth stating: the logged data was produced by a policy, the new policy shows different items, and the counterfactual is unobserved for every item that was not shown. Off-policy estimation gives you a variance-heavy tripwire, not a decision. Plan for the online experiment to be the decision from the start.

10.2 Online

On live traffic, the metrics arrange themselves from fastest and least trustworthy at the top to slowest and most trustworthy at the bottom. That ordering is the whole point of the table — the metrics you can act on quickly are the ones most easily gamed, and The dashboard that says ship and the system that got worse shows what happens when a launch is decided on the top rows alone.

10.3 The dashboard that says ship and the system that got worse

Here is a launch decision where every fast metric is up, every slow one is down, and the most important effect does not appear in the experiment at all.

Ranker v7, four-week A/B at 2% of users. p below is the p-value, the probability of seeing a difference this large if there were really no difference — small values mean the result is unlikely to be noise, and pp means percentage points.

The block has three parts, separated by blank lines: the fast metrics, then the slow and elicited ones, then two lines that were measured six weeks after the experiment ended.

sessions per DAU                    +3.1 %      p < 0.001
interactions per session            +5.4 %      p < 0.001
time spent                          +2.2 %      p < 0.001
                                                                <- everything above ships it

survey top-2-box                    41.3 % -> 39.1 %   (-2.2 pp, p = 0.004)
hides per 1,000 impressions         2.1  -> 2.5        (+18 %,   p < 0.001)
"see fewer posts like this"         +24 %              p < 0.001
reports per 1,000 impressions       +6 %               p = 0.03
D28 return                          -0.14 %            p = 0.31   (underpowered at 4 weeks)

six weeks after a full launch, on the SUPPLY side:
share of inventory classified engagement-bait     2.1 % -> 5.6 %
median posts/day by top-decile authors            3.1  -> 4.4

Every metric that reads out in a week is up. Every metric that requires asking a person, or waiting a quarter, or measuring the producers, is down. This is the same structure as the assistant-chatbot single-number trap and the watch-time trap, with one addition that is specific to feeds: the last two lines did not exist during the experiment. The supply response is invisible to any A/B smaller than the population, because creators optimize against the ranker that most of their audience sees, not the one 2% of it sees.

That is not a fixable experimental design flaw. It is a property of the system.

10.4 What to do about it, given that

If the most important effect is unmeasurable by an ordinary experiment, what remains? Five things.

• Long-term holdouts. Hold 0.5-1% of users on a frozen ranker for 6-12 months. Not to gate launches — to measure the accumulated delta of everything you launched, which no individual experiment measures. Expect the aggregate holdout delta to be smaller than the sum of the individual experiment wins, and treat the gap as your budget for novelty (a change looks good simply because it is new) plus drift (the population and content move underneath you), as in math 02.

• Reverse holdouts for supply effects: ramp a change to 100% and hold a geography or a creator cohort out, so the treated population is large enough for supply to respond.

• CUPED on the engagement metrics (math 02) to reclaim statistical power. CUPED stands for controlled experiments using pre-experiment data: it subtracts each user’s own pre-experiment behaviour from their outcome, which removes a large chunk of between-user variance and so shrinks the sample size needed. You will need it for the retention arm.

• Guardrails as independent blockers, not as terms in a weighted score. Negative feedback rate, report rate, and survey delta each block a launch on their own, because the point is precisely that they cannot be traded against a good engagement number.

• Size the retention arm honestly. The standard sample-size formula for detecting a difference delta in a rate p is n = 16·p(1-p)/delta² per arm (The sample size formula derived). For 0.2 percentage points on a 68% D28 base, p = 0.68 and delta = 0.002:

  n  =  16 x 0.68 x 0.32 / 0.002^2
     =  3.4816 / 0.000004
     =  870,000 users per arm

  870,000 users is affordable at 2 B DAU. The binding constraint is duration, not n — the effect takes 8-12 weeks to develop, and you cannot cleanly ship anything else to those users meanwhile.

11. Failure modes

Each way the system breaks gets a mechanism, a detector, and a control — the shape any answer to “what goes wrong” should take — and several of these failures are caused by the design working exactly as specified.

11.1 Engagement bait amplification

Take a real piece of engagement bait and score it four ways, and it turns out that neither of the two defences works without the other.

Four columns in the block below. The first is what the ranker’s eleven heads predict for this post. The second is each head’s population base rate from Combining where the value judgement lives. The third is the first divided by the second — the lift. The fourth applies the cap at L = 3.

The rows are sorted by lift, so read top and bottom. The post is at 22x on comments and 0.25x on the survey question. That gap is the whole failure mode in two numbers.

POST:  "99% of people can't name a country with no letter 'A'.
        LIKE if you can. COMMENT your answer. SHARE to challenge a friend."

ranker heads:                   base rate      lift    clipped at L = 3
    p_comment  0.088              0.004        22.0 x        3.00
    p_reshare  0.031              0.002        15.5 x        3.00
    p_like     0.191              0.021         9.1 x        3.00
    p_report   0.00040            0.00015       2.7 x        2.67
    p_seefewer 0.0061             0.0024        2.5 x        2.54
    p_unfollow 0.00070            0.00040       1.8 x        1.75
    p_hide     0.009              0.006         1.5 x        1.50
    p_click    0.061              0.052         1.2 x        1.17
    p_dwell10  0.042              0.038         1.1 x        1.11
    p_dwell60  0.006              0.014         0.43 x       0.43
    p_survey   0.11               0.44          0.25 x       0.25

Now score it four ways with the Combining where the value judgement lives vector, switching the two defences on and off independently. The reference throughout is the base-rate post at Value = 0.600; across a week of served impressions Value has mean 0.600 and standard deviation 0.33.

The two defences are the survey head (is it in the vector at all?) and the lift cap (is any head allowed to run past 3x?). Four combinations, and only the last one works.

(The engagement-only rows redistribute the survey head’s 0.350 share pro rata across the six behavioural heads, so all four variants are scored against the same 0.600 reference. z is quoted only for the clipped score, because the unclipped one has no usable sd — the tail that produces 8.604 is the same tail that makes the moment meaningless, which is Property 3 restated.)

Neither fix works alone, and the table is how you know.

Clipping without the survey head takes the bait from 14.3x to 1.8x. That is a large improvement that still leaves it in the top five. The reason: with the survey head removed, its 0.350 share is redistributed across the six behavioural heads, so comment, reshare and like carry more weight than before — their clipped terms come to 1.62 against a 0.948 negative block that is doing most of the work by itself.

The survey head without clipping takes it from 14.3x to 8.9x, and 8.9x is still slot 1. The survey head has 0.350 x 0.25 = 0.0875 to spend, and it is arguing against a comment term of 0.150 x 22 = 3.30. It is outgunned 38 to 1.

Together they take it to 0.74x, below the average post, and it is not served. The clip caps what solicited actions can buy; the survey head, now able to matter because no single behavioural term can run away from it, spends its whole budget against the post.

This is the concrete answer to “why isn’t calibration enough”: every probability in that block is already the calibrated truth, and three of the four rows still ship the bait.

The full breakdown of the shipped row, term by term, since the argument is in the arithmetic. Each line is a share weight from Combining where the value judgement lives times a clipped lift from the block above:

+ comment   0.150 x 3.00  =  0.450        - report    0.240 x 2.67  =  0.640
+ reshare   0.130 x 3.00  =  0.390        - unfollow  0.085 x 1.75  =  0.149
+ like      0.070 x 3.00  =  0.210        - seefewer  0.045 x 2.54  =  0.114
+ click     0.100 x 1.17  =  0.117        - hide      0.030 x 1.50  =  0.045
+ dwell10   0.080 x 1.11  =  0.088                                    -----
+ survey    0.350 x 0.25  =  0.088                                    0.948
+ dwell60   0.120 x 0.43  =  0.051
                             -----
                             1.395        Value = 1.395 - 0.948 = 0.447

Executed, from the Features input row through the Combining where the value judgement lives vector to the served/not-served decision. The first six lines are the point: five of the eleven head probabilities are computed by dividing the Features feature row’s own counts by its own impression count, so the input and the output are literally the same numbers.

# The section 4 feature row, and the eleven head probabilities it produces.
# FIVE of the eleven are the post-count features divided by that row's own
# impression count -- input and output are the same numbers, which is the
# only way to check that a worked example is worked rather than asserted.
BAIT_COUNTS = {"impressions": 41_900, "likes": 8_003, "comments": 3_687,
               "reshares": 1_299, "hides": 377}
_n = BAIT_COUNTS["impressions"]
bait = {"like":    BAIT_COUNTS["likes"] / _n,        # 0.191
        "comment": BAIT_COUNTS["comments"] / _n,     # 0.088
        "reshare": BAIT_COUNTS["reshares"] / _n,     # 0.031
        "hide":    BAIT_COUNTS["hides"] / _n,        # 0.009
        "click": 0.061, "dwell10": 0.042, "dwell60": 0.006, "survey": 0.11,
        "report": 0.00040, "seefewer": 0.0061, "unfollow": 0.00070}

print("head        p        base      raw lift   clipped   contribution")
for _k in list(POSITIVE) + list(NEGATIVE):
    _w = POSITIVE.get(_k, -NEGATIVE.get(_k, 0.0))
    print("  %-9s %.5f  %.5f  %7.2fx  %7.2f   %+.3f"
          % (_k, bait[_k], BASE[_k], bait[_k] / BASE[_k],
             lift(_k, bait[_k]), _w * lift(_k, bait[_k])))

pos_block = sum(w * lift(k, bait[k]) for k, w in POSITIVE.items())
neg_block = sum(c * lift(k, bait[k]) for k, c in NEGATIVE.items())
value = feed_value(bait, 0.04)                      # p_violating from the section 4 row
print("Value = %.3f - %.3f = %.3f   vs the average post at 0.600: %.2fx  (z = %+.2f)"
      % (pos_block, neg_block, value, value / 0.600, (value - 0.600) / 0.33))

assert abs(bait["comment"] / BASE["comment"] - 22.0) < 5e-3    # 22x on the costly head
assert abs(bait["survey"] / BASE["survey"] - 0.25) < 1e-9      # 0.25x on the elicited one
assert abs(pos_block - 1.395) < 5e-3
assert abs(neg_block - 0.948) < 5e-3
assert abs(value - 0.447) < 5e-3
assert abs(value / 0.600 - 0.74) < 5e-3
assert value < 0.600                                # NOT SERVED, which is the claim

# Neither defense works alone, which is the actual point of the table above.
_unclipped = (sum(w * bait[k] / BASE[k] for k, w in POSITIVE.items())
              - sum(c * bait[k] / BASE[k] for k, c in NEGATIVE.items()))
print("full vector, lift UNCLIPPED: %.3f  =  %.1fx the average post"
      % (_unclipped, _unclipped / 0.600))
assert abs(_unclipped - 5.348) < 5e-3 and abs(_unclipped / 0.600 - 8.9) < 5e-2

The bait wins by 22x on the costliest engagement head and loses by 4x on the only head that asked a person. Three defenses, in order of durability:

1. The survey head, weighted enough to matter, over lifts that are clipped so it can matter (Combining where the value judgement lives).
2. A bait classifier as a demotion signal, trained on the explicit-solicitation pattern. Works, and is an arms race — the phrasing mutates faster than the classifier retrains, so treat the classifier as a tax that raises the cost of bait rather than a filter that eliminates it.
3. Discount the engagement types the bait solicits, conditioned on solicitation. A like that was explicitly asked for is worth less than a spontaneous one. This is the most robust of the three because it attacks the mechanism rather than the surface form — and it is the targeted version of the lift cap, which makes the same discount unconditionally and therefore also charges it to the honest post that happens to be at 10x.

11.2 Narrowing, measured

The filter-bubble claim is usually argued in the abstract. It has a number.

Entropy measures how spread out a distribution is, in units called nats. Raw entropy is hard to interpret, so convert it with exp(entropy), which gives an effective count: the number of topics the viewer would be seeing if they saw that many in equal proportions. exp(3.41) = 30 and exp(2.28) = 10.

topic entropy of a user's impressions, over 12 weeks on a count-based affinity ranker

week          1     3     6     9    12
entropy    3.41  3.18  2.84  2.51  2.28   nats
eff. topics  30    24    17    12    10    = exp(entropy)

The user did not become less curious. The ranker stopped offering, so the user stopped clicking, so the ranker stopped offering. Topic entropy is the cheapest guardrail in the whole system and almost nobody puts it on the dashboard.

11.3 The feedback loop, formalized

That narrowing can be written as an equation, and the equation says the runaway is caused by how one feature is defined rather than by anything the model learned.

Setting up the loop

Three quantities, for one viewer and one author v:

• Affinity a_v — the feature encoding how much this viewer likes author v. Crucially, define it the way most systems do: as a decayed count of the viewer’s interactions with that author.
• Impression share s_v — the fraction of the viewer’s feed that author gets. The ranker makes this increase with affinity; take s_v proportional to exp(beta · a_v), where beta is how sensitive the ranker is to affinity.
• True interest rate r_v — how often the viewer would actually interact with a post from that author. This is the thing the system is supposed to be estimating, and it does not change.

Now close the loop. Interactions accrue at r_v times however many impressions the author got, and the decayed count settles where accrual balances decay:

steady state:   a_v  =  r_v · s_v · I / gamma          gamma = decay rate, I = total impressions
substituting:   a_v  proportional to  r_v · exp(beta · a_v)

That is a fixed-point equation with a self-reinforcing right-hand side. A fixed point is a value that reproduces itself when you plug it in. Self-reinforcing means a nudge upward makes the right-hand side push it further up rather than pulling it back. Whether a fixed point exists at all is the question.

Deriving the threshold

Collect every constant into one symbol K, because the threshold turns out to be a statement about that constant:

a  =  K · exp(beta · a)          K  =  r_v · I / gamma, times the share normalizer

A fixed point is a value of a where the two sides are equal — that is, a root of g(a) = K·exp(beta·a) - a. Since g is the sum of an exponential and a straight line, it is convex (curves upward everywhere), so it has a minimum and it has a root exactly when that minimum sits at or below zero.

Find the minimum by setting the derivative to zero:

g'(a)  =  K·beta·exp(beta·a) - 1  =  0
       ->  exp(beta·a)  =  1 / (K·beta)
       ->  beta·a       =  -ln(K·beta)
       ->  a*           =  -ln(K·beta) / beta

Evaluate g there. The first term simplifies because K·exp(beta·a*) = K · 1/(K·beta) = 1/beta, and the second is just -a*:

g(a*)  =  1/beta  +  ln(K·beta)/beta   =   ( 1 + ln(K·beta) ) / beta

Require that minimum to be at or below zero. beta is positive, so it drops out of the inequality:

g(a*) <= 0   <=>   1 + ln(K·beta) <= 0   <=>   ln(K·beta) <= -1   <=>   K·beta <= 1/e

        a fixed point exists   iff   K   <=  1 / (beta·e)
        the runaway branch is        K·beta  >  1/e  =  0.3679

What the threshold means

The threshold on K is 1/(beta·e), and it scales as 1/beta. Double the ranker’s sensitivity to affinity and you halve the accrual rate at which the system stops having a fixed point at all.

The e is the part people drop, and dropping it is not conservative. Writing the condition as beta · r_v · I / gamma > 1 — that is, K·beta > 1 instead of K·beta > 1/e — puts the threshold a factor of e = 2.718 too high. Every author whose K·beta lands between 0.368 and 1.0 gets called stable and is already on the runaway branch.

Above the real threshold, an author who gets slightly more share accrues more count, which buys more share, and the count has nothing to settle to. Nothing in Value opposes it, because Value is pointwise and this is a property of the whole slate.

The only thing that stops it is the author cap in the diversity pass (Diversity the constraint no pointwise score can express). That cap pins s_v <= k_a/25 = 3/25 = 0.12 no matter how large a_v grows, so the exponential right-hand side saturates and the runaway branch disappears.

That is a real bound and it is still a patch. It caps the damage at 12% of a user’s feed from one author, and it does nothing about the pressure that put the system against the cap in the first place.

Two fixes, and the first one is nearly free:

• Make affinity a rate, not a count. a_v = interactions_v / impressions_v, shrunk toward a prior. A rate is not mechanically increasing in exposure, and the fixed point dissolves.
• Floor the exploration. A rate is only estimable for authors who get impressions. A 4% exploration floor keeps every followed author’s rate estimate alive.

Both fixes measured, using the same effective-count reading as Narrowing measured — exp(entropy) is how many authors the viewer is effectively being shown:

author entropy of a viewer's impressions, 12 weeks

count-based affinity                3.90 -> 2.40 nats   (49 -> 11 effective authors)
rate-based affinity, 4% exploration 3.90 -> 3.50 nats   (49 -> 33 effective authors)

The code runs the fixed-point derivation numerically. has_fixed_point() applies the K·beta <= 1/e condition; settle() ignores the theory and just iterates the loop to see what happens. The last block is the point: every K the dropped-e rule calls safe runs away.

import math

BETA = 0.8                              # sensitivity of impression share to affinity
CRITICAL_K = 1.0 / (BETA * math.e)      # the derivation above, in one line


def has_fixed_point(K, beta=BETA):
    """Does `a = K*exp(beta*a)` have a solution at all?

    g(a) = K*exp(beta*a) - a is convex, so it has a root iff its minimum is
    <= 0, and that minimum sits at a* = -ln(K*beta)/beta. The whole condition
    collapses to K*beta <= 1/e. Dropping the `e` -- writing the threshold as
    K*beta <= 1 -- overstates the safe region by a factor of 2.718, so every
    author between the two thresholds is called stable and is not.
    """
    return K * beta <= 1.0 / math.e


def settle(K, beta=BETA, steps=200_000):
    """Run the loop and see. Returns the affinity it settles at, or inf."""
    a = 0.0
    for _ in range(steps):
        if beta * a > 700:              # exp() would overflow: it has run away
            return math.inf
        a = K * math.exp(beta * a)
    return a


print("beta %.2f   critical K = 1/(beta*e) = %.5f   wrong threshold 1/beta = %.5f"
      % (BETA, CRITICAL_K, 1 / BETA))
for K in (0.30, 0.45, CRITICAL_K, 0.46, 0.70, 1 / BETA):
    print("  K %.5f  K*beta %.5f  fixed point? %-5s  settles at %s"
          % (K, K * BETA, has_fixed_point(K), settle(K)))

assert abs(CRITICAL_K - 0.45985) < 1e-5
assert has_fixed_point(0.45) and math.isfinite(settle(0.45))
# at the threshold the curve is tangent to the diagonal, so the last surviving
# fixed point is a* = -ln(K*beta)/beta = 1/beta exactly.
assert has_fixed_point(CRITICAL_K) and abs(settle(CRITICAL_K) - 1 / BETA) < 1e-4
assert not has_fixed_point(0.46) and settle(0.46) == math.inf

# The whole point of the correction: every K in (1/(beta*e), 1/beta] is called
# STABLE by `K*beta > 1` and every one of them runs away.
for K in (0.46, 0.70, 1.00, 1 / BETA):
    assert K * BETA <= 1.0                     # the wrong rule says "no runaway"
    assert settle(K) == math.inf               # and it runs away regardless
assert abs((1 / BETA) / CRITICAL_K - math.e) < 1e-9      # the factor that was dropped

The general mechanism is ml 07 and its ranking form is quantified in The feedback loop quantified. The feed-specific part is that here the loop is closed by a feature definition rather than by the model — an affinity count is arithmetically increasing in exposure whether or not the model is any good, so this one is fixed in the feature store, not in training.

11.4 Degradation that writes itself into the model

An outage in a continually-retrained system does not end when the outage ends, because the outage writes itself into the training data.

Under load, the out-of-network retrieval call is the first thing to time out — it is the slowest of the four candidate sources (Scale and cost gives it 25 ms). The system falls back to in-network only.

That fallback is correct for the 27% of users with deep graphs, and catastrophic for the 18% with thin ones, who by The inventory is small and that inverts the retrieval problem need 75-90% of their feed from exactly the source that just disappeared. In one 40-minute incident the low-connectivity segment got 16 items instead of 25, sessions ending within 10 seconds went from 11% to 34%, and next-day return fell 1.8%.

The part that outlives the incident is the training data. Every impression logged during those 40 minutes was in-network, because that was all the system could serve. The next hourly update therefore trains on a distribution the system will never see again, and mis-ranks out-of-network content for the following hour. The outage lasted 40 minutes and the model damage lasted longer than the outage.

Two controls. Tag degraded requests at log time and exclude them from training. And when you must degrade, degrade the number of items, not the composition — 15 items with the normal mix beats 25 items from one source.

11.5 Producer starvation on the supply side

One failure never shows up in a viewer metric, because it consists of content that was never written.

A new author has no engagement history, so the author-level features are at their prior, so they rank low, so they get no impressions, so they never acquire a history. The loop closes on the first pass.

The middle row below is the same cohort with a 5% exploration allocation reserved for them. Compare it to the top row, not to the bottom one.

author cohort, first 30 days
                              impressions/post   authors with >= 1 interaction
new authors, no boost                    41                  38 %
new authors, 5 % exploration slot       210                  71 %
established authors                     890                  94 %

Thirty-eight percent means six out of ten new authors post into silence, and most of them stop. A 5% exploration allocation raises impressions per post 5x and nearly doubles the share of authors who get any response at all. This is a supply problem that presents as a ranking metric that looks fine — the feed’s engagement is unaffected, because the content that would have existed does not exist to be measured. The fix is an exploration allocation on the producer side, which is a different budget from the viewer-side exploration in The freshness trap your best features are missing when they matter most and is usually forgotten.

11.6 Summary

Every failure above on one page, in the mechanism-detection-control form.

12. Alternatives considered and rejected

Being able to say why you did not do the obvious thing is most of what separates a design from a description — so here are the designs that were genuinely on the table, and the number that removed each one.

13. Interviewer pushback

Eleven questions this design invites, each with the answer the chapter has already earned. The italic line under each question names what it is actually probing for.

“Just rank by predicted engagement. Why is this complicated?” Testing: whether the proxy critique is a slogan or arithmetic. Because I can show you the table. Bucket a week of impressions by content type and put the interaction rate next to a survey question. Engagement bait interacts at 0.143 against an inventory-weighted mean of 0.077 across the eight types in that table, and scores 11% on “worth your time” against a 44% mean. On any-interaction that is only 1.9x, but on the heads the ranker actually weights it is 9.1x on likes and 22x on comments (Engagement bait amplification) — the bait is tuned to the expensive actions, not to the average one. Political outrage reposts are 0.118 and 19%. The Spearman correlation between the engagement ranking and the satisfaction ranking across content types is -0.55. So an engagement-only objective has its optimum on the two worst rows, and those rows are only 8% of inventory today because the current ranker holds them down. That is not a slippery slope, it is where the argmax is.

“How is this different from ranking videos?” Testing: whether you can identify what is structurally new rather than restating a pattern. Three things. The candidate set is a graph query, not a catalog query — my median user has about 300 eligible posts in a 72-hour window, so the in-network half needs no approximate retrieval at all, and the retrieval problem is filling the deficit for the 18% of users whose whole network produces 16 posts a session. Second, the inventory turns over 33% a day against a video catalog’s 1%, so a nightly index rebuild makes 42% of the day’s engagement invisible and I need incremental inserts. Third, and this is the one that matters, the supply is endogenous: creators can see their own metrics and they optimize against my ranker within weeks. A video catalog does not rewrite itself in response to what I rank.

“Justify the survey head. You are spending real user attention for 0.0002% of the labels.” Testing: whether you can defend a cost with a number. Two arguments. First, the volume is sufficient because it is a head, not a model — it is a 33,000-parameter tower on a shared representation that 2 x 10^11 engagement labels already paid for, and 360,000 labels a day is 130 million a year, between 40 and 400 times what a head that size needs at 10-100 examples per parameter. Second, and this is the real argument: model true value as 0.3·engagement + 0.7·satisfaction with the two correlated at 0.35 at the item level, measured on the survey sample — not across content types, where the rank correlation is actually -0.55, which is the point of having the survey at all. Ranking by engagement alone correlates 0.639 with true value. Ranking with a survey head of reliability r crosses that at r ≈ 0.49, and a head on a shared representation lands at 0.55 to 0.65. So a noisy estimate of the right thing beats a perfect estimate of the wrong thing, and the extra 10^11 engagement labels buy me nothing because p_click was saturated long ago.

“You have eleven heads and a weight vector. Where do the weights come from?” Testing: whether the multi-objective score is a design or a gesture. Not from fitting — there is no per-item label for long-term welfare, so nothing to fit against. But before any of that, the weights have to be in a unit that means something, and the usual form is not. If I write weights on raw calibrated probabilities — survey: 0.70, comment: 0.55, reshare: 0.80 — and multiply each by its population base rate, the survey head is 91.7% of the score, the comment term is 0.70x the like term rather than 3.7x, and the whole negative block including report: 12.0 is 5% of the positive block. Every one of those probabilities is calibrated; calibration has nothing to say about it. The heads are on base rates from 0.44 down to 0.00015, and adding them is a units error. So I weight lifts: divide each head by its own base rate, clip at the head’s 99th percentile, and let the weights sum to 1.000 on the positive side and 0.400 on the negative. Now a weight literally is the share of an average post’s value that head carries, which is a thing a product owner can answer a question about. I elicit the shares, audit them by converting back to exchange rates — 0.150 on comment against 0.070 on like asserts one comment is worth 11.25 likes, and if the room thinks it is three, the shares are wrong — set the negative block as one number and split it, and then move at most two coordinates per long-term read-out, which at 20 read-outs a year is the entire budget. Anything I have never moved in an experiment is an opinion, and the config says so per line. One trap: the base rates are denominators, so re-measuring them re-scales every weight. I freeze them for the life of an experiment and review a re-baseline like a weight edit.

“Why not just pick the one objective that matters and maximize it?” Testing: whether multi-objective is a mechanical argument or a political one. Because every single term has a degenerate optimum and I can name each of them. Maximize clicks and you get curiosity gaps that resolve into nothing. Maximize 10-second dwell and you get slow-rendering, hard-to-parse posts. Maximize comments and you get maximum-disagreement content, because p(comment) and p(hide) peak on the same posts. Maximize reshares and you get moral outrage and unverified claims, the two highest-reshare classes there are. Maximize the survey head alone and you get a feed everyone endorses and nobody opens, plus a supply side that starves because nothing unusual is ever distributed. And the cleanest one: minimize the negatives alone and the global optimum is the empty feed — a blank slate has zero hides, zero unfollows and zero reports. So the blend is not diplomacy. It is a set of mutual constraints among terms whose individual argmaxes are each unshippable, which is also why the guardrails are independent launch blockers rather than more terms in the same sum.

“Push or pull? Give me the number.” Testing: whether you can derive the celebrity problem rather than recite it. Hybrid, and the threshold comes from RAM. Pure push: 500 M posts a day at 400 mean followers is 2 x 10^11 writes, which is fine on average and fatal in the tail — 5,000 head accounts at 4 M followers and 5 posts a day are 50% of all writes on 0.005% of the posts, and one post from a 100 M-follower account is 100 M writes, which at 3 M writes/s is 33 seconds of the entire platform’s write budget for one post. Pure pull: 8 B sessions times 200 authors is 1.6 x 10^12 fetches, eight times the operations and all of them on the critical path, and at a 20 ms p99 per fetch, 0.99^200 = 0.134 means 87% of loads hit a straggler. So: push the tail, pull the head. The head is about 50,000 accounts above a million followers; 50 recent posts each at 32 bytes is 80 MB, which fits in process memory on every feed host. That kills the network hop, so the straggler math disappears, and it removes 55% of writes and 100% of the bursts — after which the mean fanout of a pushed post is 180, right at the median follower count, which is how I know the threshold is in the right place. The reason the trade is that good is structural — the accounts that are expensive to push are exactly the ones many people follow, so their timelines are maximally cacheable.

“Your ranker keeps surfacing content that is almost-but-not-quite policy-violating. Why?” Testing: whether you understand the integrity coupling. Because interaction rate is monotone increasing in p_violating right up to the removal threshold — 0.038 in the cleanest bucket, 0.091 in the 0.70-0.85 bucket, a factor of 2.4 — and removal is a step function, so an engagement-maximizing ranker’s optimum sits pressed against the step from below. The fix is a continuous demotion below the removal line: d(p) = 1 - 0.9·clip((p-0.2)/0.65), which at p = 0.80 is 0.169, meaning that post must be 5.9x better on everything else to hold its slot. Compare 5.9x against the 2.4x engagement gradient it is fighting and the optimum moves off the boundary. That comparison is how you know the curve is strong enough. And demotion is the only way to spend a classifier at precision 0.14 — the false-positive cost of a demotion is about a tenth of a removal, so the break-even precision is roughly an order of magnitude lower.

“Engagement is up 5.4%, time spent up 2.2%, all significant. Ship it?” Testing: whether a green dashboard ends the conversation. No. On that same experiment the survey top-2-box dropped 2.2 points, hides rose 18%, “see fewer posts like this” rose 24%, and D28 was down 0.14% but underpowered at four weeks. Everything that reads out in a week is up and everything that requires asking a person or waiting a quarter is down. And the part the experiment could not see at all: six weeks after full launch, the share of inventory classified as engagement bait went from 2.1% to 5.6%, because creators optimize against the ranker their audience actually sees, not the one 2% of it sees. The supply response is structurally invisible to any experiment smaller than the population, which is why I would want a creator-cohort reverse holdout and a 1% long-term holdout before I called this a win.

“How do you know it is not just a filter bubble scare story?” Testing: whether you can instrument it. Because it is measurable and the measurement has a mechanism. Topic entropy of a user’s impressions falls from 3.41 to 2.28 nats over twelve weeks under a count-based affinity feature — 30 effective topics down to 10. The mechanism is that affinity is a decayed count of interactions, counts are mechanically increasing in exposure, and exposure is increasing in affinity. Write that as a fixed point: a proportional to r · exp(beta·a), which has a runaway branch. The fix is in the feature definition, not the model: make affinity a shrunk rate — interactions over impressions — which is not mechanically increasing in exposure, and floor exploration at 4% so the rate stays estimable for authors you currently suppress. With that, author entropy settles at 3.50 instead of 2.40.

“Where does the latency go, and what would you cut?” Testing: whether you have actually budgeted. 269 ms p50, and the two biggest items are not ML: 40 ms hydrating features for 600 candidates and 60 ms hydrating content payloads. That is 37% of the budget. The heavy ranker is 35 ms, the light ranker 18, out-of-network ANN 25. So I would batch and pipeline the payload hydration before I touched a model, because taking the ranker from 35 to 25 ms buys 3.7% and fixing hydration buys 15%. On cost, the ranker is 2.36 MFLOP per candidate, 600 candidates, 232 k/s peak — 330 TFLOP/s, about 3,000 accelerators. But FLOPs are not the constraint: 60 sparse features at 64 dims fp16 is 7.68 KB per candidate, 4.6 MB per request, 1.07 TB/s of random access at peak against a 38 GB author table that has to shard. The right move is to route each candidate to the shard holding its author row so the lookup is local and only 11 floats cross the wire.

“You have one engineer-quarter. Where does it go?” Testing: whether your analysis produces a priority. The survey pipeline, and specifically its coverage of low-connectivity users, because that is the 18% with the worst experience and the thinnest data. It is the only measurement of the objective, the breakeven reliability is 0.49 and the marginal return on more engagement labels is zero. Second would be topic and author entropy as first-class dashboard metrics with alerting, because Narrowing measured and The feedback loop formalized are cheap to detect and expensive to discover late. I would not spend it on the ranker architecture — a better shared bottom moves per-head AUC by thousandths, and none of the failures in this design are caused by the bottom being too small.

“Be honest: is this problem solved?” Testing: whether you will claim more than you can support. No, and I would not want to work with someone who said otherwise. Three things are genuinely open. The objective is contested by people acting in good faith, so the weight vector encodes a value judgement that no amount of data settles — and I can only evaluate it about twenty times a year, which means most of it is set by argument. The harms are long-term and the experiments are short-term, and the gap is not an experimental design flaw I can engineer away; the supply response literally does not occur at 2% treatment. And the feedback loop means the system’s training data is its own output, so every measurement I take is conditioned on the policy that produced it. What I can claim is that the failure modes have mechanisms, the mechanisms have detectors, and the detectors have numbers attached — topic entropy, survey delta, bait share of inventory, engagement rate by integrity bucket. That is a system you can argue about honestly, which is a lower bar than “solved” and a much higher one than most feeds clear.

Next: 11 — People You May Know.