04 — Video Search

A text-query video search engine: which evidence about a video to trust, how to turn spoken audio into a searchable text channel and what that costs, how to retrieve and rank inside a 200 millisecond budget, and how to measure a ranking change without running an experiment for a week. The aim is to be able to draw the whole system, name every model in it, state each model’s inputs and outputs, how it is trained and how it is served, and the single metric that shows it is working, with each choice justified by arithmetic.

The system in one line. A short piece of text typed by a person goes in; a ranked list of ten videos comes out, each carrying a timestamp to jump to when the query is about a specific moment inside a video.

This looks like a familiar search problem, but it is not: a web page is a document, and a video is five documents of very different quality attached to the same identifier.

The five documents — call them evidence channels from here on — are:

1. The title.
2. The description.
3. The transcript, produced by ASR — automatic speech recognition, a model that turns the spoken audio track into text.
4. The on-screen text, produced by OCR — optical character recognition, a model that reads words visible in the picture.
5. The visual track itself — the pixels, with no text at all.

Those five channels have five different accuracy profiles, five different arrival times after upload, and two different authors — an asymmetry that drives much of this chapter. Channels 1 and 2 are written by the creator, whose incentive is your traffic. Channels 3, 4 and 5 are derived from the content itself and are not under the creator’s control. (Tags are a sixth creator-written field, sized in Problem framing; they behave exactly like the title and description and are grouped with them throughout.)

Two terms borrowed from other chapters

Two ideas that other chapters derive at length are used below; both are restated in plain words at the point of use.

• NDCG — normalized discounted cumulative gain — is the standard score for “how good is this ranked list”. It is defined where it is first used in Offline metrics and derived in full in Ranking and recommendation metrics.
• An ANN index — approximate nearest neighbour — is the data structure that finds the stored vectors closest to a query vector without comparing against every one of them. It is sized in Scale and cost and its internals are derived in Vector index internals.

The general interview method this chapter follows — frame the problem, choose the metrics, then the architecture, then check it at scale — is laid out in 01 — Framework, and every step of it is worked in place below.

0. The models in this system, and what each is for

“The model” is never singular in a search design. This one contains eight distinct learned or measured components. Each row below is derived in the section it names.

One term in the last column is used repeatedly before it is derived. Recall@k is the fraction of the genuinely relevant items that appear anywhere in the top k results. It answers “was the right answer found at all”, which you have to ask before you ask “was it ranked first” — and that is exactly why a retrieval stage and a ranking stage need different numbers. Retrieval is judged on recall at a large k (did the candidate set contain the answer); ranking is judged on where the answer landed inside that set.

Offline versus online: where each component runs

Where each one runs matters as much as what it is, because it decides who pays for it.

Offline, once per video, at upload time: ASR, OCR, the visual dual encoder’s image tower, and the dense bi-encoder’s document side. Their output is written into an index and never recomputed when a query arrives.

Online, on every query, inside the 150 ms this system owns: the intent classifier, the query side of both encoders, the L1 ranker, and the L2 cross-encoder. That is four components, and they are cheap by construction.

That split is what makes the arithmetic in Scale and cost come out at roughly eight dollars a day of serving against millions of dollars of one-time indexing. It is the first thing to say when someone asks where the cost is.

The four assumptions holding this design up

Four assumptions carry real weight here. All four are checkable rather than matters of opinion, and if one turns out false a specific part of the design falls over. Know which part.

1. The creator is adversarial and the content is not. Two of the five channels are written by a party paid in your traffic. Clickbait and keyword stuffing is built entirely on that asymmetry. If creators were trustworthy, the corroboration gate would be pure cost with no benefit.
2. ASR errors are roughly independent across repeated occurrences of a term. This is what licenses the 1 - WER^k formula in Asr the highest value channel and how it fails, and therefore the whole argument about which terms survive transcription. If a decoder mangles a name the same way every time it hears it, that formula overstates survival badly.
3. A satisfied click approximates relevance. This is the label under both rankers and both encoders. Online metrics ab and position bias spends its whole length on the ways that approximation breaks.
4. The dense text bi-encoder is trained on the same click-mined pairs as the visual arm. This is the least load-bearing of the four, and it is an assumption rather than a result: this chapter derives the visual arm’s training data explicitly and does not derive the text arm’s. Treat that row of the table as design intent, not measured fact.

1. Problem framing

First fix what goes in, what comes out, how big the corpus is, and how long you have to answer — then measure how much evidence each of the five channels actually carries, which turns out to decide the entire indexing design.

What goes in is a text query, typically 1 to 40 characters, along with the searcher’s locale, their interface language, and their recent search history.

What comes out is a ranked list of videos — plus, when the query is about a moment inside a video rather than the video as a whole, a timestamp to jump to.

The corpus is 500 million eligible videos with a mean duration of 12 minutes, growing by 500,000 a day. Carry those three numbers; every capacity estimate in Scale and cost is built from them.

The latency constraint is 200 milliseconds at p95 for the whole call. p95 means the slowest 5% of calls are allowed to exceed it and the other 95% are not — it is a promise about the tail, not about the average. Of that 200 ms, roughly 150 ms belongs to search and the rest to rendering the surrounding page.

What makes it hard is that the thing the user is looking for is usually spoken, sometimes shown, and rarely written — and only the written part is cheap to index.

Sizing the evidence channels before choosing anything

Before choosing an index you have to know how much text each channel contains, because the answer differs by two orders of magnitude across the five.

Here is one median 12-minute video, channel by channel. The column to watch is unique terms — that is how many distinct words the channel actually contributes to a keyword index, since a word repeated ten times still only creates one entry in the dictionary. The right-hand column marks who wrote it.

title             8 words,     8 unique terms      creator-written
description     120 words,    70 unique terms      creator-written
tags             12 words,    12 unique terms      creator-written
ASR transcript 1,800 words,  610 unique terms      content-derived
on-screen OCR    140 words,   90 unique terms      content-derived
visual track     360 sampled frames                content-derived

Do the division: 610 / 8 = 76 and 610 / 70 = 8.7.

The transcript carries roughly 75x the lexical surface area of the title and 8.7x that of the description, and it is the only text channel the creator cannot write. That single ratio decides the indexing design.

Surface area is not signal — measure the posterior

Having lots of words is not the same as having trustworthy words. What you actually want to know is the posterior: the probability that a video really is about a term, given that the term showed up in a particular field. “Posterior” is just the Bayesian name for a probability computed after you have seen some evidence — here the evidence is “this word appeared in this field”.

That is a quantity you measure, not one you assume. The measurement is a 20,000-pair human-labeled sample of (video, term) pairs, where a rater answered one question: “is this video substantially about this term?”

The table below reports, for each field, the share of terms that earned a yes — and how many terms that field supplies per video. Look at both columns together; neither means much alone.

Two facts fall out of that table, and between them they are the whole retrieval design.

Fact 1: title has the best per-term precision and almost no coverage. Precision here is the share of a field’s terms that genuinely describe the video; coverage is how many terms the field supplies at all. The title gets 0.72 of eight terms right — accurate, but eight terms is nothing.

Fact 2: transcript has the coverage and terrible per-occurrence precision. It gets 0.09 of four hundred and thirty single-occurrence terms right — mostly noise, but it is the only field with enough terms to answer a specific query.

So you cannot pick one field, and you cannot treat them alike either. You need two things from your scorer.

You need per-field weights. The natural weight is the log-odds the table above already measures: the logarithm of p / (1 - p). Log-odds is just evidence strength rewritten on a scale where combining two pieces of evidence means adding two numbers instead of multiplying two probabilities. A positive log-odds means the evidence argues for relevance, a negative one means it argues against.

You need term-frequency saturation. That is a rule making the fifth mention of a word worth far less than the first, so a term repeated fifty times does not score fifty times higher than one mentioned once. Without it, repetition alone wins.

A scoring function that does exactly those two things already exists: BM25F, short for “Best Match 25, Fielded”. It is the standard lexical relevance formula from information retrieval, extended so a document can have several fields of differing trustworthiness. The point of arriving at it this way, rather than reaching for it as a default, is that the field boosts come out as measured quantities, not tuning knobs.

Here is that measurement turned into weights. The three log-odds values are computed from the posterior column of the table above; the last line converts the gap between two of them into a boost ratio.

log-odds(title)              = log(0.72 / 0.28)  =  +0.94
log-odds(transcript, k>=5)   = log(0.58 / 0.42)  =  +0.32
log-odds(transcript, k=1)    = log(0.09 / 0.91)  =  -2.31

boost ratio, title : transcript-dense  ≈  exp(0.94 - 0.32)  ≈  1.9

Read those four lines in order. The title’s log-odds is +0.94, a densely repeated transcript term’s is +0.32, and a term appearing exactly once in a transcript is -2.31. Because log-odds add, the boost ratio between two fields is the exponential of their difference: exp(0.94 - 0.32) = exp(0.62) = 1.86.

Two conclusions. The title should be boosted by roughly a factor of two over a densely repeated transcript term — not ten, not fifty. And a term appearing in a transcript exactly once should count for almost nothing, because -2.31 is strongly negative evidence, not weak positive evidence: seeing a word once in 1,800 spoken words actively argues the video is not about it.

Boosting the title tenfold “because it is the title” is a guess. The derivation above produces the number from measurement instead.

2. Which modality answers which query

Knowing which of the five channels actually answers each kind of query is what justifies spending money on speech recognition and on a visual index — and settling it introduces the two learned encoders that the visual arm is built from.

Start by measuring the traffic rather than guessing at it. Six query classes account for the whole mix, taken from a month of logs and hand-labeled on a sample of 5,000 queries.

The last column is recall@10 using text channels only. Read it as a pass rate: a recall@10 of 0.38 means the system fails to surface nearly two-thirds of the relevant videos in the space a user will actually look at.

The rows with the lowest recall are where the investment in this chapter goes.

Add up the transcript-dependent rows: 41% topical + 8% moment + 6% tail = 55%.

Fifty-five percent of queries are answered by a text channel that did not exist until you ran ASR, and another 14% are answered by a channel that has no text at all. That is the business case for both investments, and it also sets their very different budgets: the transcript pays for 55% of traffic, the visual index for 14%.

The visual case needs to be made precisely; “add CLIP embeddings” is not an argument.

Take the visual-descriptive slice and look at what the text channels actually contain for a video of a cat knocking a glass off a table:

title        "funny cat compilation #47"
description  "subscribe for more! follow me on ..."
transcript   [laughter]  "oh no"  [laughter]
OCR          (none)

Nobody ever says or writes what is happening. No amount of transcript quality reaches this query, because the information was never encoded in language by anyone in the first place.

That is a falsifiable claim, and falsifying it is cheap: sample 500 visual-descriptive queries and check whether the relevant videos contain the query terms anywhere in text. Measured, 71% of them do not. If that number had come back at 15%, the visual index would not be worth building and you would fix the text channels instead. Run the test before you spend the money.

What the visual arm is — name it, because “CLIP” is a checkpoint, not a design

“Add CLIP embeddings” names a downloadable checkpoint and skips every decision that matters. State the visual arm as a model: what it is, its inputs and outputs, how it is trained, how it is served, and how you know it works.

What it is. The arm is a dual encoder, meaning two separate neural networks whose outputs are designed to be comparable to each other. One is a vision tower: a ViT-B/16 image encoder — ViT is a vision transformer, the architecture that cuts an image into fixed patches (16×16 pixels here) and processes them the way a language model processes words — with roughly 86 million learned parameters, producing a 512-dimensional embedding for each frame. An embedding is just a list of numbers, 512 of them here, arranged so that things with similar meaning end up close together. The other is a text tower that turns a query string into an embedding of the same 512 numbers.

Inputs and outputs. At index time the vision tower takes sampled frames and emits one 512-d vector each; at serve time the text tower takes the query string and emits one 512-d vector. Similarity between a query and a frame is their cosine, the cosine of the angle between the two vectors, which is 1 when they point the same way and 0 when they are unrelated.

How it is trained, and why that is the load-bearing part. The two towers are trained contrastively. You build each training batch out of pairs that belong together — a frame and a phrase describing it — and then use an objective that pushes each frame toward its own caption and away from every other caption in the batch.

The objective has a name, InfoNCE. It is a cross-entropy loss in which the correct partner has to beat every other item in the batch, each of which acts as a negative example. A temperature of about 0.07 divides the similarity scores before the softmax; a small temperature sharpens the contrast, making the loss punish near-misses harder.

The shared space that training produces is the entire mechanism. At serve time the query never touches the vision tower and a frame never touches the text tower, so their cosine is meaningful only because the contrastive objective forced the two spaces to coincide. Train a frame encoder and a text encoder independently and you get two incomparable spaces, so the arm returns noise. “Just add CLIP embeddings” assumes that alignment for free and never says so. (CLIP is Contrastive Language-Image Pre-training, the published recipe this arm follows.)

Where the training pairs come from. Off-the-shelf CLIP is trained on web alt-text — the alt caption attribute of images on public web pages — and it underperforms on how people actually phrase video queries. So both towers are fine-tuned on in-domain pairs mined from your own search logs.

• Strong positives: (frame, query) pairs where that query earned a satisfied click on that video. A satisfied click is a click the user did not immediately bounce back from; the exact definition is in Online metrics ab and position bias.
• Weak positives: (frame, title) and (frame, on-screen text) pairs, which are plentiful but noisier.

The label here is a click, not a human judgement. That is the assumption doing the most work in this arm, and Online metrics ab and position bias is devoted to stress-testing it.

How it is served. Frames are encoded offline, once, when the video is uploaded, so at serve time the arm is a single forward pass of the text tower — about 2 ms — followed by an approximate-nearest-neighbour lookup against the stored frame vectors. That is how it fits inside a 9 ms budget on the 14% of queries that need it.

How you know it works. On the visual-descriptive slice, text-only recall@10 is 0.38 (the query-mix table above). Add the visual arm, fuse the two result lists, and it goes to 0.67 — recovering most of the 71% of relevant videos that name the answer in no text channel at all.

The fusion method is RRF, reciprocal rank fusion: score each result by 1/(rank + constant) in each list it appears in, then add those scores. RRF needs no comparable scores between the lists, only ranks, which is why it works across arms whose outputs are on different scales.

That 0.38 → 0.67 pair is the whole justification for a routing lane and for the roughly 1 TB segment tier Scale and cost sizes. Without an “after” number, the tier is unaccountable and the first cost-cutting review deletes it.

Routing: a cheap classifier decides which arms fire

The query mix decides a route, not a blend. You do not run every arm on every query — you run a tiny classifier first and let it turn arms on and off.

That classifier is a linear model costing 4 ms. Its inputs are character n-grams (every run of 3 to 5 consecutive characters in the query, which tolerates typos and needs no fixed vocabulary) plus a detector for quoted spans, since an explicit quotation is nearly always a moment query.

The diagram below is that routing decision. The top node is the classifier; the six branches are the six query classes from the table, each labelled with its share of traffic and with which retrieval arms it fires; all six then converge on one blend stage.

flowchart TD
    Q{"Query intent<br/>linear classifier · 4 ms"}
    Q -->|"Navigational · 22%"| N["Lexical only<br/>title/channel boost 3x<br/>skip visual arm"]
    Q -->|"Topical / how-to · 41%"| T["Lexical + dense<br/>transcript-weighted<br/>skip visual arm"]
    Q -->|"Moment / quote · 8%"| M["Lexical + quote index<br/>localization REQUIRED"]
    Q -->|"Visual-descriptive · 14%"| V["Visual ANN + dense<br/>lexical demoted"]
    Q -->|"Tail / conversational · 6%"| C["Dense-heavy<br/>query broadening on"]
    Q -->|"Entity + attribute · 9%"| E["All arms<br/>fusion by RRF"]

    N --> B["Blend · quality prior<br/>diversity · policy"]
    T --> B
    M --> B
    V --> B
    C --> B
    E --> B
    B --> R(["Ranked results"])

    style Q fill:#1d3557,color:#fff
    style T fill:#2d6a4f,color:#fff
    style V fill:#bc6c25,color:#fff

The two coloured lanes are the two that pay for a channel this chapter spends real money building. Topical how-to, in green, is the 41% of traffic that the ASR backfill of Asr the highest value channel and how it fails exists for. Visual descriptive, in orange, is the 14% that the visual index of Scale and cost exists for. The other four lanes are served by evidence the system would have had anyway, which is why they are uncoloured.

Now read the lanes one at a time. Each is a different answer to “which evidence do I trust for this query”.

• Navigational (22%) — someone naming a video they already know exists. Lexical retrieval only, with title and channel boosted threefold. Skips the visual arm entirely.
• Topical / how-to (41%) — answered by what someone said out loud. Lexical arm weighted toward the transcript, plus the dense text arm. Also skips the visual arm.
• Moment / quote (8%) — adds the quote index and requires localization. Returning the right video without the timestamp counts as a failure for this class.
• Visual descriptive (14%) — fires the visual ANN arm and the dense arm, with the lexical arm demoted, meaning its contribution is down-weighted rather than removed. Demoted rather than removed because for these queries the words in the video are actively misleading (“funny cat compilation #47”), but occasionally right.
• Tail / conversational (6%) — leans on the dense arm and switches on query broadening, which relaxes the requirement that every query term be matched.
• Entity + attribute (9%) — the one class where no single channel suffices, so all four retrieval arms fire and their four candidate lists are combined by RRF.

All six lanes converge on one blend stage. Blend applies a per-video quality prior — an estimate of how good this video is independent of any query — plus diversity rules that stop one channel from occupying the whole page, plus a policy filter, before the ranked results go out.

Add the lanes up before quoting a fan-out

The fan-out figures you quote later have to account for the entity-attribute lane, because it fires everything. Two arms therefore run on more traffic than their headline class:

visual arm   = visual-descriptive + entity-attribute  =  14% + 9%  =  23%
quote index  = moment-quote       + entity-attribute  =   8% + 9%  =  17%

So the visual arm fires on ~23% of queries and the quote index on ~17%. Both are affordable at those rates and would be prohibitive at 100%. That is the general lever behind any cheap-classifier-routes-to-unequal-lanes design: the expensive path is paid for by the classifier that keeps it off the common path.

3. Temporal granularity — the derivation people skip

What is a “document” in a video index — the whole video, or a 30-second slice of it? The obvious answer turns out to be wrong, for a reason that has nothing to do with storage.

The tempting design is to index 30-second segments, because it gives you deep links: a result that drops the user at 8 minutes 32 seconds rather than at the start. Deep links obviously help, so the question is what they cost, and both sides have to be priced.

Side 1: what segmentation buys. On moment queries, landing the user at the start of a 12-minute video forces them to scrub through it looking for the part they wanted.

Measured, that scrubbing is expensive. Compare the two rows below: the abandon rate falls by more than a factor of three and the satisfied-click rate rises by 28 points when the result lands at the right moment instead of at zero.

moment queries, video-level result, land at t=0:
    abandon within 20 s   38%      satisfied-click rate  0.41
moment queries, segment-level result, land at t=8:32:
    abandon within 20 s   11%      satisfied-click rate  0.69

Side 2: what segmentation costs, and it is not index size. The reflex objection is “that is 24 times the documents” — a 12-minute video cut into 30-second slices is 720 / 30 = 24 documents where there was one. That objection is nearly wrong, and knowing why is the point.

A lexical index is a set of postings: one entry per (term, document) occurrence, recording which document a word appeared in and where. The posting count here is unchanged by segmentation, because the same 1,800 transcript tokens get posted either way — just against 24 document identifiers instead of 1. The dictionary of distinct terms and the skip lists that let a scan jump ahead do grow, but compressed postings dominate the total, so the real increase is about 1.35x, not 24x.

The real cost is evidence fragmentation, and it is a recall cost rather than a storage cost.

Trace one query through both index designs. The five content terms of a repair query are spread across nine minutes of speech, which is fine for a whole-video document and fatal for a 30-second one:

query "civic rear brake pad torque spec"      5 content terms

video-level index:  all 5 terms appear somewhere in the 1,800-word transcript
                    -> conjunctive match, strong BM25F score
segment-level:      "civic" at 0:14, "brake pad" at 4:02, "torque spec" at 9:41
                    -> NO 30-second window contains all five
                    -> every segment is a weak partial match

A conjunctive match is one where every query term is present in the same document, and it is what a strong lexical score requires. Segmenting makes conjunctive matches impossible for long queries by construction.

The table below measures that. Each column is a query length; each cell is recall@100. Read across a row for stability and down a column for the damage: the whole-video row barely moves, the segment row collapses.

At 4+ terms the drop is 0.77 → 0.41, which means segmenting loses nearly half the relevant videos it would otherwise have found.

Segmenting trades recall for localization, and the recall it destroys is concentrated in exactly the long, specific queries that were the reason you built search. Overlapping the windows softens it — 0.41 → 0.55 — and costs postings, but it does not fix it, because a 4-term query spread over 9 minutes fits in no window of any size that is still short enough to be a useful deep link.

The resolution: localize at rank time, not at index time

The way out is to stop treating localization as an indexing decision at all, and to move it into the ranking stage, where you are looking at 100 documents instead of 500 million — five million times fewer.

Here is the sequence. Retrieve against whole videos, so no recall is lost. Run the L1 ranker — the cheap first-pass ranker of Retrieval architecture, which cuts about 1,700 candidates down to 100. Then, for those surviving 100 videos only, fetch each video’s segment payload and score its segments against the query, which you already have in hand at that point.

Localization is now a re-ranking operation over 100 documents rather than an indexing decision over 500 million.

Price it. The payload is small per video, and the fetch is small per query:

segment payload, per video:  24 segments × (start_ms, 40 top terms + positions)
                             ≈ 2.1 KB compressed
500 M videos × 2.1 KB        =  1.05 TB          keyed by video id, not searchable
fetch 100 payloads at rank time, 8-way parallel  ≈ 18 ms

You get the deep link without the recall loss, because the segment structure is a payload rather than an index. A payload is data stored under a key and fetched when you already know the key; an index is data you can search by content. The 1.05 TB of segment payloads is keyed by video identifier and is never searched, which is exactly why it costs recall nothing.

The one query class this does not serve is the moment query whose terms exist only inside one segment and are too weak to retrieve the whole video at all — quotations, mostly. Those get a genuine segment-level index, restricted to high-IDF n-grams: IDF is inverse document frequency, the standard measure of how rare and therefore how informative a term is, so a high-IDF n-gram is an uncommon phrase. Restricting the segment index to quoted spans of four or more tokens keeps it at roughly 3% of the postings a full segment index would need.

4. Retrieval architecture

The pieces now assemble into a serving path: four retrieval arms that produce candidates, two ranking models that narrow them, and a millisecond budget that forces the shape.

The diagram below is the whole request path, top to bottom. Follow the candidate count as you read down it — four arms fan out in parallel, then every stage after the merge narrows: ~1,700 candidates, then 100, then 10. Every box carries its latency in milliseconds.

flowchart TD
    Q(["Query"]) --> QU["Query understanding<br/>spell · segmentation<br/>intent class · language<br/>8 ms"]
    QU --> LEX["Lexical BM25F<br/>title · desc · tags<br/>transcript · OCR<br/>12 shards · 25 ms"]
    QU --> DEN["Dense transcript ANN<br/>768-d text emb · HNSW<br/>see RAG ch 6 §4 · 12 ms"]
    QU --> VIS["Visual ANN<br/>pooled frame emb<br/>visual + entity-attr · 23% · 9 ms"]
    QU --> QUO["Quote index<br/>4-gram spans<br/>moment + entity-attr · 17%"]

    LEX --> MRG["Merge · dedupe by<br/>content hash + channel cap<br/>~1,700 candidates · 3 ms"]
    DEN --> MRG
    VIS --> MRG
    QUO --> MRG

    MRG --> L1["L1 ranker · GBDT<br/>190 cheap features<br/>1,700 -> 100 · 15 ms"]
    L1 --> SEG["Segment payload fetch<br/>+ localization scoring<br/>100 videos · 18 ms"]
    SEG --> L2["L2 cross-encoder<br/>query × title+best segments<br/>6 layers · 100 pairs · 11 ms"]
    L2 --> BLEND["Blend · quality prior<br/>freshness · diversity<br/>policy filter · 4 ms"]
    BLEND --> OUT(["10 results<br/>+ deep-link timestamps"])

    style LEX fill:#1d3557,color:#fff
    style L1 fill:#bc6c25,color:#fff
    style L2 fill:#2d6a4f,color:#fff
    style SEG fill:#7209b7,color:#fff

The four coloured boxes are the four stages that own a budget line worth arguing about, and each colour marks a different kind of bottleneck.

• Lexical retrieval, blue — the widest fan-out and the slowest single arm at 25 ms.
• L1, orange — makes the biggest cut, 1,700 down to 100.
• Segment fetch, purple — the only I/O-bound stage in the path: 18 ms spent waiting on 100 key-value reads, with almost no arithmetic in it.
• L2, green — the only GPU-bound stage: 603 GFLOP of dense matrix work, with almost no waiting in it.

Those last two sit next to each other in the diagram and are near opposites, which is why they do not share a colour. The lever that shortens the segment fetch — issue more reads in parallel — does nothing at all for L2, and the lever that shortens L2 — a bigger GPU — does nothing for the fetch. Confusing the two is how teams buy the wrong hardware.

Walking the serving path

Several of those boxes carry a design decision rather than plumbing, so walk the path once in words.

Query understanding (8 ms) does four cheap things: spelling correction; segmentation, meaning splitting a run-together query such as brakepads into two words; the intent classification of Which modality answers which query; and language detection.

Four retrieval arms then run in parallel, so the cost of the group is the cost of its slowest member, not the sum.

• The lexical BM25F arm scores the query against all five fields — title, description, tags, transcript, on-screen text — across 12 shards. Sharding means the index is cut into 12 pieces held on 12 machines that are queried at the same time, so the arm is as slow as its slowest shard: 25 ms.
• The dense transcript arm looks up the query’s 768-dimensional text embedding in an HNSW index. HNSW is hierarchical navigable small world, the most common approximate-nearest-neighbour structure. It stores the vectors as a graph with long-range links at the top and short-range links at the bottom, so a search hops coarsely toward the right region and then refines. Its ef_search parameter, set to 200 in the budget below, is how many candidates the search keeps alive while walking the graph — larger is more accurate and slower.
• The visual arm and the quote index fire only when the intent classifier says so, on the 23% and 17% of traffic derived in Which modality answers which query.

Merge (3 ms) does two jobs beyond concatenating the four lists. It removes duplicates by content hash, so the same media re-uploaded twice does not take two slots. And it applies a channel cap, so one prolific creator cannot fill the page. About 1,700 candidates survive.

L1 (15 ms) is a gradient-boosted decision tree ensemble that scores all 1,700 candidates on 190 deliberately cheap features — field match counts, video age, click priors, all of it already in memory, none of it requiring a document fetch. It keeps the best 100.

Segment payload fetch (18 ms) happens only now, for those 100 videos, and produces the deep-link timestamp. This ordering is the whole point of Temporal granularity the derivation people skip.

L2 (11 ms) re-scores those 100 with a cross-encoder, and blend (4 ms) applies the quality prior, freshness, diversity and policy filtering before ten results go out.

The latency budget, added up

The budget below is the constraint that makes every “why not just do the expensive thing everywhere” question answer itself, so add it up explicitly rather than eyeballing it.

The key to reading it is which lines are parallel and which are serial. Parallel lines run at the same time as another line, so only the largest of the group counts. Serial lines run one after another, so they all add. Here the three retrieval arms are parallel with each other; everything else is serial. p50 means the median call.

query understanding                                      8 ms
lexical retrieval, 12 shards in parallel, slowest       25 ms
dense ANN, HNSW ef_search=200                           12 ms   (parallel with lexical)
visual ANN, ~23% of queries                              9 ms   (parallel)
merge + dedupe                                           3 ms
L1 GBDT, 1,700 × 190 features, 400 trees                15 ms
segment payload fetch + localization                    18 ms
L2 cross-encoder                                        11 ms
blend + policy                                           4 ms
                                                       ------
                                            p50         84 ms   (serial path only)
                                            p95        140 ms

The 84 ms is not the sum of all nine lines — the dense and visual arms hide inside the lexical arm’s 25 ms. The serial path is:

8 + 25 + 3 + 15 + 18 + 11 + 4  =  84 ms

That leaves 66 ms of headroom against the 150 ms search budget, which is what absorbs the tail: a slow shard, a cold page cache, a GPU queued behind another request. Hence p95 at 140 ms rather than 84.

The two rankers, stated as models

Before pricing L2, state both rankers plainly, since between them they decide the final page.

Inputs and outputs. L1 takes cheap in-memory features for all ~1,700 candidates and emits a score. L2 takes the actual text of 100 (query, document) pairs and emits a much better score.

How they are trained. Both on the same labels — satisfied clicks, corrected for position bias (Online metrics ab and position bias). That shared label is what makes the pair coherent: both approximate the same target, and they differ only in how much evidence each is allowed to look at.

How they are served. Both online, on every query.

How you know each works. L2 by NDCG@10. L1 by something different: whether the 100 candidates it keeps still contain what L2 would have ranked on top. That is the stage-wise attribution Offline metrics insists on, and it is the only way to catch L1 quietly throwing away the right answer before L2 ever sees it.

Deriving L2’s 11 ms

The 11 ms allowed for L2 is the tightest line in the budget, so derive it rather than quoting it.

L2 is a cross-encoder. The bi-encoder arms embed query and document separately and compare the two vectors, which means the document side can be precomputed offline. A cross-encoder instead concatenates query and document into one sequence and runs a transformer over both together, so every query word can attend to every document word. That is why it is far more accurate — and far more expensive, because nothing can be precomputed.

The configuration: 6 transformer layers, hidden width d = 384 (the length of the vector carried per token), T = 256 tokens per query-document pair, and 100 pairs per request. A FLOP is one floating-point operation; a GFLOP is a billion of them and a TFLOP a trillion.

The cost has two parts, computed below: the matrix multiplications inside each layer, and the attention comparisons between tokens.

params per layer (matmuls)  =  12 d^2  =  12 × 384^2  =  1.77 M
6 layers                                              = 10.6 M
matmul FLOPs   = 2 · params · tokens · pairs = 2 × 10.6e6 × 256 × 100  = 543 GFLOP
attention term = 6 · 4 · T^2 · d · pairs     = 6 × 4 × 256^2 × 384 × 100 =  60 GFLOP
                                                                          ---------
                                                                          603 GFLOP
on one A10G at ~60 TFLOP/s effective                                      = 10.1 ms

Three things to notice in that derivation.

The 2 · params · tokens shape is the standard rule for a forward pass: each parameter is used once in a multiply and once in an add, so a matrix multiply costs two FLOPs per parameter per token. The matmul term grows linearly with sequence length.

The attention term grows with the square of the sequence length, because attention compares every token with every other token — T^2 = 256^2. At 256 tokens it is only 60 of the 603 GFLOP, so matmuls dominate here. Double the sequence length and that balance shifts.

The hardware line. An A10G is a mid-range inference graphics card. “60 TFLOP/s effective” means 60 trillion floating-point operations per second actually achieved in practice, which is well below the datasheet peak because real workloads never keep every unit busy. Dividing: 603 GFLOP / 60 TFLOP/s = 0.0101 s = 10.1 ms, which is where the 11 ms budget line comes from.

The cross-encoder is affordable only because it runs on 100 documents. Run the same model over all 1,700 merged candidates and you get 603 GFLOP × 17 = 10.3 TFLOP, which is 10.3 / 60 = 0.172 s = 172 ms — more than the entire 150 ms budget, spent on one stage.

That is the two-stage retrieval argument in its cheapest form: a cheap model to get from thousands to hundreds, an expensive model to order the hundreds. It is the same shape as the reranking discussion in Reranking in the serving path.

Why lexical retrieval does not go away

Why keep a keyword index at all in a system that already has two embedding arms? The defense is a specific class of query, not a general preference for redundancy.

First, name the dense arm precisely, because this chapter has two dense arms and they are not the same thing.

This one is a text bi-encoder over the transcript and title. “Bi-encoder” means one encoder applied twice and independently — once to the document, once to the query — so the document side can be computed offline and stored. It produces a 768-dimensional embedding of the words in the video, encoded once at index time; the query is embedded by the same tower at serve time and matched by cosine similarity.

This arm has never seen a pixel. The visual dual encoder in What the visual arm is name it because clip is a checkpoint not a design is the only arm that touches frames. Keep those separate or half the fan-out arithmetic stops making sense.

With that straight: dense-only retrieval is tempting, and wrong here, for a reason specific to video. The example below is one query where the two arms give completely different answers, and the comment explains the mechanism:

query   "torx t25 vs t27"
dense   returns general "screwdriver bits" and "socket set" videos — the
        embedding does not separate t25 from t27, because a lossy 768-d
        compression of a 1,800-word transcript keeps topic and drops one digit
lexical returns the video whose transcript says "t27" eleven times

Long-tail exact tokens — part numbers, model years, error codes, proper nouns — are precisely the terms an embedding compresses away, and they are over-represented in video search because video is where people go for repair, gaming, and product content. Hybrid retrieval is not hedging; the two arms answer disjoint query classes. Fuse with reciprocal-rank fusion at merge time rather than trying to calibrate a BM25 score against a cosine — those two numbers live on incomparable scales and any linear blend of them is a hyperparameter you will retune every index build.

5. ASR: the highest-value channel, and how it fails

Automatic speech recognition is the channel that answers most of the traffic. It deserves three hard looks: what it costs, why its headline accuracy metric is the wrong one for search, and how its errors land unevenly across speakers in a way that is a fairness problem rather than a rounding error.

Cost, derived

Transcribing every video in the corpus is the largest single bill in this design, so derive it rather than waving at it.

Two units first. A GPU-hour is one hour of one graphics processor — the thing you rent. “120x realtime” means one such processor transcribes 120 hours of audio in one hour of wall-clock time. That multiplier is achievable only because the work is batched: many audio streams are processed together so the hardware is never idle waiting on one stream.

The backfill is the one-time job of transcribing everything that already exists:

backfill: 500 M videos × 12 min          =  100 M audio-hours
distilled ASR, batched, ~120x realtime   =  100e6 / 120  =  833,000 GPU-hours
833,000 GPU-hours × $2.50/GPU-hour       =  $2.08 M      one-time

Check the first line: 500e6 × 12 min = 6,000e6 min, and 6,000e6 / 60 = 100e6 hours. That is the whole corpus expressed as audio time.

Two things about that derivation, because both are assumptions and neither is a FLOP count.

The 833,000 GPU-hours comes entirely from the throughput figure. A distilled ASR model — one trained to imitate a larger, more accurate model at a fraction of the cost — running at about 120x realtime on a batched offline pipeline. No operations-per-second rate enters this derivation at all, so the number an interviewer should challenge is the 120x, not some arithmetic rate.

The $2.50 per GPU-hour is deliberately not the serving rate. It is the rate for a datacenter-class, training-grade accelerator run as an offline batch job. The cross-encoder in Retrieval architecture and Scale and cost is priced on an A10G at $0.75/hr, a cheaper inference-class card. The backfill wants throughput per hour and can afford the expensive card because it runs once; serving wants cost per query and takes the cheap card. Quoting one rate for both is the mistake.

Steady state is the recurring job of transcribing new uploads, and it uses the same two numbers on a much smaller input:

steady state: 500 k new videos/day × 12 min  =  100,000 audio-hours/day
                                             =  833 GPU-hours/day
              833 × $2.50/GPU-hour           =  $2,083/day
              $2,083 / 500 k videos          =  $0.0042/video

The backfill is a budget line item and the steady state is a rounding error — under half a cent per video. That is the normal shape for a derived channel, and it is why the decision is “do it once, properly” rather than “roll it out gradually to control cost”.

One sensitivity worth carrying, because it is the largest lever on the largest number. Unbatched, the same model runs at roughly 20x realtime instead of 120x:

unbatched: 100e6 / 20  =  5.0 M GPU-hours  ×  $2.50  =  $12.5 M
batched:                  833 k GPU-hours          =  $2.08 M
                                                      ---------
                          12.5 / 2.08  =  6x

Batching is a 6x lever on an eight-figure number, so it belongs in the answer rather than in an implementation footnote.

Why WER does not translate linearly into retrieval loss

A speech-team metric has to be converted into a search-team metric here, and the conversion is the reason the two teams should not be reporting the same number.

WER is word error rate: the fraction of spoken words the transcript gets wrong, counting substitutions, deletions and insertions. An aggregate WER of 12% sounds like it should cost roughly 12% of retrieval. It costs far less on most terms and far more on the terms that matter, and the mechanism is repetition.

Here is the mechanism. A topical term appears k times in a transcript, and the transcript only has to get it right once for the term to be searchable — one correct posting is enough to make the video findable by that word. So the term is lost only if ASR botches every occurrence.

If errors are independent across occurrences, the probability of botching all k of them is WER^k, and therefore:

P(term survives)  =  1 - WER_term^k

Independence is an assumption worth stating explicitly, because the whole formula rests on it and it fails for a decoder that mishears the same name identically every time it hears it. In that case the true survival probability is closer to 1 - WER regardless of k, and everything below is optimistic.

The table applies the formula to five kinds of term. Watch the interaction between the two middle columns: the terms with high error rates are also the terms that get said only once or twice, so the two effects compound rather than cancel.

Work two rows by hand so the pattern is concrete.

common topical noun:  1 - 0.07^9  =  1 - 0.0000000000403  ≈  1.0
product / model name: 1 - 0.34^2  =  1 - 0.1156           =  0.884  -> lost 11.6%
person named once:    1 - 0.41^1  =  1 - 0.41             =  0.590  -> lost 41%

Nine repetitions at a 7% error rate makes loss essentially impossible. One utterance at a 41% error rate makes loss a coin flip.

Repetition rescues exactly the low-IDF terms you did not need and abandons exactly the high-IDF terms the query depends on. (IDF is defined in The resolution localize at rank time not at index time.) A query is usually anchored by one rare term, and losing that anchor means the video is never retrieved at all. The failure is total, not graded — you do not get a slightly worse ranking, you get nothing.

That is why aggregate WER is the wrong ASR metric for search. The right one is entity WER at the video level: the fraction of videos where at least one named entity present in the audio is absent from the transcript. It is measured at 0.37 for this corpus, which is the share-weighted average of the accent table below, and it is the number the search team should be reporting back to the speech team.

One sanity check on that aggregate: it cannot be lower than its best group, 0.26. An aggregate that lands below every group it averages is a reporting bug, not a good result.

Three fixes, cheapest first

Three things reduce entity loss without training a better ASR model. They are listed in increasing order of cost.

a) Contextual biasing. The title, the description, the tags, and the vocabulary of the channel’s past videos are all available before the audio is decoded. Feeding them to the decoder as a shallow-fusion bias — nudging the probability the decoder assigns to each candidate word at each step, without retraining anything — costs about 2% of ASR throughput and cuts entity WER by roughly 38% in relative terms: from 0.34 to 0.21 on product and model names, and from 0.37 to 0.23 at the corpus level. The creator already told you the hard words; the decoder just was not listening.

b) Lattice or n-best indexing. An ASR decoder does not really produce one string; it produces a ranked set of competing hypotheses (an “n-best list”, or a lattice if you keep the graph of alternatives rather than flattening it) and then throws all but the top one away. Instead of indexing only that single best string, index the top three hypotheses for each span, with each term’s frequency weighted by the decoder’s confidence in it. Postings on the transcript field grow 2.4x, and entity recall@100 rises by 6.1 points. That trade is worth taking as long as transcript postings are not already the storage bottleneck, which Scale and cost sizes.

c) Phonetic fallback in the query path. Store a double-metaphone key alongside each term — a short code that captures roughly how a word sounds rather than how it is spelled, so that words pronounced alike collide on purpose — and when a query returns nothing at all, retry it against those keys. That catches a user typing "navara" against a transcript that heard "navarro". The index cost is near zero, and because the retry is restricted to the zero-result path it cannot damage precision on queries that were already working.

The accent problem, which is a fairness problem

The errors above are not spread evenly across speakers, and an accuracy gap in a speech model turns into an income gap for a group of creators.

The rows below are clusters of speaker accent. AAVE is African-American Vernacular English, a dialect general-purpose speech models are consistently worse at. Read the middle column — entity WER — and compare the best row to the worst; the right-hand column is what share of the corpus sits in each row.

speaker accent group     WER     entity WER    videos affected
US general               0.081      0.26           41%
UK / IE                  0.094      0.29           11%
Indian English           0.163      0.44           14%
Nigerian English         0.178      0.47            4%
AAVE-leaning             0.171      0.45            7%
Non-native, other        0.192      0.51           23%

The raw gap is 0.192 / 0.081 = 2.4x in WER between the best and worst groups.

A 2.4x WER gap becomes a discoverability gap, and discoverability is income for these creators. The compounding is worse than the ratio suggests, because of the 1 - WER^k curve above. At k = 1 — a name said once, which is the common case — entity loss goes from 0.26 to 0.51. More than half the named entities in an affected creator’s video are simply not searchable.

This will not show up in any aggregate search metric. Those creators are a minority of the corpus, so aggregate NDCG (the ranking quality score defined in Offline metrics) moves about 0.3 points — indistinguishable from noise. It shows up only when you slice.

Report retrieval recall by speaker-accent cluster as a standing metric, in the same table as aggregate NDCG, or the regression is invisible by construction. Same argument as slicing in ml/09 Step 3.

6. Offline metrics

Start with the numbers you can compute without shipping anything.

The primary metric needs defining before it is used. NDCG@10 stands for normalized discounted cumulative gain at 10, and its four words are four steps:

1. Gain — each result gets a score based on how relevant a human rater graded it, here on a 0-to-4 scale.
2. Discounted — divide each gain by a discount that grows with the result’s position, so a hit at rank 1 counts for more than the same hit at rank 9.
3. Cumulative — sum those discounted gains over the top ten results.
4. Normalized — divide the sum by what the best possible ordering of the same results would have scored.

The output is 1.0 for a perfect page and near 0 for a useless one. That fourth step, normalization, is where two of this chapter’s traps live, and The long tail with no good result works both out in numbers.

No single metric here is safe to gate on alone, which is what the table shows. Read the right-hand column first — each row’s trap is the reason the row below it exists.

The full derivation of that position discount — why it falls off logarithmically with rank rather than at some steeper rate — is in Ndcg with the discount derived. Two things about it are specific to video:

Graded relevance must include a duration-aware grade. A 3-hour livestream that contains the answer at 1:47:00 is not the same result as a 4-minute video about exactly that. If the rater guideline does not say so, raters will grade on topicality alone and your NDCG will happily promote the livestream. Add “effort to reach the answer” to the rubric explicitly.

Stage-wise attribution or you will optimize the wrong stage. Report recall@1000 (retrieval), NDCG@10 given the retrieved set (ranking), and end-to-end NDCG@10. If end-to-end is flat while ranking-given-retrieval improved, retrieval regressed and the ranker absorbed it — a real and common failure that a single number hides.

7. Online metrics, A/B, and position bias

Once real users are involved the measurements change, and they end at the reason ranking teams almost never use a plain A/B test — an experiment that splits users into two arms and compares an average — to compare two rankers.

The metric definitions that matter

Every online metric here is a definition you choose, not a quantity sitting in the logs waiting to be read. Write the definitions down before arguing about the numbers, because most disagreements about online metrics turn out to be disagreements about the definition.

The four definitions below are the ones this chapter uses. Note that “satisfied click” carries three conditions joined by AND, and the third one — no return to the results page — is what separates a good result from a misleading thumbnail.

click                 any result click
satisfied click       click AND (watch >= 30 s OR watch >= 30% of duration)
                      AND no return-to-results within 60 s
good abandonment      no click, no reformulation, no re-query in 10 min
                      -> the answer was in the snippet or the user gave up; ambiguous
reformulation         a new query within 60 s sharing >= 1 content term

Satisfied-click rate is the primary metric; reformulation rate is the sharpest negative one. Raw CTR — click-through rate, the share of impressions that get any click at all — is actively misleading here, because clickbait raises it by construction, as Clickbait and keyword stuffing shows. So CTR stays as a diagnostic and never becomes a goal.

Watch time deserves the warning it gets in 06 — Video Recommendation: total watch time from search rewards long videos mechanically. Use watch fraction thresholds for satisfaction and keep total watch time as a guardrail only.

Position bias, and why it makes A/B underpowered

A straightforward A/B test cannot practically decide between two rankers, and deriving why points directly at what to use instead.

An observed click is not evidence of relevance on its own. It is relevance gated by whether the user looked at that rank at all, and the two factors multiply:

P(click | result r, rank j)  =  examination(j) × relevance(r)

examination(j) is the probability the user’s eye reached rank j; relevance(r) is the probability they would click the result if they saw it. You observe only the product.

The examination term is measurable, and the measurement is a cheap experiment: swap the results at two positions for 1% of traffic and watch whether the click rate follows the position or the result. Whatever follows the position is examination.

That experiment gives the curve below. Read it as “probability the user’s attention got this far down the page”:

rank   1      2      3      4      5      6      8     10
exam  0.72   0.51   0.39   0.31   0.26   0.22   0.17  0.13

Rank 1 is examined 0.72 of the time and rank 10 only 0.13 — a 5.5x falloff. Two consequences follow, and together they are why ranking experiments are not run as plain A/Bs.

Consequence 1: most of the click mass is decided by rank 1

Two systems that differ only on ranks 3 through 10 produce nearly identical click-through rates, because the ranks they differ on are barely examined. The experiment ends up measuring the position-1 difference plus noise.

Size the experiment and see how bad that gets. Three terms first:

• MDE, the minimum detectable effect: the smallest true difference the experiment can reliably distinguish from zero.
• Alpha of 0.05: the false-positive rate you tolerate — a 5% chance of declaring a difference that is not there.
• Power of 0.8: an 80% chance of detecting a difference that really is there.

The standard sample-size formula for comparing two proportions at those settings is n ≈ 16 · p(1-p) / delta^2, where p is the baseline rate and delta the effect you want to detect. The 16 is what alpha 0.05 and power 0.8 work out to; change either and that constant changes.

baseline satisfied-click rate p = 0.44
MDE 0.5 points absolute  ->  delta = 0.005,  delta^2 = 0.000025

n per arm  =  16 × 0.44 × 0.56 / 0.000025
           =  3.9424 / 0.000025
           =  157,700 queries

At 1 M queries/day that is fine — it reads out inside a day.

But 0.5 points is a big effect. The MDE you actually need for a ranking change that touches ranks 3-10 is closer to 0.1 points, and sample size scales with 1/delta^2, so cutting the MDE by 5x multiplies the sample by 25:

delta = 0.001,  delta^2 = 0.000001
n per arm  =  3.9424 / 0.000001  =  3,942,400  ≈  3.9 M queries

Now you are running for a week per experiment and fighting seasonality — weekday and weekend traffic are not the same population.

Consequence 2: interleaving removes the confound instead of estimating it

Team-draft interleaving builds one result list by alternating picks from the two rankers, then attributes each click to whichever ranker contributed that result.

The mechanism is the whole point: because both rankers contribute results at comparable positions, examination(j) is the same for both and cancels in the comparison. You never have to estimate it. Compare that to an A/B, where each ranker owns its own page and the position effect has to be measured out of the noise.

The payoff in sample size:

same decision (0.1-point MDE), team-draft interleaving:
    sessions needed for the same power   ≈  6,000 - 15,000
    speedup over the A/B above           =  3.9e6 / 15,000  =  260x
                                            3.9e6 /  6,000  =  650x

for reference, the easier 0.5-point decision:
                                            157,700 / 15,000  =  11x
                                            157,700 /  6,000  =  26x

Quote the speedup against the decision you are actually making. The often-repeated “10 to 100x” is the 0.5-point band — and at a 0.5-point MDE an A/B was already affordable, since 157,700 per arm reads out in a day. The whole reason interleaving is on the table is the 0.1-point decision, where the A/B needs 3.9 M per arm and the factor is 260x to 650x. Understating your own case by 5-25x is a strange way to win an argument.

Two functions implement this. The first builds the interleaved list; the thing to look at is the coin flip at the top of each round, which is what keeps the position distributions fair. The second decides who won a session, and its docstring names the statistical bug that makes naive implementations report false wins.

import random


def team_draft(list_a, list_b, k=10, rng=random.Random(0)):
    """Interleave two ranked lists and record which ranker contributed each slot.

    Both rankers end up occupying comparable position distributions, so the
    examination term cancels in the click comparison instead of having to be
    estimated. The coin flip per round is what makes the position
    distributions exchangeable; without it the first list wins rank 1 every
    time and you have rebuilt the bias you were removing.
    """
    cursor = {"A": 0, "B": 0}
    lists = {"A": list_a, "B": list_b}
    seen, blend, credit = set(), [], []
    while len(blend) < k and any(cursor[n] < len(lists[n]) for n in lists):
        order = ["A", "B"] if rng.random() < 0.5 else ["B", "A"]
        for name in order:
            src, i = lists[name], cursor[name]
            while i < len(src) and src[i] in seen:
                i += 1
            if i < len(src) and len(blend) < k:
                seen.add(src[i]); blend.append(src[i]); credit.append(name); i += 1
            cursor[name] = i
    return blend, credit


def session_outcome(credit, clicked_ranks):
    """A session votes for A, for B, or ties. Sessions are the unit of
    inference, never clicks: counting a six-click session as six observations
    inflates significance (math/02, randomization units)."""
    a = sum(1 for r in clicked_ranks if credit[r] == "A")
    b = sum(1 for r in clicked_ranks if credit[r] == "B")
    return (a > b) - (a < b)

The unit of inference matters as much as the interleaving; the variance bug from choosing it wrong is derived in Randomization unit and the variance bug it causes.

Interleaving is the right default for ranking changes and the wrong tool for anything else — it cannot measure a UI change, cannot measure a change in the number of results, and cannot measure long-term effects, because both arms are inside the same session. Ship-decision A/B still runs afterward; interleaving is how you get from 40 candidate rankers to 3.

8. Scale and cost

Size everything the system has to keep in memory and everything it burns per query, and the two turn out not to be remotely the same order of magnitude — which changes what you should optimize.

Index sizes, derived

Storage is what this system actually spends money on, so derive every tier rather than quoting a total.

Three terms used in the blocks below. Delta + varint is the standard posting-list compression: store the difference between consecutive document identifiers rather than the identifiers themselves, then encode each difference in as few bytes as it needs — small differences take one byte. fp16 means each number is stored in 16 bits, int8 in 8 bits, so an int8 vector is a quarter the size of a 32-bit one at some cost in precision.

First the lexical tier — everything keyword search needs. Every line is count × size; the totals at 3.6 bytes per compressed posting are what to check:

transcript postings   500 M videos × 1,800 tokens          =  900 B postings
                      compressed (delta + varint) ~3.6 B/posting
                                                            =  3.24 TB
title/desc/tags/OCR   500 M × 280 tokens = 140 B postings   =  0.50 TB
                      (8 title + 120 description + 12 tags + 140 OCR, §1)
n-best expansion (if adopted, ×2.4 on transcript)           = +4.54 TB
segment payloads      500 M × 2.1 KB                        =  1.05 TB
quote index, 4-gram spans, high-IDF only                    =  0.31 TB
                                                              --------
lexical tier, 1-best                                        ≈  5.10 TB

The 5.10 TB is 3.24 + 0.50 + 1.05 + 0.31. The n-best line is not in that total — it is a proposal from Asr the highest value channel and how it fails that gets priced in “The index, priced” below and is not part of the shipping design.

Now the vector tier — everything the two embedding arms need:

dense text embeddings    500 M × 768-d × 2 B (fp16)         =  768 GB
                      (transcript + title, §4 — not the pixels)
HNSW graph overhead      see RAG ch 6 §4.3 for the derivation; at M=32
                         layer-0 links 2M × 4 B = 256 B, upper layers
                         (1/(M-1))·M·4 ≈ 4 B, bookkeeping 16 B
                         -> 1,812 B/vector, i.e. +18% not 2x   ≈  0.91 TB
visual pooled embeddings 500 M × 512-d × 1 B (int8)         =  256 GB
                                                              --------
vector tier                                                 ≈  1.16 TB

The HNSW block deserves a second read, because it contradicts a rule of thumb you will hear repeated.

The graph overhead here is a fraction of the payload, not a doubling, and the reason is dimensionality. M = 32 is how many neighbour links HNSW keeps per vector. That link block is a fixed ~276 bytes per vector — 256 for layer-0 links, 4 for the upper layers, 16 of bookkeeping — no matter how long the vector itself is.

So the links double memory only when the payload happens to be about 276 bytes, which is roughly the case at 128 dimensions in fp16, as in 06 — Video Recommendation. At 768 dimensions in fp16 the payload is 768 × 2 = 1,536 bytes, and 276 / 1,536 = 18%. Copying “HNSW roughly doubles memory” across chapters is how a 0.9 TB tier turns into a 1.5 TB budget request.

Sharding

Sharding means cutting one logical index into pieces held on separate machines. Here there are 12 lexical shards, and 5.10 TB / 12 = 425 GB per shard, with each shard’s hot subset resident in the operating system’s page cache — the memory the kernel uses to hold recently read file blocks.

The split is by a hash of the video identifier, not by topic. Topic sharding is the alternative and it loses on both counts: a query does not know its topic in advance, so it fans out to every shard anyway, and grouping a topic on one machine creates a hotspot the moment that topic is popular. Hashing spreads both storage and load evenly by construction. The multi-tenancy and shard-routing arguments behind this are in Sharding and multi tenancy.

Frame embeddings: the number that kills naive designs

The most useful thing to derive here is the design you did not build, because it is the one most likely to be proposed.

The four lines below are four choices of how finely to store visual embeddings, from one vector every two seconds down to one vector per video. The note at the top explains a deliberate rounding; the gap between the top and bottom totals is the point.

512-d fp16 = 512 × 2 B = 1,024 B per frame embedding, rounded to
1 KB = 1,000 B for the four totals below — so each of them runs 2.4%
low, and the ratios they are used for are exact. 500 M videos:

    1 frame / 2 s               360 frames/video   ->  180 TB
    shot keyframes, shot 4.2 s  171 frames/video   ->   86 TB
    30-second pooled segments    24 per video      ->   12 TB   <- WHOLE corpus
    video-level pooled            1 per video      ->  500 GB

The ratio between the two ends is 180 TB / 500 GB = 360x, which is two and a half orders of magnitude — not the five people reach for when they call it “impossible”.

Frame-level visual indexing is off by 360x, so the resident visual index is the video-level pooled vector. That is the 256 GB int8 line in the vector tier above. Do not be confused by the two different figures for the same thing: the block above prices it at 500 GB because it uses fp16 throughout, for comparability across the four rows. Stored as int8 — one byte per dimension instead of two — the same 500 M vectors are 500e6 × 512 B = 256 GB. Same vectors, half the precision, half the bytes.

But mean-pooling has a cost the storage column hides, and it is the reason the design does not stop there.

Pooling averages every frame into one vector, so a brief event is averaged into invisibility. The glass leaving the table lasts about 2 seconds, which is 1 of the 360 frames sampled from a 12-minute video, so it contributes 1/360 of the pooled vector. Pool to 30-second segments instead and the same event is 1 of 15 frames — 1/15 of its segment vector, a 360/15 = 24x stronger signal.

That dilution, not impression volume, is why the 30-second segment tier exists. A query answered by one brief moment cannot survive being averaged against 359 unrelated frames.

Read the 12 TB line as what segmenting everything would cost, because that is not what gets built. The tier is built lazily and only for the visual-descriptive slice, over the top ~40 M videos by impression volume — how often a video is shown in results, so this is the popular head of the corpus. That is 8% of videos, and it covers 91% of visual-query traffic:

whole corpus   500 M × 24 segments × 1 KB   =  12 TB     never built
top 40 M only   40 M × 24 segments × 1 KB   =  0.96 TB   what ships
                                               ------
                                               1/12th the storage, against
                                               the 180 TB of a full frame index

So the shipped segment tier is about 1 TB, and note where it is not: it is an on-disk payload keyed by video id, exactly like the 1.05 TB of Temporal granularity the derivation people skip segment payloads, so it never entered the 6.3 TB of lexical-plus-vector tiers that have to be held resident and replicated. Quoting the tier at 12 TB both prices a corpus you did not segment and implies a memory bill you do not pay.

Query cost

The other half of the bill is what one query costs, and the point of computing it is to discover how small it is. QPS below means queries per second.

This is marginal serving cost only — the compute and I/O one extra query consumes. The memory holding the index is a fixed cost that does not depend on query volume, so it is priced separately in the next subsection and deliberately excluded here.

per query: L1 GBDT 15 ms CPU + L2 603 GFLOP GPU + 18 ms of I/O

GPU: 603 GFLOP / 60 TFLOP/s = 10.1 ms of one A10G
     one A10G serves 1 / 0.0101 = 99 QPS
     $0.75/hr / (99 QPS × 3,600 s/hr) = $0.75 / 356,400  -> $0.0000021 per query
CPU + I/O, costed the same way (index memory excluded)  -> $0.0000058 per query
                                                           -----------
                                                        ≈  $0.0000079 per query

1 M queries/day × $0.0000079  =  $7.90/day  =  $2,880/year of marginal serving

Serving is nearly free. The cost in this system is the index — memory to keep 6.3 TB resident and replicated — plus the one-time ASR backfill. The instinct is to optimize the per-query path, but the per-query path is $8 a day.

The index, priced

Having just called the index the money, price it. “The index is expensive” is not a number, and someone who cannot say whether it is $50 k or $5 M a year cannot make the trade the previous paragraph is asking for.

Three inputs go in — resident size, replication factor, and RAM price — and all three are assumptions to be challenged, not facts. The block shows each one and where it came from:

resident tiers, §8            5.10 TB lexical + 1.16 TB vector  =  6.3 TB
replication factor            3   (one primary + two replicas: survives one
                                   machine loss without losing a shard, and
                                   lets a replica be rebuilt without a
                                   read-path outage)
                              6.3 TB × 3  =  18.9 TB  =  18,900 GB resident

RAM price                     $7.00/GB-month
                              (a 64 GB machine at $0.62/hr is $453/month, i.e.
                               $7.07/GB-month, and RAM is what you are buying;
                               the cores come along for free at 25 QPS)

18,900 GB × $7.00/GB-month    =  $132,300/month
                              =  $1.59 M/year

Divide the two recurring numbers: $1.59 M / $2,880 = 550.

About $1.6 M a year of resident index against $2,880 a year of marginal serving — the index costs 550x the per-query path — plus $2.08 M once for the ASR backfill. Put the three on one line and the priority orders itself:

resident index, recurring     $1.59 M/year
ASR backfill, one-time        $2.08 M once   (§5)
marginal serving, recurring   $2,880/year

Two consequences follow.

A tier that is not resident is nearly free. That is why Temporal granularity the derivation people skip’s 1.05 TB of segment payloads and the ~1 TB visual segment tier live on disk keyed by video id, and are excluded from the 18.9 TB above. Moving either into RAM would be a quarter-million-dollar decision, so the disk/RAM boundary is a design choice, not an implementation detail.

N-best transcript indexing (Asr the highest value channel and how it fails) is the one proposal in this chapter that moves this line. It adds 4.54 TB resident, and at 3x replication and $7 per GB-month:

4.54 TB × 3 replicas × 1,000 GB/TB × $7/GB-month × 12 months  =  $1.14 M/year

That nearly doubles a $1.59 M index bill, and it buys 6.1 points of entity recall@100. A real trade with a real price. Interviewer pushback is where it gets defended.

9. Failure modes

Production finds six ways to break this system, and each comes with the same three handles: the mechanism that causes it, the number that detects it, and the control that limits it. The table at the end is the summary to memorize.

9.1 Clickbait and keyword stuffing

This is the failure that follows directly from who writes each channel, and the defense falls out of the log-odds table in Sizing the evidence channels before choosing anything rather than from a blocklist.

The mechanism is structural, not accidental: two of your five evidence channels are written by a party whose payoff is your traffic.

Here is what that looks like on one video. Compare the two creator-written channels against the two content-derived ones — the terms a searcher would type are all in the top half and none of them are in the bottom half:

title        "I FIXED MY CIVIC BRAKE PADS TORQUE SPEC ROTOR CALIPER *SHOCKING*"
description  "civic brake pads, honda brake pads, brake pad replacement, brake
              torque spec, civic si brakes, 2016 civic brakes, ..." × 40 terms
transcript   [music] "hey guys welcome back to the channel, smash that like
              button" ... 11 minutes ... no torque value is ever spoken
OCR          (none)

BM25F on the creator-written fields:  strong
BM25F on the content-derived fields:  near zero

The defense is corroboration: trust a creator-written term only to the extent that a content-derived channel independently attests it.

The evidence for that comes from splitting a number you already have. The 0.72 posterior for title terms in Sizing the evidence channels before choosing anything was measured over the whole corpus, mixing honest and stuffed titles together. Split the same sample on whether the term also appears in a content-derived field and the 0.72 falls apart into two very different populations:

P(about term | in title AND in transcript with k>=3)   =  0.89
P(about term | in title AND NOT in transcript/OCR)     =  0.19

Compare 0.19 against the 0.09-to-0.58 range for transcript terms in Sizing the evidence channels before choosing anything. An uncorroborated title term is worth less than a densely repeated transcript term, not more — which inverts the boost you would have applied from the aggregate.

Implement it as a multiplicative gate on the creator fields’ contribution to the score:

score_creator_fields  ×=  0.25 + 0.75 · corroborated_fraction

Read that formula at its two ends. At corroborated_fraction = 0 the creator fields keep a quarter of their weight — a floor, not a zero, because a brand-new upload has no transcript yet. At corroborated_fraction = 1 the multiplier is 1.0 and nothing is penalized.

Measured effect: spam-labeled results in the top 10 fall from 6.1% to 1.4%. NDCG@10 on the healthy slice moves −0.2 points. That is the price, and it is worth paying.

The code below implements the gate and the BM25F score it modifies. Read the two docstrings rather than skimming them — they carry the two things that make this different from a textbook BM25F, and both are explained again after the block.

CREATOR_FIELDS = ("title", "description", "tags")
CONTENT_FIELDS = ("transcript", "ocr")
MIN_CORROBORATING_TF = 3       # saying it aloud once must not count


def corroboration_multiplier(query_terms, field_tf):
    """Scale creator-written field contributions by how much of the query the
    content-derived channels independently attest.

    `field_tf[field][term]` is raw term frequency. The floor of 0.25 keeps a
    brand-new upload -- whose transcript does not exist yet -- retrievable at
    all; §9.4 is why that floor is also the freshness attack surface.
    """
    if not query_terms:
        return 1.0
    hits = sum(
        1 for t in query_terms
        if any(field_tf.get(f, {}).get(t, 0) >= MIN_CORROBORATING_TF
               for f in CONTENT_FIELDS)
    )
    return 0.25 + 0.75 * (hits / len(query_terms))


def bm25f_score(query_terms, field_tf, field_weight, idf,
                field_len, avg_field_len, k1=1.2, b=0.75):
    """Field-weighted BM25F, length-normalized, gated on corroboration.

    Published BM25F does NOT saturate each field and then sum them. That
    variant is unbounded in the number of fields: put a term in one more field
    and the score rises again, with no ceiling -- which is precisely the
    keyword-stuffing move this whole section defends against. Real BM25F does
    the opposite. It pools a length-normalized, field-weighted term frequency
    ACROSS fields first, then saturates the pooled quantity ONCE, so a term's
    contribution is bounded by its IDF no matter how many fields it is smeared
    across.

    Length normalization is the other half of the defense: without the
    `1 + b*(len/avg - 1)` factor, tf=1 in an 8-word title and tf=1 in an
    1,800-word transcript both score tf/(tf+k1)=0.4545, so dumping a keyword
    once into a long transcript is free. With it, the long field is discounted
    by exactly how much longer than average it runs.
    """
    gate = corroboration_multiplier(query_terms, field_tf)
    total = 0.0
    for term in query_terms:
        pooled = 0.0                             # weighted tf pooled ACROSS fields
        for f, w in field_weight.items():
            tf = field_tf.get(f, {}).get(term, 0)
            if not tf:
                continue
            length = field_len.get(f, 1)
            avg = avg_field_len.get(f, 1) or 1
            norm_tf = tf / (1.0 + b * (length / avg - 1.0))   # per-field length norm
            pooled += w * norm_tf * (gate if f in CREATOR_FIELDS else 1.0)
        term_score = idf.get(term, 0.0) * pooled / (k1 + pooled) if pooled else 0.0
        # Saturation is applied ONCE, to the pooled count, so a term can never
        # contribute more than its IDF however many fields it is stuffed into.
        # The per-field-saturate-then-sum bug is unbounded in field count and
        # trips this assert the moment stuffing starts to pay -- it is the
        # regression test for the stuffing-friendly direction.
        assert term_score <= idf.get(term, 0.0) + 1e-9, (
            "field stuffing broke the saturation bound: pool across fields, "
            "then saturate once -- never saturate per field and sum")
        total += term_score
    return total

The two subtleties, restated plainly:

1. Saturation is applied once, to a count pooled across all fields — not per field and then summed. The per-field variant is unbounded in the number of fields: put a term in one more field and the score rises again, forever. Pooling first means a term can never contribute more than its IDF no matter how many fields it is smeared across. The assert in the code is the regression test for exactly that bug.
2. Each field’s term frequency is divided by how much longer that field runs than average. Without it, one mention in an 8-word title and one mention in an 1,800-word transcript score identically, so dumping a keyword into a long transcript is free.

Two further controls sit on top of that code, because corroboration alone is gameable by a creator who simply reads the target terms aloud once:

• Pooled TF saturation plus length normalization (both in the BM25F above) blunt raw repetition: the tenth stuffed keyword barely moves the pooled count, and a term dumped into an over-long transcript is discounted by that field’s length. Saturation does not, on its own, stop a term said once from counting — that is the job of the k >= 3 corroboration gate, which is what makes read-the-keywords-aloud unprofitable.
• Satisfied-click rate as a per-video quality prior in the blend stage. Slow, but it is the only signal that sees a video which passes every content check and still wastes the user’s time.

9.2 ASR errors compounding into retrieval misses

The mechanism was derived in Why wer does not translate linearly into retrieval loss; tracing one query through it is worth the space, because the shape of the failure — total rather than graded — is what decides where the fix has to live.

Follow the trace below line by line. The gold video is a genuine match; two ASR errors (“redeye” heard as two words, “whine” heard as “wine”) remove two of the four content terms, and each arm then fails for a different reason:

query        "hellcat redeye supercharger whine"
gold video   transcript says "hellcat red eye" (split) and "supercharger wine"
lexical      "redeye" as one token: 0 postings match
             "whine" vs "wine": 0 postings match
             2 of 4 content terms lost -> BM25F score below the shard cutoff
             -> the video is never a candidate
dense        cosine 0.61, rank 340 of 500 returned -> survives retrieval,
             dies in L1 because every lexical feature is zero
result       gold video not in top 100. NDCG@10 = 0 for this query.

The failure is total, not graded, because it happens at retrieval. The document never becomes a candidate, so no ranker downstream can recover it — you cannot re-rank something that is not in the list.

Two structural mitigations, neither of which requires a better ASR model:

• Index a compound-split variant of every out-of-vocabulary token. “Out-of-vocabulary” means a token absent from the standard word list, which is a good signal that it is a run-together compound. Post redeye and also post red and eye, so the split transcript still matches.
• Give dense retrieval a candidate floor. Reserve a fixed quota of ~150 candidate slots for the dense arm, so it cannot be crowded out by a lexical arm that is confidently wrong. In the trace above the dense arm did find the video at rank 340 — it just lost the merge.

9.3 The long tail with no good result

Here the ship-gate metric is not merely imprecise but structurally blind, and the fix is a second metric rather than a better-tuned first one.

Fifteen percent of queries have nothing in the corpus that genuinely satisfies them. NDCG has no opinion about this. Showing ten barely-relevant results scores better than showing nothing, so the metric actively rewards filling the page with junk.

To see why, remember step 4 of the NDCG definition in Offline metrics: divide by the best possible ordering. “Best possible” is not a fixed thing — it depends on which pool of results you compare against, and there are two common conventions. Below, the same failed query is scored under both.

query                     "2011 subaru forester timing belt idler pulley torque"
best available result     a 2014 Forester general timing belt job, grade 1
NDCG@10, ideal drawn from    1.000     the label pool holds nothing above
  the query's own label pool           grade 1, so this page IS the ideal
NDCG@10, fixed grade-4       0.067     (2^1-1)/(2^4-1) — a weak query
  ideal, gain 2^rel - 1                on a scale that has no failure state
user outcome                 3 clicks, 3 back-buttons, reformulation, exit

The two conventions differ only in what they divide by.

Convention A normalizes against the best ordering of the labels this query actually has. The best label is grade 1, this page shows the grade-1 result, so the page is the ideal ordering and scores a perfect 1.000.

Convention B normalizes against a hypothetical perfect grade-4 result. Using the standard exponential gain 2^rel - 1, that is (2^1 - 1) / (2^4 - 1) = 1 / 15 = 0.067.

Neither convention can say “failure”. One calls the page perfect; the other calls it a weak query on a scale that has no failure state. And showing nothing at all — arguably the honest answer — scores 0 under both, which is worse than showing junk.

Notice what that means: the choice of convention changes the number by 15x and changes nothing about the blindness. When a parameter choice moves the number an order of magnitude without moving the conclusion, the metric is the wrong instrument, not a badly tuned one.

So carry a separate number. Zero-good-result rate is the fraction of queries where no returned result is graded >= 3, and it gets its own launch gate.

It also justifies a product surface NDCG will never justify: a “no strong match” state that offers a broadened query, a related channel, and the closest partial answer labeled as partial. Measured, that surface raised session-level satisfaction on the affected slice by 9 points while lowering CTR — which is precisely why CTR cannot be the goal.

9.4 Freshness, and partial documents

Two requirements that are each reasonable alone interact badly here — the kind of finding that only appears when you write down the arrival time of every channel.

The five channels do not arrive together. Write down when each one becomes available, measured from the moment of upload — and notice that the two channels available first are the two you just learned to distrust:

t = 0 s      title, description, tags, channel      available at upload
t = 40 s     visual embeddings, thumbnail OCR       transcode + encode
t = 3 min    ASR transcript, 12 min video           after transcode completes
t = 15 min   full OCR over sampled frames
t = 6 h      early engagement priors

A breaking-news video must be findable in seconds, but at t=10 s it has only the creator-written channels — the two you just learned to distrust. The freshness path is structurally the spam-vulnerable path, which is the non-obvious point.

Resolution: index at t=0 into a small “fresh” shard with a separate scorer that leans on channel-level priors (this channel’s historical corroboration rate, subscriber base, past policy strikes) instead of on content corroboration it cannot have yet, and re-index into the main shard when the transcript lands. Cap the fresh shard’s share of the top 10 at 2 slots except for queries classified as news-seeking. The staleness-window arithmetic for a pipeline like this is derived in Freshness and the staleness window derived.

9.5 Near-duplicate flooding

The page fills up with the same video ten times — and the cheapest effective signal for stopping it turns out to be derived from speech rather than from pixels.

One popular clip gets re-uploaded 400 times. Deduplicating by content hash catches exact re-encodes and nothing else — not a crop, not a mirror, not a three-second intro spliced onto the front.

Three detectors stacked on top of each other fix it. Each line below is cumulative: the percentage is the share of true duplicate pairs caught once that detector is added to the ones above it.

exact SHA of the media                                    catches 31% of pairs
+ video pHash, TMK/PDQF over keyframes                    catches 84%
+ ASR shingle, MinHash over transcript 5-grams, J >= 0.8  catches 96%

Each of those names a specific technique. SHA is a cryptographic hash of the raw media bytes, so it matches only byte-identical files. A pHash, or perceptual hash, is a short code computed from the appearance of a frame, built so that visually similar frames get similar codes; TMK and PDQF are the standard video and image variants of that idea, applied to keyframes. A shingle is a sliding window of consecutive words — five-word windows of the transcript here — and MinHash is a technique that estimates how much two sets of shingles overlap without comparing them directly. J is the Jaccard similarity of those two sets: the size of their intersection divided by the size of their union, so J >= 0.8 means the two transcripts share at least 80% of their five-word windows.

The transcript shingle is the cheap win and nobody expects it: a re-upload has the same words in the same order even after the video is cropped, mirrored, and sped up 2%. Cluster the near-duplicates, pick a canonical member by upload time and channel authority, and collapse the rest behind a “400 similar videos” control.

9.6 Cross-lingual mismatch

Not every retrieval failure has a retrieval fix; sometimes the honest answer is a product decision.

A Spanish-language repair video is the best answer to an English query, and the searcher may or may not want it.

lexical arm     0 match — different vocabulary entirely
dense arm       matches, if the encoder is multilingual

The decision is a product decision, not a model one: surface cross-language results below the fold with a language badge, and let a per-locale ratio be a tuned parameter. Do not solve it by translating queries into every corpus language — that multiplies your retrieval fan-out by the language count and returns worse results, because query translation loses exactly the entity terms that the query depended on.

Summary

One row per failure above, with the mechanism that causes it, the number that would show it to you, and the control that holds it down — this is the table to be able to reproduce from memory.

10. Alternatives considered and rejected

Every design above had a plausible alternative, and a design is only defensible if you can say what you did not build and why. Each rejection here is quantitative — a number, not a preference.

11. Interviewer pushback

This section collects the hardest challenges to the design above, each with a direct answer. Use them as a self-test: if an answer surprises you, reread the section it draws on.

“Where does the signal come from? Just index the title and description.” Testing: whether you sized the channels or guessed. The title is 8 terms and the transcript is 610 unique terms, so the transcript is about 75x the lexical surface area — and on a labeled sample, 55% of queries are topical, moment, or tail queries answered by spoken words. The title has the better per-term posterior, 0.72 against 0.09 for a single transcript hit, so it earns a boost of roughly 2 from the log-odds, not 10. And the decisive point is authorship: title, description, and tags are written by someone whose payoff is my traffic, while the transcript and frames are derived from content that cannot lie. I would not build a search system whose only evidence is supplied by the party being ranked.

“Index 30-second segments so you can deep-link. Why not?” Testing: whether you can find the cost that is not index size. The cost is not postings — the same 1,800 tokens are posted either way, so it is about 1.35x, not 24x. The cost is evidence fragmentation. A query like “civic rear brake pad torque spec” has its five terms at 0:14, 4:02, and 9:41; no 30-second window contains all five, so every segment is a weak partial match and recall@100 on 4+ term queries drops from 0.77 to 0.41. Overlap gets it to 0.55 and cannot fix it, because a query spread over nine minutes fits no window short enough to be a deep link. So I retrieve on whole videos and localize at rank time: a 2.1 KB per-video segment payload, fetched for the 100 candidates that survive L1, scored against a query I already have. That is 1 TB of payload instead of a second index, and I keep the deep link. The one exception is a quote query, where the terms exist only in one segment and are too weak to retrieve the video — those get a real segment index restricted to high-IDF 4-grams, about 3% of the postings.

“Why not just use embeddings for everything?” Testing: whether “hybrid” is a reason or a hedge. Because the two arms answer disjoint query classes. A 768-d vector compressing 1,800 words keeps the topic and discards one digit, so “torx t25 vs t27” retrieves general screwdriver-bit videos. Video search is disproportionately repair, gaming, and product queries, which means part numbers, model years, and error codes are over-represented — exactly the tokens embeddings erase. Conversely, “why does my sourdough taste sour” has almost no lexical overlap with the video that answers it. I fuse with reciprocal-rank fusion rather than a linear blend, because a BM25 score and a cosine live on incomparable scales and any weight I pick will need retuning after every index rebuild.

“ASR has 12% WER. How much retrieval does that cost you?” Testing: whether you can turn a model metric into a product metric. Much less than 12% on most terms and much more on the ones that matter, because errors are roughly independent across occurrences: a term survives with probability 1 - WER^k. A common topical noun appears nine times, so it survives essentially always. A product name has entity WER around 0.34 and appears twice — it is lost 11.6% of the time. A person named once at WER 0.41 is lost 41% of the time. Queries are anchored on exactly those rare terms, and losing the anchor means the video is never a candidate, so the failure is total rather than graded and no ranker can recover it. The metric I would report to the speech team is not aggregate WER, it is entity WER at the video level — currently 0.37, share-weighted across accent groups that run from 0.26 to 0.51. The cheapest fix is contextual biasing: the title and description already contain the hard words, so shallow-fusing them into the decoder cuts entity WER by about 38% relative — 0.34 to 0.21 on product names, 0.37 to 0.23 corpus-wide — for about 2% of throughput.

“You are done. Ship it. NDCG@10 is up 1.2 points.” Testing: whether one number moves you. Not on that alone, for three reasons.

First, stage attribution. If end-to-end NDCG rose but recall@1000 fell, retrieval regressed and the ranker is masking it — and that shows up as a cliff on the next query-mix shift.

Second, NDCG cannot see the 15% of queries where nothing in the corpus is relevant. Filling ten slots with grade-1 results scores 1.000 if the ideal list comes from the query’s own label pool, and 0.067 against a fixed grade-4 ideal. Neither number reads as failure, while the user clicks three times, backs out three times, and leaves. That needs its own gate: zero-good-result rate.

Third, I would check recall sliced by speaker-accent cluster. Entity WER is 0.26 for US-general speakers and 0.51 for non-native speakers, and those creators are a small enough share of the corpus that a 20-point recall gap moves aggregate NDCG by about 0.3 points. Aggregates hide it by construction.

“Creators are stuffing keywords and outranking real content. Fix it.” Testing: whether the fix comes from the mechanism or from a blocklist. The mechanism is that two of five channels are creator-written, so I make the creator channels conditional on the content channels. The measured posterior for a title term is 0.72 overall, but 0.89 when the term also appears three or more times in the transcript and 0.19 when it appears in no content-derived field — so an uncorroborated title term is worth less than a dense transcript hit, not more. I gate it multiplicatively: creator-field score times 0.25 + 0.75 · corroborated_fraction. That took spam in the top 10 from 6.1% to 1.4% and cost 0.2 NDCG points on the healthy slice. The k >= 3 requirement is what stops the obvious counter-move of reading the keyword list aloud once. Beyond that I need satisfied-click rate as a per-video prior, because it is the only signal that catches content which passes every structural check and still wastes eleven minutes.

“How do you measure a ranking change without waiting a week?” Testing: whether you know interleaving exists and why it works. Position bias. Clicks factor as examination times relevance, and examination at rank 1 is 0.72 against 0.13 at rank 10, so a change that reshuffles ranks 3 through 10 barely moves CTR and the A/B is mostly measuring noise. For a 0.1-point MDE on a 0.44 baseline I need about 3.9 M queries per arm, so a week and a fight with seasonality. Team-draft interleaving builds one list by alternating picks and attributes each click to the contributing ranker, so both rankers occupy comparable positions and the examination term cancels instead of being estimated — same decision on 6 to 15 thousand sessions, which against 3.9 M per arm is 260x to 650x faster. The “10 to 100x” figure people quote is the speedup on the easier 0.5-point decision, where the A/B was affordable anyway; it understates the case by an order of magnitude. I use interleaving to get from 40 candidate rankers to 3, then A/B the finalists, because interleaving cannot measure UI changes, result-count changes, or anything long-term, since both arms live inside the same session.

“Size the index and tell me what it costs.” Testing: whether you can produce numbers and know which one matters. Lexical tier: 500 M videos at 1,800 transcript tokens is 900 B postings, about 3.24 TB compressed, plus 0.50 TB for the creator fields and OCR, plus 1.05 TB of segment payloads and 0.31 TB of quote index — call it 5.1 TB.

Vector tier: 500 M at 768-d fp16 is 768 GB, and HNSW at M=32 adds a fixed ~276 B of links and bookkeeping per vector, so 0.91 TB resident. That is 18% over the payload, not the doubling people quote from the d=128 case. Plus 256 GB of int8 visual embeddings.

So about 6.3 TB resident, replicated 3x to survive machine loss, which is 18.9 TB of RAM. At about $7 a GB-month — a 64 GB machine at $0.62 an hour, and RAM is the thing I am actually buying — that is $132 k a month, $1.59 M a year.

Serving is nearly free by comparison. The cross-encoder is 603 GFLOP for 100 pairs, 10 ms on one A10G, which is $0.0000021 per query; everything else brings it to about $8 per day at 1 M queries, so $2,880 a year. The index costs 550x the marginal per-query path.

The money is the resident index and the one-time ASR backfill. 100 M audio-hours at 120x realtime batched is 833 k GPU-hours, which at $2.50 a GPU-hour is $2.08 M once, and 833 GPU-hours a day — $2,083 a day — in steady state. Unbatched at 20x realtime that backfill is 5.0 M GPU-hours and $12.5 M, so batching is a 6x lever on an eight-figure number.

“Why not index every frame? Storage is cheap.” Testing: whether you do the arithmetic before agreeing. It is not cheap at this shape. One frame per two seconds over a 12-minute video is 360 frames; at 512-d fp16 that is 1 KB each, so 500 M videos is 180 TB. Shot-boundary keyframes only get me to 86 TB. Segmenting the whole corpus into 30-second pooled vectors is 12 TB and pooling to the video level is 500 GB — a factor of 360 between the ends, which is two and a half orders of magnitude and not the five people reach for, for a channel that answers 14% of queries. So the video-level pooled embedding is the default, and I build the segment tier lazily for the top 40 M videos by impressions — 8% of the corpus, so 40 M × 24 × 1 KB = 0.96 TB, a twelfth of the 12 TB whole-corpus figure — and that covers 91% of visual-query traffic. It is also a disk payload keyed by video id rather than a resident index, so it stays out of the 6.3 TB I have to hold in RAM.

“A new video needs to be searchable in ten seconds. What breaks?” Testing: whether you notice the interaction between two requirements. At t=10 s the only channels that exist are title, description, and tags — the three the creator wrote. The transcript lands around three minutes in and the visual embeddings around forty seconds. So the freshness path is structurally the spam-vulnerable path, and the corroboration gate I just built has nothing to corroborate against. My answer is to index immediately into a separate fresh shard with a different scorer: instead of content corroboration it uses channel-level priors — the channel’s historical corroboration rate, its policy-strike history, its subscriber base — and I cap the fresh shard at two of the top ten slots unless the query is classified news-seeking. When the transcript arrives the document re-indexes into the main shard under the normal scorer.

“Give me one thing you would build next, and why that one.” Testing: whether your numbers produce a priority. N-best transcript indexing. It costs 2.4x on the transcript postings — 3.24 TB to 7.8 TB, and at 3x replication and $7 a GB-month that +4.54 TB is $1.14 M a year, which nearly doubles a $1.59 M index bill — and buys 6.1 points of entity recall@100. I would rank it above a better ranker because retrieval misses are total: if the anchor term is not in the index, no amount of ranking recovers the document, and my entity WER at video level is 0.37, meaning more than one video in three has a searchable name missing. And it disproportionately helps the accent slices where entity WER is 0.45 to 0.51, which is the gap I have the least other leverage on. If memory were the binding constraint instead, contextual biasing is the same fix for a tenth of the cost — it just requires the ASR team to ship it.

Next: 05 — Harmful Content Detection.