02 — Visual Search

A user photographs a chair they saw in a cafe. The system shows them chairs they can buy.

This chapter designs a visual product search system from the photograph to the purchase, and derives each of its numbers rather than asserting them. The hard part is not the network architecture; it is the training objective, specifically which wrong answers the model sees while it learns.

The chapter covers four things:

• why a single carefully chosen counter-example is worth several thousand random ones,
• how to harvest a billion training labels without paying an annotator,
• how to size the search index and the latency budget,
• which five failure modes break the system in production.

The input is one photograph: a JPEG from a phone camera, a couple of megabytes, often poorly lit and often containing several objects when the shopper cares about one.

The output is an ordered slate of about 20 buyable listings. The first slot reports one of two outcomes: the exact item photographed, with every seller who carries it, or no exact match, followed by the closest items in style.

Between those two ends sit three components:

1. one model that turns a picture into a list of 512 numbers,
2. one index that finds the nearest such lists among 200 million products in a few milliseconds,
3. one ranker that puts the survivors in order.

Five terms, defined once:

• An embedding is a fixed-length list of numbers a model produces from an input — here, 512 numbers from one image. It is trained so that pictures of the same thing land close together and pictures of different things land far apart.
• Cosine similarity is how “close” is measured: the cosine of the angle between two such lists. It runs from 1.0 (pointing the same way) through 0 (unrelated) to -1 (opposite).
• A SKU (stock keeping unit) is the retailer’s identifier for one distinct physical product. Two sellers offering the same chair share one SKU even though each has its own listing.
• An encoder is the model that produces the embedding.
• Recall@k is the fraction of the genuinely correct answers that appear anywhere in the top k results. It measures whether the right item was found at all, before the question of whether it was ranked first.

This chapter is the two-stage pattern from Stage 5 model choice and the two stage pattern derived made concrete: retrieve a few hundred cheap candidates, then score them expensively.

The important part is the loss, specifically which negatives the model sees. Saying “train an encoder with a triplet loss and put it in an approximate-nearest-neighbour index” describes the system correctly but does not distinguish a working system from one that returns red curtains for a red dress.

Two more terms from that sentence, both derived in full below. A triplet loss trains on three images at a time: an anchor, a matching item and a mismatched item. An approximate nearest neighbour (ANN) index trades a small amount of exactness for the ability to search hundreds of millions of vectors without comparing against all of them.

Two ideas from elsewhere, restated so this page stands alone:

• A convolutional network reuses one small set of weights at every position in an image. That is what lets it learn “an edge is an edge wherever it appears” from a reasonable amount of data. Full argument: Convolution weight sharing and what it buys.
• A vector index is a data structure that, given one query vector, returns the nearest few among hundreds of millions without scanning them all. It does this by keeping a navigable graph or a coarse partition over the vectors. Internals: Vector index internals.

1. Framing: “similar” is four different questions

Before any modelling starts, decide what the system is being asked for, then derive the similarity threshold from which the rest of the architecture follows.

The word “similar” hides four different products. They have different labels, different metrics, and in two cases different models, so decide which one you are building first.

Each column of the table is one of the four products, and each row asks the same question of all four. The two rows that matter most are “Invariant to” and “Label source”: those are where the four stop being variations on a theme and become different engineering projects.

The last two rows need spelling out.

Instance-level metric learning means training a model to place two pictures of the same individual object close together, rather than two pictures of the same kind of object. It is the distinction between “this chair” and “a chair.”

The four metrics in the “Right metric” row:

• Recall@k — the share of correct answers found anywhere in the top k.
• Precision@k — the share of the top k that are correct.
• NDCG (normalized discounted cumulative gain) — scores an ordering when some answers are better than others rather than merely right or wrong, discounting each position further down the page.
• Attach rate — the share of sessions where the shopper adds the suggested companion item to the same order.

The two that get conflated are exact-product and same-style, and they want opposite invariances. “Invariant to X” means the embedding does not change when X changes. Exact-product must be invariant to color shift from bad white balance; same-style must be sensitive to color, because a navy sofa and a mustard sofa are different products a shopper is choosing between. You cannot serve both from one embedding without saying which one you optimized.

The rest of this chapter designs for exact product first, style as the fallback when no exact match clears a threshold, which is what a shopping product needs. State the decision rather than assuming it. The diagram below is that decision; it has one moving part.

flowchart TD
    Q(["User photo"]) --> D{"Is the top-1 cosine<br/>above tau_exact = 0.83?"}
    D -->|"yes · 32%"| E["Exact-product lane<br/>rank by price, availability,<br/>seller quality"]
    D -->|"no · 68%"| S["Style lane<br/>rank by visual similarity<br/>+ category prior"]
    E --> R(["Results"])
    S --> R
    D -.->|"0.83 derived below from<br/>the cost of a wrong<br/>'this is the same item'"| C["Cost matrix<br/>ml 06 section 11"]

    style D fill:#1d3557,color:#fff
    style E fill:#2d6a4f,color:#fff
    style S fill:#bc6c25,color:#fff

The diagram is one question asked of every incoming photo: is the top-1 cosine above tau_exact = 0.83? If it is, the request goes down the exact-product lane, where the matching item is already decided and the only remaining job is to rank the sellers who carry it by price, availability and seller quality. If it is not, the request goes down the style lane, where the system ranks by visual similarity plus a prior over which category the shopper is probably shopping in. The dotted arrow points to the next subsection: 0.83 is not a tuning artefact, it comes from a cost matrix, a two-by-two table of what each kind of mistake costs, of the sort derived in Choosing a threshold from the cost matrix.

tau_exact, derived

That threshold is the only free number in the diagram, and the two-lane architecture follows from it, so derive it first, and from money rather than from a sweep. (The infonce version where the vanishing is quantitative also writes tau, for the InfoNCE temperature, a different quantity that shares the letter by convention. This chapter writes the routing threshold tau_exact to keep them apart.)

The decision is “declare this candidate the same physical item,” and there are exactly two ways to get a yes-or-no decision wrong. A false positive (C_fp) is saying yes when the answer is no; a false negative (C_fn) is saying no when the answer is yes. Price both in the same units — dollars of margin — and Choosing a threshold from the cost matrix turns them into a threshold p* on a probability: say yes only when you are at least p* sure.

Where p* comes from, in one line. If you believe the pair matches with probability p, saying yes costs (1 - p) · C_fp on average and saying no costs p · C_fn. Saying yes is the better bet exactly when (1 - p)·C_fp < p·C_fn, and solving that for p gives p > C_fp / (C_fp + C_fn). So the threshold is nothing but the share of the total error cost that the false positive owns.

Now put the two dollar figures in:

C_fp   a wrong "this is the same item": the shopper buys, the box holds a
       different chair, they return it and stop trusting the feature
                                                    ->  $14.00 of margin
C_fn   a real exact match routed to the style lane instead: the shopper
       still sees the item, one row lower, in a slate that converts worse
                                                    ->  $ 0.60 of margin

p*  =  C_fp / (C_fp + C_fn)  =  14.00 / 14.60  =  0.9589

The asymmetry is 23:1, and that is what forces two lanes rather than one ranked list. At those costs the system may not say “same item” unless it is 96% sure — and everything it refuses on still has to be shown to somebody. The style lane is where the refusals go; it is a consequence of the threshold, not a second feature bolted on beside it.

p* is in probability units and a cosine is not a probability, so the shipped number comes from a calibration of one against the other — an empirical table that says, for each range of cosine values, what fraction of those pairs really were the same product.

It is built on the 3,000-query gold set of Data labels without a single annotation: 3,000 real query photos whose correct answers a human has verified exhaustively. Note which population it is binned on — query photos against catalog items, which is what the router actually sees, and not the catalog-against-catalog population Mining strategies and the trap in the best one audits.

Read the table top to bottom and stop at the last row whose measured P(same product) is still above 0.9589.

tau_exact = 0.83. The three clearing buckets hold 214 + 331 + 408 = 953 of the 3,000 queries, so 32% of traffic takes the exact-product lane and 68% takes the style lane. The fallback is the common case, worth knowing before spending a quarter optimizing the exact lane.

Two properties of that number:

It belongs to an encoder version, not to the problem. The mapping from cosine to P(same product) is a property of the embedding, so every model version reships its own calibration table. Embedding drift on reindex’s version pinning therefore covers the threshold along with the vectors: a v2 encoder served against a v1 threshold is the same class of bug as a v2 query served against a v1 shard.

It is insensitive to the cost estimates in exactly the way you want. Halve C_fp to $7 and p* = 7 / 7.6 = 0.921, which the 0.80-0.83 bucket clears at 0.934 — so the shipped cosine moves one bucket, to 0.80. Double it to $28 and p* = 28 / 28.6 = 0.979, which only the 0.86-0.90 bucket clears at 0.981 — so it moves one bucket the other way, to 0.86. A 4x swing in the cost estimate moves the threshold by 0.06 of cosine. You do not need the margins to three digits; you need to know the ratio is tens rather than ones, which anybody in the room can tell you.

Four further questions finish the framing. Each answer changes something structural downstream, which the right-hand column records:

Those cells use terms worth defining:

• HNSW (hierarchical navigable small world) is a vector index built as a graph you walk downhill toward the query. It accepts new vectors one at a time.
• IVF (inverted file) instead partitions the vectors into clusters up front and searches a few of them. It is cheaper to delete from and clumsier to insert into.
• A tombstone is a deleted item’s vector left physically in place and merely filtered out of results, because the graph still needs it as a stepping stone.
• p99 is the 99th percentile: the latency only one request in a hundred exceeds. It is the number a user actually complains about, as opposed to the median.
• CTR is click-through rate, the share of shown results that get clicked.

What must be true for this framing to hold. Four assumptions:

1. The catalog carries a reliable product identifier, so “same product” is a fact the data already knows rather than a judgement call. Without it there is no label source and no way to compute recall.
2. A meaningful fraction of queries genuinely have an exact match in the catalog. At the 32% match rate derived above the two-lane split earns its complexity; at a rate ten times lower it would not.
3. A wrong “same item” really costs tens of times what a miss costs. That asymmetry is what forces two lanes rather than one list.
4. Traffic is dominated by phone photos rather than catalog re-uploads. That is what makes Metrics’s domain gap the largest single term in the error budget.

Change 2 and you ship one ranked list. Change 3 and the threshold collapses toward 0.5 and the lanes merge.

2. The ML objective: a relation, not a class

The input is one image and the output is 512 numbers. Everything else follows from that. Formally you are not learning a label; you are learning a function f that maps an image to a point in d-dimensional space (R^d is just “a list of d real numbers”) such that

cos(f(a), f(p))  >  cos(f(a), f(n)) + m      for every (anchor, positive, negative)

Read that line in words: for every training example — an anchor image, a positive that shows the same product, and a negative that shows a different one — the anchor must sit closer to the positive than to the negative, by at least a margin m. The margin exists so the constraint is satisfied with room to spare rather than balanced exactly on the boundary, where a small change would flip it.

Notice what is not in that objective: no class, no category, no fixed set of answers. A classifier has to know its answer set in advance, and this catalog turns over 3% of its items every week, so the only thing that survives is a model that learns a relation between two images.

Why not train a classifier and take the penultimate layer

A common alternative is to train an ordinary category classifier and reuse the layer just before its output as the embedding. It underperforms, and the reason is worth knowing.

Such a classifier is trained with a softmax: a function that turns the model’s raw output scores into probabilities over a fixed list of classes. Here that list is 4,800 leaf categories — the most specific level of the retailer’s product taxonomy, the level where “lounge chairs” stops splitting into anything finer.

A softmax classifier reaches its lowest loss the moment the representation is sufficient for category and nothing more. Within-category variation — which of the 200e6 / 4,800 = 41,667 items in the average leaf category this one is — carries zero gradient, meaning that changing it does not change the loss. Training therefore has no reason to preserve it.

The classification objective actively encourages collapsing the exact distinctions the retrieval task is built on.

The table below measures that on the same backbone, the same data and the same compute, so the only variable is the objective. The first column is the metric the classifier was trained for; the next two are the metrics you actually ship.

A linear probe is the diagnostic in that last column: freeze the embedding, fit the simplest possible classifier on top of it, and see how much of some property the embedding already contains. InfoNCE is the contrastive loss The infonce version where the vanishing is quantitative derives — for now, read it as “the softmax version of the triplet loss.”

The probe column is top-1 accuracy on a 1,000-way SKU discrimination set, chance 0.001 — not R^2. R^2 measures the share of variance explained, and SKU id is a nominal target: its values are names rather than quantities. There is no distance between SKU 88213 and SKU 90447, so there is no variance to explain, and R^2 would just be a function of whichever integers the catalog happened to assign. The red dress that returns red curtains makes that concrete and executable.

The classifier wins the classification metric and loses the metric you ship. That is the compact form of the argument.

There is one legitimate use for the classification head: as an auxiliary loss carried alongside the main one at a small weight, because category structure is a useful regularizer early in training — something that constrains the model toward simpler solutions and keeps it from wandering. Weight it at 0.1 and anneal it to zero, meaning reduce that weight smoothly to nothing as training proceeds.

f in full: what the encoder actually is

The objective says nothing about architecture, so state and defend the architecture choice by choice.

Four decisions, one row each. The middle column is what ships; the right-hand column names the alternative it beat and the number that beat it.

That table is dense with names, so unpack them:

• Backbone — the bulk of the model, the part that does the seeing.
• Head — a small layer bolted on top that shapes the backbone’s output into what a particular task needs.
• ViT-B/16 — a Vision Transformer, “Base” size, that chops the image into 16x16 pixel patches and lets every patch attend to every other patch from the very first layer. That all-to-all comparison is the property being bought here.
• ResNet-50 — the convolutional alternative. Each layer only looks at a small neighbourhood, and the region a neuron can see (its receptive field) grows slowly with depth.
• GFLOP — a billion floating-point operations, the standard unit for “how much arithmetic does one forward pass cost.”
• L2-normalized — every output vector rescaled to length 1, which is what makes a dot product equal to a cosine.
• fp16 — 16-bit floating point, two bytes per number instead of four.
• ImageNet-normalized — each colour channel shifted and scaled by the standard constants every pretrained vision model expects.
• EXIF-oriented — the photo rotated according to the orientation tag the camera wrote into the file. Without it, a quarter of phone photos arrive sideways.

Two heads is not two indexes, and that distinction is what keeps Scale and cost’s bill where it is. Only the exact-product head is indexed — 200M vectors, d = 512.

The style head is a reranking feature instead. A color-invariant embedding still retrieves the right shelf, so the style lane reranks the same 800 candidates the exact head returned, reading style vectors out of the metadata store alongside price and stock in Retrieval ranking and the arithmetic that sizes them’s existing 6 ms fetch.

That store is 200M × 512 × 2 B = 204.8 GB ≈ 205 GB of NVMe, about $16/month — the same tier and the same price Retrieval ranking and the arithmetic that sizes them puts the binary-rescore payloads on. A second ANN index would instead double the 234 GB of RAM in Retrieval ranking and the arithmetic that sizes them, which is the expensive resource, and it would add nothing the candidate set does not already contain.

The head itself is 768 × 512 = 393,216 params, which is 393,216 / 86e6 = 0.46% of the backbone. Carrying two heads instead of one costs under half a percent of the model and $16 a month of disk — the cheapest thing in the design, and the one that makes Framing similar is four different questions’s two lanes expressible at all.

The encoder, assembled. Everything above is one model. Six questions about it:

• What is the model? A ViT-B/16 image encoder with two 768->512 linear heads. 86M parameters, 34.9 GFLOP per image.
• What is the input? One 224x224 RGB crop, produced by the object detector of Query side failures, EXIF-oriented and ImageNet-normalized. Pixels only, no catalog metadata, because anything the item side knows must also be knowable for a photograph a shopper just took.
• What are the labels, and where do they come from? Pairs, not classes: “these two images show the same product.” Harvested for free from the catalog’s own product identifiers, from purchases, and from returns (Data labels without a single annotation). Never from a human annotator, except on the evaluation set.
• How is it trained? Contrastively, in four curriculum stages, against negatives that are mined rather than sampled (The loss and why random negatives stop teaching), with a logQ correction and a time-and-content-grouped split (4a training the split the correction and the cadence). Total: $297 of GPU time.
• How is it served? One forward pass per query at batch 1, 8 ms, version-pinned so a v2 query can never reach a v1 index (Serving architecture, Embedding drift on reindex). The item side runs offline in batches of 256, and its output is what lives in the index.
• How do you know it works? Recall@100 on the 3,000-query gold set, sliced by query provenance — 0.703 traffic-weighted against 0.912 on catalog re-uploads (Metrics) — plus the linear probes of The red dress that returns red curtains, which say what the embedding actually encodes rather than how well it scores.

What must be true for this encoder to work. Three assumptions:

1. The catalog’s photographs are broadly honest depictions of the product, since the whole label supply rests on “same SKU implies same object.”
2. Visual appearance is sufficient to identify the product. This fails for items distinguished only by size, firmware or fabric weight. There, no encoder of any size can help, and the reranker’s catalog attributes must carry the decision.
3. Shoppers photograph one dominant object. That is exactly the assumption Query side failures measures — and finds violated on 21% of traffic.

3. The loss, and why random negatives stop teaching

One thing separates a visual search system that works from one that does not: the choice of negative examples. Three claims, each argued below in arithmetic:

1. A randomly chosen wrong answer teaches the model almost nothing.
2. One deliberately chosen near-miss teaches it thousands of times more.
3. The strategy that mines the hardest near-misses walks straight into a trap you have to design around.

Three terms used throughout:

• A batch is the group of examples the model looks at before it updates its weights once — here 256 or 1,024 images at a time.
• An epoch is one full pass over the training set.
• A negative is a wrong answer shown to the model on purpose. Pushing it away is what teaches the model what “right” means.

3.1 Triplet loss has an exactly-zero-gradient region

The triplet loss is the natural first thing to write down for the objective in The ml objective a relation not a class — and it stops teaching after roughly one pass through the data. Here is why.

The loss takes the anchor, the positive and the negative, and charges you only when the constraint from The ml objective a relation not a class is violated:

L = max(0, d(a, p) - d(a, n) + m)          d = 1 - cos

Here d is a distance rather than a similarity — 1 - cos, so 0 means identical and 2 means opposite. If d(a, n) > d(a, p) + m, meaning the negative is already comfortably further away than the positive, the loss is 0 and the gradient is exactly zero — not small, zero. A zero gradient is a training example that changes no weight at all: the model learns nothing from it. So the question is how often a randomly drawn negative is already that easy. With a catalog of 200M items in 4,800 leaf categories:

P(random item is in the anchor's category)  =  1 / 4,800  =  2.1e-4

with an in-batch scheme, batch 1,024:
E[same-category negatives per anchor]  =  1,024 × 2.1e-4  =  0.21

An in-batch scheme means the negatives for each anchor are simply the other items that happen to be in the same batch. It costs nothing extra, because those items are already encoded.

The expectation of 0.21 says that on average only one anchor in five even sees a same-category item to push against. Four out of five anchors in a batch see no informative negative at all.

Empirically, after the first epoch 96.4% of randomly drawn triplets have zero loss. The effective batch — the number of examples actually contributing gradient — is therefore 3.6% of the nominal one.

That is worse than it sounds, because the gradient estimate’s variance (how much the direction of a single update jitters around the true one) scales as 1 / n_effective:

1 / 0.036       =  27.8   ->  ~28x the gradient variance you think you have
sqrt(27.8)      =   5.3   ->  5.3x the standard deviation

So the wall-clock cost per useful gradient is 28x the number on your dashboard.

3.2 The InfoNCE version, where the vanishing is quantitative

The loss people ship turns “random negatives are weak” from a slogan into an exact number.

InfoNCE (information noise-contrastive estimation) is a softmax contrastive loss: instead of one negative at a time it scores the positive against N negatives at once and asks the model to pick the positive out of the lineup. The temperature tau divides every score before the softmax, so a small tau sharpens the distribution and makes the loss care almost exclusively about the closest competitors. Written out, over one positive and N negatives:

L  =  -log[ exp(s_p/tau) / (exp(s_p/tau) + sum_j exp(s_j/tau)) ]

The useful property is that the gradient with respect to negative j carries weight equal to that negative’s share of the softmax denominator. The softmax share is the learning signal, so a negative whose share rounds to zero is an example the model never learns from.

Two consequences of that, worth holding onto because the arithmetic below leans on both. All the shares sum to 1, so the total weight spread across every negative is 1 - p_pos. And s_p is the cosine to the positive, s_j the cosine to negative j.

Now put in real numbers. Set tau = 0.07, the positive at cosine 0.80, and 255 in-batch negatives at cosine 0.10 — which is what random items from a 200M catalog look like:

exp(0.80/0.07)  =  exp(11.43)  =  91,900         the positive's term
exp(0.10/0.07)  =  exp(1.429)  =   4.173         one random negative's term
                                   4.173 × 255  =  1,064   all 255 of them

denominator z  =  91,900 + 1,064               =  92,964

p_pos  =  91,900 / 92,964             =  0.9886
L      =  -log(0.9886)                =  0.0115
weight on each random negative        =  4.173 / 92,964   =  4.49e-5
total negative-side gradient          =  1 - 0.9886       =  0.0114

The positive already owns 98.9% of the denominator, so there is almost nothing left for the loss to push against. That is the vanishing, in numbers.

Now add one mined hard negative — a different chair that looks nearly identical — at cosine 0.75. Nothing else changes:

exp(0.75/0.07)  =  exp(10.71)  =  45,000         the hard negative's term

denominator z  =  91,900 + 45,000 + 1,064      =  137,964

p_pos  =  91,900 / 137,964            =  0.6661
L      =  -log(0.6661)                =  0.406
weight on the hard negative           =  45,000 / 137,964  =  0.3262
total negative-side gradient          =  1 - 0.6661        =  0.334

One item at cosine 0.75 contributes a term half the size of the positive’s, because the exponential turns a 0.05 gap in cosine into a factor of exp(0.05/0.07) = 2.04. That is what a small temperature buys.

Adding one near-miss to a lineup of 255 easy items changes three quantities at once:

loss:                      0.0115  ->  0.406       35x
total negative gradient:   0.0114  ->  0.334       29x
per-negative weight:      4.49e-5  ->  0.3262      7,260x

One hard negative carries roughly 7,300 times the gradient of one random negative, so a batch of 255 random negatives is worth about 3% of a single mined one (0.0114 / 0.334 = 3.4%). That is the argument for hard-negative mining.

The corollary matters just as much: scaling the batch size is a very expensive way to buy what mining buys for free. Go from 256 to 4,096 in-batch negatives — 4,095 negatives instead of 255, all still at cosine 0.10:

4,095 × 4.173  =  17,088
p_pos  =  91,900 / (91,900 + 17,088)  =  0.8432
total negative-side gradient          =  0.1568      vs 0.0114   14x

Sixteen times the memory buys 14x the negative-side gradient — sublinear in the count, because the positive’s softmax share is already near 1 and can only fall so far. One mined negative moved the same quantity by 29x. A 16x batch buys about half what a single mined negative buys.

from math import exp, log


def infonce_weights(s_pos, s_negs, tau=0.07):
    """Softmax shares in an InfoNCE batch -- i.e. per-example gradient weights.

    Returns the loss and the weight carried by each negative. The point of
    computing it: the weights are the gradient, so a negative whose weight
    rounds to zero is an example the model never learns from, regardless of
    how many of them you put in the batch.
    """
    e_pos = exp(s_pos / tau)
    e_negs = [exp(s / tau) for s in s_negs]
    z = e_pos + sum(e_negs)
    return {
        "loss": -log(e_pos / z),
        "p_pos": e_pos / z,
        "neg_weights": [e / z for e in e_negs],
        "total_neg_gradient": 1.0 - e_pos / z,
    }


def effective_batch(loss_values, eps=1e-6):
    """Fraction of a triplet batch that is actually contributing gradient.

    Track this during training. If it falls below ~0.2 the run has stopped
    learning and no amount of extra epochs fixes it -- the fix is the
    negative sampler, not the optimizer.
    """
    live = sum(1 for v in loss_values if v > eps)
    return live / max(len(loss_values), 1)

3.3 Mining strategies, and the trap in the best one

If a mined negative is worth thousands of random ones, the next question is how to find them — and the most effective method contains a hazard which makes the model worse the harder it mines.

There are five ways to choose negatives, and they trade cost against quality:

Two of those strategy names need unpacking:

• A memory bank is a rolling cache of embeddings computed on earlier batches. It lets you draw negatives from far more items than fit in one batch. The price is that those cached vectors were produced by a slightly older version of the encoder and drift out of date.
• Offline ANN mining means building a search index over the current embeddings between epochs and asking it, for each anchor, which non-matching items it currently ranks closest. It is a direct search for the model’s own worst confusions.

The trap: the hardest negative is very often an unlabeled positive, meaning a genuine match your labels failed to record. If you mine the top-1 nearest non-labeled item for each anchor, on a real catalog roughly 16% of them are the same physical product listed by a different seller, or the same SKU with a different photo that your product-id join missed.

And that 16% carries the largest weight in the batch — the 0.33 computed above. So you are training the model, with maximum force, to separate items you actually want together.

Here is where that 16% comes from. Five hundred mined top-1 negatives were pulled and inspected by hand; the four rows are the categories they fell into:

mined top-1 negatives, audited on 500 anchors:
   genuinely different product        82%
   same product, different seller     11%       <- false negative
   same product, different photo       5%       <- false negative
   same product, different colorway    2%       <- ambiguous, depends on §1

A colorway is one colour variant of an otherwise identical design — the navy and the mustard version of the same sofa. Whether that counts as the same product is exactly Framing similar is four different questions’s exact-product-versus-style question, which is why that row is marked ambiguous rather than wrong. The two rows in the middle, 11% + 5% = 16%, are unambiguously wrong.

Two fixes, both cheap.

Semi-hard band. Instead of taking the single hardest negative, sample from a band: negatives whose cos(a, n) falls in [cos(a, p) - 0.25, cos(a, p) - 0.10]. With the positive at cos(a, p) = 0.80 that band is [0.55, 0.70].

The upper edge has to sit strictly below the mined top-1, and 0.70 does: it excludes the 0.75 region where the false negatives concentrate rather than sitting on its boundary.

What survives is still enormously more informative than random. Take a negative in the middle of the band, at cosine 0.60, and compare its softmax term against a random negative’s at cosine 0.10 — the ratio is the ratio of their exponentials:

exp(0.60/0.07) / exp(0.10/0.07)  =  exp((0.60 - 0.10)/0.07)  =  exp(7.143)  =  1,265

So a semi-hard negative is worth about 1,300 random ones — three orders of magnitude, against the 7,300x of the riskiest possible pick. You give up a factor of ~6 and remove most of the false-negative hazard.

Dedupe before mining. A perceptual hash is a short fingerprint computed from an image so that visually near-identical pictures get identical or near-identical fingerprints, which makes finding duplicates a lookup rather than a search. Run one over the catalog together with an exact-embedding-distance pass, cluster items above 0.97 cosine, and treat a whole cluster as one entity for both mining and evaluation. This costs one offline pass and removes most of the 16%. That cluster id is used three more times in this chapter — to group the train/test split (4a training the split the correction and the cadence), to dedupe the slate (Near duplicate flooding), and to count the diversity guardrail (Metrics) — so it is worth building once and naming clearly.

What interviewers probe: whether “hard negative mining” is a phrase or a mechanism. The follow-up is “why does it matter,” and the answer is the 7,300x, followed by the false-negative hazard: a candidate who knows only the first half builds a system that gets worse the harder it mines.

4. Data: labels without a single annotation

The previous two sections left one question open: where do the training pairs come from? Every label this system needs already exists inside the business, each source teaches a different invariance, and human annotation is worth paying for in exactly one place.

You never pay for a “these two images are the same product” label. You harvest it from things the business already records.

Six sources are available, and they are not interchangeable. The two probability columns carry the argument: P(same product) is the chance that a pair drawn from that source really is the same physical item, P(same category) the chance it is merely the same kind of thing. A source where the second is high and the first is low is a category signal disguised as a product signal, and row six is exactly that.

Read the co-view row carefully: it is a category signal, not a product signal. At P(same product) = 0.34, using co-view pairs as positives for an instance-level objective means two thirds of your positives are wrong, and the model learns to pull together items that merely belong to the same aisle. That is the correct label for the style lane and the wrong one for the exact-product lane — the same distinction as Framing similar is four different questions, now visible in the data.

An augmentation is a deliberately distorted copy of an image — cropped, recoloured, blurred, re-compressed — paired with its original. It manufactures a guaranteed positive out of nothing, which is why its P(same product) is exactly 1.00 and why it teaches nothing except the distortions you chose.

The first three rows of that table, plus the mined negatives of The loss and why random negatives stop teaching, stack into a curriculum: the training run introduces them in order, easiest signal first, so the model has learned the cheap invariances before it has to learn the expensive distinctions.

stage 1   augmentation pairs only          -- learns invariances, converges fast
stage 2   + same-SKU multi-photo           -- the main signal, ~1.7B pairs
stage 3   + user-photo/purchase pairs      -- closes the domain gap to real queries
stage 4   + mined hard negatives           -- section 3, the largest single jump

The augmentation set is where you install invariances deliberately, and also where you can install the wrong ones. If you never color-jitter, the model is free to use color as its primary discriminator (The red dress that returns red curtains). If you jitter aggressively, exact-product matching for products that differ only in colorway breaks. The augmentation policy is a product decision, not just a data-pipeline setting.

Human labeling appears in exactly one place: the evaluation set. It is 3,000 real user query photos, each with an exhaustively verified list of matching SKUs in the catalog, stratified by category and by photo quality so that no slice of traffic is silently absent.

The bill:

3,000 photos × 8 min each  =  24,000 min  =  400 hours
400 hours × $22/hr         =  $8,800

That is infrastructure, not a project. Without exhaustive verification, recall@k is uncomputable: recall is “how many of the right answers did we find,” and you cannot divide by a denominator you do not have. It is also the set Framing similar is four different questions’s calibration table is binned on, so its cost buys the routing threshold as well as the recall number.

What must be true for this data plan to hold. Three assumptions:

1. The product-identifier join is largely correct. A SKU split across two ids becomes a false negative in mining and a false miss in evaluation.
2. Purchases can be attributed back to the search that caused them within a 7-day window. That is what makes user photos labelable at all, and it sets the floor on retraining cadence in 4a training the split the correction and the cadence.
3. The catalog’s 4.6 photos per SKU are not concentrated in the head. If tail items have one photo each, stage 2 teaches the model only about popular products — the failure 4a training the split the correction and the cadence’s last summary row calls a tail item with no learned embedding.

4a. Training: the split, the correction, and the cadence

The loss and why random negatives stop teaching fixed the loss and Data labels without a single annotation fixed the pairs. Turning those into an actual run is ch 01 Stage 6, and all four of its decisions — how to split the data, how to correct for popularity, how often to retrain, and what it costs — are places this particular system will quietly cheat itself if you take the default, measuring well and working badly.

Split on time and on content, because there are two different leaks. A leak is any path by which information from the test set reaches the model during training, which makes the test score an overestimate of what will happen in production. The layout is train on weeks 1-8, validate on week 9, test on week 10, with the 3,000-query gold set frozen out of all three.

One more word first, because the second leak is unreadable without it. An impression is one item shown to one user in one result slate — the atomic row of the logs, and the thing every engagement counter counts.

The two leaks, each stated as mechanism then fix:

leak 1   a SKU with 9 catalog photos yields 36 same-SKU pairs. A random
         split puts (A,B) in train and (A,C) in test, so the test pair
         shares an image with a training pair and measures memorization.
         -> group the split on the content-cluster id from section 3.3,
            the same key section 9.1 dedupes on

leak 2   the reranker's ctr_30d / purchase_rate_30d / return_rate
         (section 9.4) are counters. Computed today for a training row
         from six weeks ago, the 30-day window contains the click that
         IS that row's label.
         -> point-in-time join on the impression timestamp, which is the
            trap ml 01 section 8 names (temporal features and lookahead
            leakage -- NOT section 5, which is categorical encoding)

The 36 in leak 1 is 9 × 8 / 2, the number of unordered pairs you can make from 9 photos — which is why one popular SKU can seed dozens of training pairs and why a random split almost certainly splits them across train and test.

A point-in-time join means computing every feature as it stood at the moment the row was logged, rather than as it stands today. It is the discipline Temporal features and lookahead leakage names, and this chapter needs it twice. (Section 5 of that chapter is categorical encoding, which this chapter also cites, once, correctly, for shrinkage in Cold start for new items. Two different sections, two different traps.)

Leak 2 is worth a number, because it is invisible without one: rebuilding the counters point-in-time takes offline NDCG@10 from 0.71 to 0.58, and the 13 points that vanish were never real. A team that skips the rebuild ships a reranker that measures better and ranks worse, and the only place it shows up is the offline-to-online gap Metrics is about.

logQ, and why this chapter in particular cannot skip it. In-batch negatives are drawn from whatever the batch happened to contain, and batches are drawn from impressions. So a popular item appears as somebody’s negative in proportion to its popularity — and every appearance as a negative pushes its score down.

The fix is to subtract, from each item’s score, the log of how often that item gets sampled, which cancels the effect exactly. The correction is ch 01 Stage 6’s, and it is one term inside the exponent. A logit here means the raw score fed to the softmax, before it becomes a probability:

logit'(a, i)  =  cos(f(a), f(i)) / tau  -  log Q(i)

Q(i)  =  P(item i appears as an in-batch negative), estimated as a decayed
         streaming count of appearances per content-cluster id

Size the correction before deciding it is optional. The head of a catalog is its small set of very popular items; the tail is the long stretch of rarely-seen ones. Here the impression share between them spans about 1,000x — that ratio is measured off the impression logs, and it is the one input this derivation takes on faith.

Work the correction through in two steps. First in logit units, then converted to cosine by multiplying by tau, since the correction is subtracted from cos / tau. (A nat is a unit of log-odds, the natural-logarithm counterpart of a bit.)

log Q(head) - log Q(tail)  =  log(1,000)      =  6.91 nats of logit
6.91 nats × tau            =  6.91 × 0.07     =  0.48 of cosine

0.48 of cosine is not a refinement — it is six times the entire 0.83-to-0.75 band the router lives in (Framing similar is four different questions): 6 × (0.83 - 0.75) = 0.48.

Uncorrected, then, a cosine of 0.83 means “same item” for a tail chair and “not remotely” for a head chair, and a single global tau_exact is not a legal object. The threshold would have to be per-item — a calibration table with 200M rows. logQ is what buys the one number the whole architecture is built on.

Note what it does not buy, because these two failures are conflated constantly and Failure modes’s summary table carries them as separate rows.

logQ fixes the score scale along the popularity axis. It does nothing for a tail item that appears in too few positive pairs to have learned an embedding at all. That is a data problem, and its fix is on the positive side: upsample tail SKUs’ multi-photo pairs in stage 2. Two failures, two controls, and conflating them is the common error.

Cadence is derived, not chosen, and the thing that decays is not the freshness of the index. A new listing is retrievable minutes after its photo is encoded, so freshness is an index property, not a model property. The encoder decays only as the catalog’s visual distribution drifts.

To measure that, freeze one model and keep scoring it against gold data that keeps getting fresher. w0 is the week the model was frozen, w26 is six months later, and the numbers are recall@100 on one slice of traffic:

recall@100, frozen encoder, measured on the LOW-LIGHT user-photo slice
  -- §9.2's post-fix 0.752, i.e. 40% of the 70% user side, 28% of traffic

  w0 0.752   w4 0.749   w8 0.744   w16 0.735   w26 0.721

r  =  (0.752 - 0.721) / 26           =  0.00119 recall/week

Name the population: “user photos” means three different things in this chapter, and only one decays at 0.0012 a week. The three:

• 0.703 — the traffic-weighted aggregate over everything (Metrics).
• 0.6127 — the user-photo side of that aggregate alone.
• 0.631 -> 0.752 — the low-light sub-slice, before and after The red dress that returns red curtains’s colour fix. The decay series above starts from that 0.752.

So the rate — and therefore k* and the retraining cadence — is measured on a sub-stream and assumed to carry to the whole pipeline.

That assumption is defensible: drift in the catalog’s visual distribution has no reason to prefer one lighting regime. It is still an assumption, so it belongs in the same sentence as the answer. If the low-light slice decays faster than the clean one, biweekly is a floor rather than an optimum. Re-measuring the series on the full traffic-weighted mix is a cheap check.

That series says the encoder loses about 0.0012 of recall per week it goes un-retrained. Turning a decay rate into a retraining interval needs two more inputs, and you have to ask for both: what a unit of recall is worth per week (V) and what one retrain-and-deploy costs (Y).

V first, in four steps. The last one is the only measured input — a ramp being the staged rollout of an experiment to a growing share of traffic, which is where an elasticity like this gets estimated:

V   value of recall
      600,000 searches/day × 2.1% purchase rate   =  12,600 purchases/day
      12,600 × $12 margin                          =  $151,200/day
      1 point of recall@100 moves purchases 0.4% relative (measured on the ramp)
      so 0.01 of recall  =  0.4% × $151,200        =  $604.80/day
                         =  $604.80 × 7            =  $4,234/week
      V (per WHOLE unit of recall, not per point)  =  $4,234 / 0.01
                                                   =  $423,360 per recall-unit-week

Y is a sum of four line items, three of which are derived elsewhere in this chapter:

Y   cost of one retrain-and-deploy
      $  297   training run (priced at the end of this section)
      $   16   full re-encode of the catalog (§8)
      $  223   the dual-index memory window during the swap (§10)
      $  600   half an engineer-day
      -------
      $1,136

Now the interval. Two costs pull against each other: waiting k weeks means the model is stale by an average of r·k/2 recall (it starts fresh and ends r·k behind, so the average shortfall is half of that), while retraining every k weeks amortizes Y over those weeks:

cost per week  =  V·r·k/2   +   Y/k
                  staleness     retraining

minimised at   k*  =  sqrt(2Y / (V·r))

V·r  =  423,360 × 0.00119            =  504.8  dollars per week, per week
                                               of staleness accumulated
k*   =  sqrt(2 × 1,136 / 504.8)      =  sqrt(4.50)   =  2.12 weeks

The table below evaluates those same two terms at four candidate intervals. V·r·k/2 = 504.8 × k / 2 is the staleness column and 1,136 / k is the retrain column:

That formula is the economic order quantity from inventory theory: you pay a fixed cost every time you restock and a cost that grows with how long you wait between restocks, and the square root is what balances them. The table gives the answer and, just as usefully, shows that the answer is not delicate — every interval from 1 to 4 weeks lands within 30% of the optimum.

Biweekly, and the curve is flat enough that monthly is defensible for a non-cost reason. Monthly costs $1,294 - $1,073 = $221/week more, which is 221 × 4.35 = $960/month, and it halves the number of atomic index swaps.

Embedding drift on reindex is an incident caused by exactly that operation. Trading $960/month for half as many exposures to your worst failure mode is a real argument, and the square-root shape of the curve is what makes it affordable.

There is also a floor the economics cannot see: the user-photo/purchase pairs need the 7-day attribution window to close before they are labels at all. So no cadence below one week exists here, regardless of what k* says.

Warm start does not buy you out of the re-encode, which is the only reason anyone wants it. A warm start means beginning the new training run from the previous model’s weights instead of from random initialization. It converges 5-10x faster, and the temptation is to conclude that the new space is close enough to the old one to skip the 200M-image backfill.

It is not. Embedding drift on reindex measures cos(v1(x), v2(x)) = 0.31 between versions. Warm-starting narrows that gap without closing it, and “narrower” is not a property the index can use.

So measure the cosine every cycle rather than assuming it, and cold-start once a quarter as the drift control. A warm chain makes the model a function of its own history and hides exactly the degradation the frozen probe set of Embedding drift on reindex exists to catch.

The bill. Training an image costs a forward pass plus a backward pass. The backward pass — the one that computes how each weight should change — is roughly twice the arithmetic of the forward pass, so a training step costs 3 × 34.9 = 104.7 GFLOP per image encoded.

The block below counts encodes, not pairs: each pair is two images, and stage 4 adds a third because the mined negative has to be encoded too. Then it converts encodes to FLOP, FLOP to GPU-hours at the 300 TFLOP/s of Scale and cost, and GPU-hours to dollars:

stage 1   augmentation pairs            60M pairs × 2 views    =  120M encodes
stage 2   + same-SKU multi-photo       300M pairs × 2          =  600M
stage 3   + user-photo/purchase         40M pairs × 2          =   80M
stage 4   + mined hard negatives       100M × (2 + 1 mined)    =  300M
                                                                 ----------
                                                                 1,100M

1.1e9 × 104.7 GFLOP  =  1.152e20 FLOP
1.152e20 / 3.0e14    =  384,000 s  =  106.7 GPU-hours
+ 4 epochs of ANN mining rebuilds, 3 GPU-hours each  =  12
                                                        -----
                                                        118.7 GPU-hours
118.7 × $2.50  =  $297

The whole curriculum is $297 and a full reindex is $16, so neither compute nor money is the constraint on shipping a new encoder. The constraint is the atomic swap — replacing the whole index in one indivisible step, all shards or none — and the memory window it needs (Scale and cost), which is why the cadence table above is priced on the deployment, not on the training.

What must be true for this training plan to hold. Three assumptions:

1. The catalog’s visual distribution drifts slowly. That is what makes a 0.0012-per-week decay rate, and therefore a two-week cadence, legitimate. A category launch or a seasonal turnover that changes what the catalog looks like invalidates the frozen-encoder measurement and the cadence with it.
2. The popularity distribution is stationary enough that a decayed streaming count estimates Q. This fails for a newly viral item whose count has not caught up.
3. You can hold two full indexes in memory at once for the length of a deployment. Without that window there is no atomic swap, and Embedding drift on reindex’s incident is the only remaining option.

from math import log, sqrt


def logq_shift(freq_ratio, temperature=0.07):
    """How far apart the logQ correction pushes a head and a tail item.

    Returned in *cosine* units, which is the unit the routing threshold
    tau_exact is in -- that is the whole point. If this number is larger than
    the width of the band tau_exact sits in, one global threshold is not a
    legal object and the correction is load-bearing, not optional.
    """
    return log(freq_ratio) * temperature


def retrain_interval(retrain_cost, value_per_unit_week, decay_per_week):
    """Economic-order-quantity optimum for a linearly decaying model.

    cost/week = V*r*k/2 + Y/k, minimised at k* = sqrt(2Y / (V*r)). The square
    root is why halving Y shortens the interval by 1.41x and not 2x, so
    automating the retrain buys less freshness than people expect.
    """
    return sqrt(2 * retrain_cost / (value_per_unit_week * decay_per_week))


_shift = logq_shift(1_000)
assert abs(_shift - 0.4835) < 5e-4                  # 0.48 of cosine
assert _shift > 6 * (0.83 - 0.75)                   # six times the router's band

_k = retrain_interval(retrain_cost=1_136,
                      value_per_unit_week=423_360,
                      decay_per_week=(0.752 - 0.721) / 26)
assert abs(_k - 2.12) < 5e-3

# the curve is flat near the optimum: monthly costs 21% more, not 2x
def _weekly(k, Y=1_136, Vr=423_360 * (0.752 - 0.721) / 26):
    return Vr * k / 2 + Y / k


assert round(_weekly(2)) == 1073 and round(_weekly(4)) == 1294
assert _weekly(4) / _weekly(2) < 1.25

5. Retrieval, ranking, and the arithmetic that sizes them

The serving half of the system is three things: an index that finds a few hundred candidates out of 200 million, a second-stage ranker that reorders them using information the index structurally cannot see, and a latency budget that decides whether any of it ships. The budget, built honestly, misses on the first pass, and this section says so.

The index

The index is the most expensive object in the system, and its cost — and therefore its failure mode — is memory rather than throughput.

The index holds 200M items, one embedding each after per-product pooling (Near duplicate flooding), at d = 512 dimensions in fp16, meaning two bytes per number. The per-vector cost of an HNSW graph is derived in Hnsw memory per vector derived.

Four terms, and the vector itself is only the first of them. M is the tuning parameter that sets how many neighbour links each vector keeps, and every link is a 4-byte integer id:

payload         512 × 2 B                     = 1,024 B
layer-0 links   2M × 4 B, M = 16              =   128 B
upper layers    (1/(M-1)) × M × 4 B           =     4 B
bookkeeping                                   =    16 B
                                                -------
                                                1,172 B  ≈ 1.17 KB

200M × 1,172 B  =  234 GB    ->  59 GB per shard at 4 shards, on 64 GB machines

Two of those lines need a word. Layer-0 is the bottom layer of the graph, the one that holds every vector; it is built with 2M links per node rather than M because the construction algorithm allows twice the budget there. The upper layers hold only a thinning sample of the vectors — each layer keeps roughly 1/M of the one below it, so the expected number of upper-layer copies of any given vector is 1/M + 1/M² + ... = 1/(M-1), and each copy carries M links of 4 bytes. At M = 16 that is (1/15) × 16 × 4 = 4.3 B, which is why the graph structure is essentially free above layer 0 and the payload is 87% of the bill.

Sharding means splitting the index across several machines because it does not fit on one. The reason to shard here is memory, not QPS — QPS being queries per second, how much traffic one machine can answer.

That distinction changes the failure mode. Losing a shard costs recall on a slice of the catalog rather than availability, so the system degrades quietly instead of erroring. Monitor per-shard hit counts.

The other index dial is efSearch, usually shortened to ef: how many candidate nodes the graph walk keeps alive as it searches. It trades search time for accuracy and costs no memory at all.

At M = 16, efSearch = 128 the graph returns 0.98 of the true top-100 against an exact scan of the same vectors. ef is the dial that bought that, not M — because ef is a search-time knob with zero memory cost, while every extra unit of M is 4 more bytes on every one of the 200M vectors.

Do not rank that 0.98 against the numbers in genai 06; the source says so itself. Two things block the comparison.

The k is different. Its The comparison with the columns that actually decide it table reads 0.96 for M = 16, ef = 64 — but that is a Recall@10. Recall@100 is strictly higher than recall@10 on the same index, because the correct answer only has to survive into a pool ten times deeper. So “0.98 beats 0.96” compares two different quantities, and the true gap is smaller than it looks.

The baseline is different. Its Hnsw memory per vector derived quantization table also happens to read 0.98, and states in its own text that its baseline is a third operating point, is “not a §4.4 row”, and must be read “as a set of differences, not as absolute numbers, and never against §4.4”. The deltas in that table transfer between chapters; the levels do not.

So 0.98 here is this index’s own measurement at k = 100, and the only thing borrowed from genai 06 is the shape of the trade.

One point worth stating whenever anyone quotes a recall number. Metrics’s traffic-weighted 0.703 is the product of two independent factors — what the encoder plus the query distribution can achieve against a perfect index (0.717), times what the approximate graph keeps of it (0.98):

0.98 × 0.717  =  0.703       missing from a perfect 1.0:  0.297
0.717 - 0.703 =  0.014       the graph's share of that:   1.4 points
0.297 - 0.014 =  0.283       everything else:            28.3 points

The graph costs 1.4 points and the encoder and the query distribution cost the other 28.3. A team that spends a quarter tuning ef is optimizing the 1.4.

The delete problem, which Framing similar is four different questions promised and HNSW bills for

The index choice made in Framing similar is four different questions has a hidden cost: a graph index cannot really delete anything, and at 3% weekly churn that bill arrives within a month.

Framing similar is four different questions chose HNSW over IVF because 3% weekly turnover means continuous inserts. The other side of that choice is that HNSW has no true delete: a delisted item is tombstoned — filtered from the result set, still present in the graph as a routing hop (Deleted documents that stay retrievable). At 3% per week rather than genai 06’s 5% per month, the bill arrives fast.

The table below has five columns and no header, so work one row before reading the rest. Take w = 13: after 13 weeks, 200M × 0.03 × 13 = 78M vectors are dead; the index now holds 200 + 78 = 278M vectors, of which 78 / 278 = 28.1% are dead; that is 278 / 200 = 1.39x the original memory, or 234 × 1.39 = 326 GB. Live vectors stay at 200M throughout, because delistings are replaced by new listings at the same rate.

tombstones after w weeks   =  200M × 0.03 × w      (live stays at 200M)

w =  4.3   26M dead   11.5%   1.13x memory    265 GB
w = 13     78M dead   28.1%   1.39x           326 GB
w = 26    156M dead   43.8%   1.78x           417 GB
w = 52    312M dead   60.9%   2.56x           600 GB   -> 13 shards, 39 machines

The four-shard layout has three weeks of headroom, and that is the number that sizes the cluster. Four 64 GB machines hold 256 GB against 234 GB live:

headroom       (256 - 234.4) / 234.4    =  9.2%
weeks to fill  0.092 / 0.03 per week    =  3.1 weeks

Two consequences, and both are design decisions rather than operations tickets.

Compaction is scheduled, not triggered. Compaction means rebuilding a shard from scratch containing only its live vectors — the only way a graph index reclaims the space its tombstones hold.

Rebuild one shard at a time behind the alias. An alias is a name that points at whichever index version is current, so readers can be moved from one to another without knowing it happened. This is the same machinery Embedding drift on reindex already builds for encoder upgrades, at a quarter of the risk, because the vectors themselves do not change.

one shard, 40M vectors:
   40e6 × 1.2 ms build         =  48,000 s single-threaded
   / 32 build threads          =   1,500 s  =  25 min
   3 spare machines (one per replica) × $0.62/hr × 0.42 h  =  $0.78

The shipped layout is five shards, not four. Compacting one shard every 5 days gives each of five shards a 25-day cycle, so a shard’s tombstone load peaks just before its turn comes round:

five shards
   peak tombstone share   25 days / 7 days per week × 3%  =  10.7%
   shard size             234 GB / 5                      =  46.9 GB live
   peak                   46.9 × 1.107                    =  51.9 GB   fits 64 GB

four shards, same 5-day rotation
   cycle 20 days -> 20/7 × 3% = 8.6%
   58.6 × 1.086                                           =  63.6 GB   does not

63.6 against 64 is not headroom, it is a coincidence. Five shards of 47 GB replicated 3x is 15 machines, and Scale and cost prices that.

There is one non-cost reason the interval is a commitment rather than a preference: a delisting is sometimes a takedown — counterfeit, recalled, or a seller exercising erasure — and a tombstoned vector still exists in RAM and in every snapshot. The compaction interval is the takedown SLA, the service level agreement that says how long a removal is allowed to take, which is what makes 5 days a number somebody signs rather than a number somebody tunes.

The lever the cost structure points at

Since the index is almost all of the bill, the one compression that changes it is worth pricing, with its cost stated honestly rather than sold as free.

Binary quantization stores each of the 512 dimensions as a single bit — is this number positive or negative — instead of two bytes. It then rescores the few hundred survivors against their true full-precision vectors to recover the ordering.

It compresses the payload and nothing else, because the graph links are integers with no redundancy to squeeze. Rerun the per-vector arithmetic with 512 bits (64 B) of payload instead of 1,024 B:

payload         512 bits  =  64 B          was 1,024 B
layer-0 links                 128 B        unchanged
upper layers                    4 B        unchanged
bookkeeping                    16 B        unchanged
                              ------
                              212 B        was 1,172 B  ->  5.5x, not 8x

200M × 212 B  =  42 GB

The 5.5x rather than 8x is the whole point of writing it out: a naive /8 of the entire structure ignores the 148 B of links and bookkeeping that do not compress. 42 GB fits one 64 GB machine with room for a compaction cycle’s tombstones on top (47 GB at peak), so three machines instead of fifteen.

The term people drop is that rescoring needs the full vectors, which have to live somewhere. That is 200M × 1,024 B = 205 GB of fp16 payloads on local NVMe — solid-state disk attached directly to the machine, far slower than RAM to read and far cheaper per gigabyte to rent. At $0.08/GB-month that is about $16/month, plus roughly 2 ms of random reads for a top-200 fetch (Hnsw memory per vector derived), which lands as a tenth line in the latency budget below.

And the recall is not free either. The same source tables binary at 0.71 raw, 0.96 rescored, against 0.98 for fp16 — read as that table insists it be read, as a two-point delta rather than as two levels. Two points of index recall is not a rounding error against the 1.4 points the index costs today. It doubles it:

graph 0.98:  0.98 × 0.717  =  0.703 end to end   index costs 1.4 of the 29.7 missing
graph 0.96:  0.96 × 0.717  =  0.688 end to end   index costs 2.9 of the 31.2 missing

The share the index owns of the total shortfall barely moves — 4.8% to 9.2% — because the encoder and the query distribution are so much larger. But the index’s own contribution to the error doubles, which is the honest way to state it: you are buying 3.4x on the bill (Scale and cost) with the one term in the error budget you had already made small. That is still a good trade, and it does not need to be sold as free.

Whether it is right here depends on Framing similar is four different questions: the exact-product lane rescores its top 200 anyway before comparing against tau_exact, because a routing threshold read off a quantized score is a threshold on the wrong quantity.

The reranker, and what it can see that the index cannot

The system needs a second model for one reason: the index can only compare two vectors that were computed without ever seeing each other, and a great deal of what decides a purchase is a property of the pair.

The ANN score is a single dot product between two independently computed vectors. That independence is exactly the constraint that makes indexing possible at all — you can precompute an item’s vector because it does not depend on the query (ch 01).

It is also exactly what the second stage is for: features that are functions of the pair. Each row below is one such feature, and the right-hand column says why a dot product between two independent vectors structurally cannot contain it.

Two of those rows use names worth unpacking. Earth-mover distance between two colour histograms measures how much colour mass you would have to move to turn one picture’s palette into the other’s, which is a far more forgiving comparison than counting exact matches. OCR is optical character recognition — reading the text baked into the pixels, such as a logo, so it can be compared against the catalog’s brand field.

The model that consumes these is a GBDT (gradient-boosted decision tree ensemble): a large collection of small decision trees, each trained to correct the errors of the ones before it, and the standard winner on tabular features of this kind (ml 03 on why a tree and not a neural network here).

A GBDT over ~60 such features costs about 3 microseconds per item, so on 200 candidates:

200 candidates × 3 us  =  600 us  =  0.6 ms of scoring
                                     8.0 ms of feature assembly

That is the chapter 01 inversion again: fetching the numbers costs thirteen times more than using them.

The reranker, assembled. Same six questions as the encoder:

• What is the model? A GBDT over about 60 pair features, scoring the 200 candidates that survive dedupe.
• What does it eat? The cross-features in the table above, plus the business features — price, availability, seller quality, return rate — plus the style-head embedding read out of the metadata store. None of these are pixels, so the reranker never re-encodes anything.
• Where do its labels come from? Logged impressions from the search results themselves: clicks, purchases and returns attributed back to the search that produced them, graded as in Metrics — exact match 3, same style 1, other 0.
• How is it trained? On those logged slates, with a point-in-time join on every counter feature and a temporal split. That is leak 2 above, and it is worth 13 points of NDCG@10 if you get it wrong.
• How is it served? Synchronously, inside the 9 ms slot in the latency budget below — 8 ms of feature fetching and 0.6 ms of trees.
• How do you know it works? NDCG@10 offline on held-out slates, and attributed purchase rate online.

What must be true for the reranker to hold. Three assumptions:

1. The candidate set already contains the right answer. A reranker cannot retrieve; its ceiling is retrieval’s recall, which is why Metrics calls recall the ceiling on everything downstream.
2. Catalog attributes such as dimensions and brand are populated and roughly accurate, since half its cross-features read them.
3. Engagement counters exist for the candidates. This is exactly false for new listings, and it is the whole of Cold start for new items.

The latency budget

The per-stage timings now assemble into an end-to-end budget, which has to clear the requirement from Framing similar is four different questions — and, on the first pass, does not.

The encoder is the one stage whose cost you can derive rather than measure, so derive it.

A ViT-B/16 at 224 pixels cuts the image into (224/16)^2 = 196 patches, plus one extra summary token — the CLS token, a learned slot whose final value is used as the whole image’s representation. That is 197 tokens. With L = 12 layers and width d = 768, the matrix multiplications hold 12 · 12d^2 = 12 × 12 × 768^2 = 85M parameters.

Two terms then, and the second is far smaller than people expect at this sequence length. The first line is 2 FLOP (one multiply, one add) per parameter per token; the second is the all-to-all attention, quadratic in the 197 tokens:

matmul     2 × 85e6 × 197                        =  33.5 GFLOP
attention  12 × 4 × 197^2 × 768                  =   1.4 GFLOP
                                                    ----------
                                                    34.9 GFLOP / image

A TFLOP is a trillion of those operations, so “300 TFLOP/s effective” in Scale and cost is the arithmetic rate one modern GPU actually sustains on this work.

Every stage of the request now gets a line in the budget below. Three things to know before reading it:

• MMR is maximal marginal relevance, the diversity step derived in Near duplicate flooding, which gives up a little relevance to stop the slate filling with near-identical items.
• The first number column is p50, the median — half of requests are faster. The second is p99. Blank means that stage has no meaningful tail.
• “Launch-bound at batch 1” means the GPU spends more time being told what to do than doing it, because a single 34.9 GFLOP image does not come close to filling it.

One asymmetry in the middle of the table is deliberate and is explained in Near duplicate flooding: the index is asked for 800 candidates but the reranker only scores 200, because near-duplicate collapse and eligibility filtering eat the difference.

                                                  p50    p99
upload + JPEG decode + resize                      45    108
safety / moderation classifier                      6
object detection + crop (query has clutter)        18
ViT-B/16 encode, batch 1                            8           35 GFLOP, and
ANN, 5 shards in parallel, efSearch 128             3     25    launch-bound at
eligibility filter + metadata fetch, 800 cands      6           batch 1, not
rerank 200 candidates (assembly 8 + score 0.6)      9           FLOP-bound
content-cluster dedupe + MMR diversity              2
jitter across the six stages with no p99 column          +8
                                                  ---    ---
                                                   97    190

190 ms p99 against Framing similar is four different questions’s 150 ms requirement: it misses by 40 ms, and this configuration is not shippable.

Naming which term to attack takes one more observation. A p99 is not really a sum of p99s — it is dominated by whichever single term has the fattest tail, because the odds of two independent terms both being at their worst on the same request are tiny.

Here the fattest tail is upload on a mobile network, which goes 45 -> 108 ms. A distant second is the shard fan-out: waiting on five shards in parallel means waiting on roughly the p99.8 of any one of them, which is why 3 ms becomes 25.

So the fix is the term that dominates the tail, which is not the term that dominates the mean:

client-side downscale to 640 before upload
   p50   45 -> 12 ms          p50 becomes   97 -  33  =   64 ms
   p99  108 -> 30 ms          p99 becomes  190 -  78  =  112 ms   <- clears 150

That subtraction is only legal because one term dominates. Take away the dominant tail and you can no longer keep subtracting.

To hedge a request is to send a duplicate to a second replica when the first is slow and take whichever answers first, which cuts a tail at the cost of a little extra load. Hedge the ANN fan-out too — re-issue to a second replica for any shard silent at 8 ms, taking 25 -> 9 — and you may not subtract another 16. Once upload stops dominating, the p99 is set by the co-occurrence of several medium tails, and that has to be measured rather than added. Treat 112 ms as the number and the hedge as margin.

Two things follow that are worth saying in the room.

Half the p50 and the whole p99 problem are upload and decode, which is not a model problem. The levers are a client change and streaming the first 8 results before the reranker finishes the rest, which matters more than shaving the encoder.

The client change is a launch dependency, not an optimization. Until the app ships it, the p99 is 190 and the requirement is missed. The downscale to 640 node in Serving architecture’s diagram is only the server-side fallback for old app versions and web uploads: it caps decode cost, but it runs after the 45 ms has already been spent on the wire. It protects the p50 and does nothing for the p99.

6. Metrics

“How do you know it works?” has three answers here: what to measure offline for each stage, why the offline number will be about 21 points too optimistic if you build the evaluation set the obvious way, and which online metric to ship against.

Offline

Each stage of the funnel gets its own metric, because a stage can only be held responsible for the job it does:

That 0.976 is a cumulative number, and the temptation is to quote a marginal one, a 1.4-point error. The lane’s Precision@1 is the verified-same-product rate across everything the lane admits, not the rate in the bucket that happens to sit on the threshold. Weight Framing similar is four different questions’s three clearing buckets by the queries in each:

(214 × 0.994  +  331 × 0.981  +  408 × 0.962)  /  953   =   0.9758

0.962 is the 0.83-0.86 bucket ALONE -- the marginal bucket, the worst of
the three, and the one the threshold was read off. It is the right number
for "does this bucket clear p*"; it is the wrong number for "how good is
the lane", which mixes in 545 queries scoring higher and cleaner.

The marginal number answers where the threshold goes; the cumulative one answers how the lane performs. Quoting the first as the second understates a shipped lane by 1.4 points — and The demote threshold falls out of a ratio you can measure spends a subsection on the same confusion running the other way, where a cumulative precision of 4.4% is quoted to justify an action whose in-band precision is 0.98%. Same error, opposite sign, and in both cases the fix is to ask which items the number is supposed to describe.

The evaluation error that dominates this problem: your gold set is catalog photos and your traffic is phone photos. Query provenance means where the query image came from — a clean catalog render, or a phone in a dim cafe.

Measure the gap, because it is large and it is invisible if you never split the evaluation set that way. In the block below, the middle column is recall@100 on that slice and the right column is the slice’s share of traffic — 30% of all traffic is catalog re-uploads, and the remaining 70% splits three ways:

recall@100, same model, same index

catalog image re-uploaded as the query      0.912      30% of traffic
user phone photo, good light                0.784      70% x 0.30
user phone photo, low light or motion       0.631      70% x 0.40
user photo with 3+ objects, no crop         0.417      70% x 0.30
                                            -----
user side  =  0.30(0.784) + 0.40(0.631) + 0.30(0.417)
           =  0.2352 + 0.2524 + 0.1251        =  0.6127

overall    =  0.7 × 0.6127  +  0.3 × 0.912
           =  0.4289 + 0.2736                 =  0.703

The sub-weights are the point of that block, not decoration. The user side is 40% low-light because that is what photographing furniture in a cafe looks like, and the 0.703 is unreproducible without them. Ask for the split before you quote the weighted number.

Reporting 0.912 because that is what the gold set contains overstates the shipped system by 21 points. The fix is the query-photo pairs from Data labels without a single annotation in both training and evaluation, and a per-provenance slice table on every experiment.

Online: the metric that actually matters

Offline numbers rank models; one online number decides whether the product is better, and the obvious alternative to it is disqualified by a specific mechanism.

The metric is attributed purchase within 7 days of a visual search, per search — attributed meaning the purchase is credited back to the search that led to it, within a 7-day window. Not CTR — and the reason is specific rather than general. Near-duplicate flooding (Near duplicate flooding) raises CTR@1, because 12 photos of the same appealing chair produce a very clickable first result, while destroying the session because the user has no alternatives to compare. A metric that improves when the product gets worse is not a metric.

The last question is how long an experiment has to run. Size it on the fast proxy at its 31% baseline against a 2% relative MDE — the minimum detectable effect, the smallest change you insist on being able to see.

The 16 in the formula is the standard constant for 80% power at 5% significance on a two-sided test of two proportions; p is the baseline rate and delta is the absolute change you want to detect, which here is 2% of 31%:

delta      =  0.02 × 0.31            =  0.0062 absolute

n per arm  ≈  16 · p(1-p) / delta^2
           =  16 × 0.31 × 0.69 / (0.0062)^2
           =  3.4224 / 3.844e-5
           ≈  89,000 searches per arm

At 600k visual searches/day and a 50/50 split, each arm sees 300k searches a day, so 89,000 / 300,000 = 0.30 of a day — about seven hours. The primary metric at a 2.1% purchase base rate needs roughly 20x that.

So the fast proxy gates the ramp and the primary metric gates the launch.

7. Serving architecture

Every component built so far now takes its place on one request path, in order. The diagram below reads top to bottom, one box per stage of a single request; the coloured boxes are the four stages that carry the numbers derived earlier: the encoder, the ANN search, the reranker, and — in red, because it is the one most systems get wrong — the dedupe.

flowchart TD
    U(["User photo"]) --> DEC["Decode + EXIF orient<br/>downscale to 640"]
    DEC --> MOD{"Moderation<br/>+ CSAM hash"}
    MOD -->|block| REJ(["Reject"])
    MOD -->|pass| DET["Object detector<br/>largest salient box"]
    DET --> CROP["Crop + pad to 224"]
    CROP --> ENC["ViT-B/16 encoder<br/>35 GFLOP · 8 ms<br/>version-pinned"]
    ENC --> QC{"Query cache<br/>key: pHash + model_version"}
    QC -->|"hit 9%"| RES
    QC -->|miss| ANN["ANN · 5 shards<br/>efSearch 128 · 3 ms<br/>k' = 800"]
    ANN --> FIL["Eligibility filter<br/>region · stock · policy"]
    FIL --> DD["Content-cluster dedupe<br/>image hash, not product_id<br/>612 listings -> ~211 clusters"]
    DD --> RR["Reranker · GBDT<br/>60 cross-features<br/>200 candidates · 9 ms"]
    RR --> DIV["MMR diversity<br/>lambda = 0.75"]
    DIV --> RES(["Top 20"])
    RES --> LOG[("Impression log<br/>+ propensity + model_version")]

    style ENC fill:#1d3557,color:#fff
    style ANN fill:#2d6a4f,color:#fff
    style RR fill:#bc6c25,color:#fff
    style DD fill:#9d0208,color:#fff

Walk the path once:

1. Decode. Decode the JPEG, apply EXIF orientation so the image is the way up the camera intended, and downscale to 640 pixels on the long edge.
2. Safety. Two checks: a moderation classifier, and a CSAM hash lookup. CSAM is child sexual abuse material, and it is checked by exact fingerprint against a known list rather than by any model.
3. Detect and crop. An object detector picks the largest salient box — the dominant object in the frame. Crop + pad to 224 turns that box into the fixed-size square the encoder expects, padding rather than stretching so nothing is distorted.
4. Encode. The ViT turns that square into 512 numbers in 8 ms. It is version-pinned, meaning the model version travels with the request.
5. Cache. A query cache keyed on the pHash (perceptual hash, Mining strategies and the trap in the best one) plus that model version answers 9% of requests outright.
6. Search. The rest go to five ANN shards searched in parallel at efSearch 128, which return k' = 800 listing ids.
7. Filter. An eligibility filter drops anything out of region, out of stock or against policy. 612 survive.
8. Dedupe. A content-cluster dedupe collapses those 612 listings to about 211 distinct products.
9. Rerank. The GBDT scores those with its 60 cross-features.
10. Diversify. An MMR pass — maximal marginal relevance, which trades a little relevance for a slate whose items are not all the same thing (Near duplicate flooding) — picks the final 20.
11. Log. Every impression is logged with its propensity: the probability that item was shown in that slot, which is what makes the log usable for training later.

Three things in that diagram are load-bearing and easy to leave out.

The cache key includes the model version. Without it, an encoder rollout silently serves v1 results to v2 queries (Embedding drift on reindex).

The dedupe happens before the reranker, not after. The 612 listings that survive the eligibility filter collapse to 211 clusters, so reranking first would spend (612 - 211) / 612 = 66% of the expensive stage on candidates that are about to be thrown away. That 211 is also why k' is 800 rather than Near duplicate flooding’s mean requirement of 580: the reranker’s 200 slots have to be full after both filter attrition and the collapse.

And it keys on the image-hash cluster, not on product_id. Near duplicate flooding is the failure that distinction exists to prevent.

Three smaller ones:

The 9% cache hit rate is for the tail, not the bill. The hits are viral products and screenshot re-uploads — the same image arriving from many users. It saves 9% of 8 + 3 = 11 ms, which is about 1 ms of mean latency and no meaningful compute at 25 QPS. It is also the reason the model version has to be in the key: a cache is the one place a v1 vector can outlive a v2 rollout.

The CSAM hash is a separate control from the moderation classifier. It is an exact-match lookup against a known-hash list with a legal reporting obligation attached, not a score with an operating point. You do not tune its threshold, and it does not share the classifier’s 6 ms failure budget.

The impression log carries the propensity because Cold start for new items’s exploration slot is randomized. Without the probability each item was shown with, that slot produces logs you cannot debias later — which is the entire reason to pay for it.

Index lifecycle

The request path above assumes an index exists and is current. It gets that way through three separate flows, which must never be confused with each other.

flowchart LR
    NEW["New/updated item"] --> IQ[["Ingest queue"]]
    IQ --> EMB["Encode<br/>batch 256"]
    EMB --> POOL["Pool photos -> one<br/>vector per product"]
    POOL --> INS["HNSW insert<br/>1.2 ms/vector"]
    INS --> LIVE[("Live index<br/>alias -> v_n")]
    DELETED["Delisted item"] --> TOMB["Tombstone<br/>filtered post-search"]
    TOMB --> LIVE
    REBUILD["Encoder upgrade"] --> FULL["Full re-encode<br/>200M images"]
    FULL --> V2[("Shadow index v_n+1")]
    V2 -.->|"atomic alias swap<br/>ALL shards or none"| LIVE

    style LIVE fill:#2d6a4f,color:#fff
    style V2 fill:#bc6c25,color:#fff

The three flows read left to right, and they are independent of each other.

Top row — a new or changed item. It enters the ingest queue, is encoded in batches of 256, and is then reduced by the rule that governs the third node: pool a product’s photos into one vector, so the index holds products rather than pictures (Near duplicate flooding). It is added to the graph at an HNSW insert cost of 1.2 ms per vector, and is answerable from the live index alias within minutes.

Middle row — a delisting. The item is tombstoned rather than removed.

Bottom row — an encoder upgrade. All 200M products are re-encoded into a shadow index built beside the live one, which is then promoted by a single atomic alias swap across all shards or none. This is the dangerous flow, and Embedding drift on reindex is the incident it causes when it is done gradually instead.

Two nodes mean more than they appear to:

Encode batch 256 is the ingest queue’s whole reason to exist. The same encoder that costs 8 ms at batch 1 in Retrieval ranking and the arithmetic that sizes them reaches the 300 TFLOP/s Scale and cost prices only at batch 256 — a 68x utilization gap. So ingest buffers into batches and serving does not. Those are two deployments of one model with opposite latency contracts, and pretending they are one is how ingest starves the query path.

Tombstone · filtered post-search is not a delete. The node stays in the graph as a routing hop, and Retrieval ranking and the arithmetic that sizes them’s compaction schedule is what eventually removes it. Anything downstream that counts index size — including the per-shard memory alarm — has to count tombstones separately from live vectors, or it will read healthy at 61% dead.

One query, end to end

Every figure below has already been derived above; no new numbers. Assembling them once on a single request shows the components working as a system:

2.1 MB phone photo of a chair, shot in a cafe

  upload + EXIF orient + decode + resize          45 ms   -> 640 px long edge
  moderation classifier 0.003, CSAM hash miss      6 ms   -> pass
  detector: 3 boxes, largest is the chair @ 0.71  18 ms   -> crop + pad to 224
  ViT-B/16, exact-product head, L2-normalized      8 ms   -> 512-d fp16
  query cache: pHash+v7 miss                              -> search
  ANN, 5 shards, ef 128, k' = 800                  3 ms   -> 800 listing ids
  eligibility: region, in-stock, policy            6 ms   -> 612 survive
  content-cluster dedupe, 612 / 2.9 mean                  -> 211 clusters
  rerank, GBDT, 60 cross-features, top 200 of 211  9 ms   -> scored
                       (the reranker's slot count is 200, per the budget
                        and the §7 diagram; the 11 lowest-scoring clusters
                        are dropped by the ANN's own ordering, which is
                        the slack k' = 800 was chosen to leave)
  MMR, lambda = 0.75                               2 ms   -> 20 clusters
                                                  -----
                                                   97 ms

  top-1 cosine 0.86  >=  tau_exact 0.83   ->  EXACT-PRODUCT LANE
  slot 1 expands the winning cluster back into its 4 seller listings,
  ranked by price and seller quality -- which is what the lane is FOR
  slots 2-20: the next 19 clusters, one listing each

Both of the decisions the user can feel were derived rather than chosen: 0.86 clears 0.83 because Framing similar is four different questions priced a wrong “same item” at $14 against $0.60, and the slate holds 20 distinct products rather than 2 because the dedupe keys on the content cluster of Mining strategies and the trap in the best one rather than on product_id. Note also that collapsing to clusters and then expanding the winner is not a contradiction — it is the two lanes doing their separate jobs: dedupe guarantees the slate has 20 things to compare, and the exact-product lane guarantees the one you asked for is shown with every way to buy it. Change either number and the same 97 ms of pipeline produces a different product.

8. Scale and cost

The whole system gets priced twice below: once for the largest one-off operation, rebuilding the entire index, and once for a month of steady serving. The money is somewhere most people do not guess.

Full reindex. Re-encoding the whole catalog means 200M images at 34.9 GFLOP each, plus reading them, decoding them and rebuilding the graph:

compute   200e6 × 34.9 GFLOP  =  7.0e18 FLOP
          at 300 TFLOP/s effective, one GPU  =  23,300 s  =  6.5 GPU-hours
image read  200e6 × 150 KB    =  30 TB  at 10 GB/s aggregate  =  50 min
JPEG decode 200e6 × 8 ms CPU  =  1.6e6 CPU-s  =  444 CPU-hours
HNSW build  200e6 × 1.2 ms    =  240,000 s single-threaded, 5 shards parallel
                                 with 32 build threads each  =  240,000 / 160
                                                             =  1,500 s  =  25 min

The encoder is 6.5 GPU-hours and the index build is 25 minutes, so a full reindex is well under a day and costs well under $100 of compute. Priced out:

GPU   6.5 GPU-hours × $2.50/GPU-hour     =  $16.25
      (H100, the same 300 TFLOP/s effective the encode is sized on)
CPU   444 CPU-hours × $0.035/CPU-hour    =  $15.54
                                            ------
                                            $31.79   plus 30 TB of reads

That number changes the conversation. The constraint on upgrading the encoder is not compute. It is the atomicity of the swap and the 2x memory during the dual-index window — 234.4 GB × 2 = 469 GB while both versions are resident. Provision for that window or you cannot roll back.

It is also, per 4a training the split the correction and the cadence, the term that sets the retraining cadence: the training run is $297 and the deployment is what you are actually scheduling.

Steady-state serving. The recurring bill, at 600k searches/day and a peak of 25 queries per second. The first four lines size each component; the lines after the break turn the one that matters into dollars:

encoder   25 QPS × 35 GFLOP = 0.87 TFLOP/s   ->  well under one GPU; batch it
ANN       25 QPS × 5 shards, 3 ms            ->  trivial; the 15 machines exist for RAM
rerank    25 × 200 × 3 us  =  15 ms/s of CPU ->  negligible
index RAM 15 machines × 64 GB                ->  the entire cost

15 machines × $0.62/hr × 720 h/month          =  $6,696/month
+ GPU for encode, 0.5 H100 reserved:
   0.5 × 720 h × $2.50/GPU-hour               =  $  900/month
+ ingest re-encode, 3% weekly churn:
   6M images/week × 34.9 GFLOP = 2.094e17 FLOP/week
   2.094e17 / 3.0e14 = 698 s                  =  0.1939 GPU-hours/week
   0.1939 × 4.35 weeks/mo × $2.50/GPU-hour    =  $ 2.11/month
+ compaction, one shard every 5 days (section 5):
   6/month × 3 spare machines × 0.42 h × $0.62 =  $ 4.69/month
                                                 ------------
                                                 $7,603/month

88% of the cost is RAM holding an index that answers 25 queries per second (6,696 / 7,603 = 88%), and at the end of each compaction cycle a tenth of that RAM is holding items nobody can buy. Both halves point somewhere.

The tombstones point at Retrieval ranking and the arithmetic that sizes them’s compaction schedule, which costs $4.69 a month and is the difference between 15 machines and the 39 that an uncompacted year ends at.

The RAM points at the quantization lever. Binary with rescore drops the index to 42 GB — one shard instead of five, so three machines instead of fifteen:

RAM     3 machines × $0.62/hr × 720 h        =  $1,339
GPU     unchanged                            =  $  900
churn   unchanged                            =  $ 2.11
NVMe    205 GB × $0.08/GB-month              =  $16.38   (the rescore payloads)
                                                -------
                                                $2,258/month

$7,603 / $2,258  =  3.4x

That is a 3.4x win for two points of index recall (Retrieval ranking and the arithmetic that sizes them) and 2 ms of random reads. Quote it that way rather than as free.

The reserved half-GPU is priced at the same $2.50/GPU-hour as everything else, not at an unstated reserved discount. It is 360 GPU-hours a month, against the 0.8 GPU-hours the churn re-encode consumes and the 6.5 of a full reindex. You are paying for availability, not throughput, the same shape as the RAM line, one order down.

9. Failure modes

This system breaks in production in five ways, and each has the same anatomy: a mechanism that causes it, a measurement that would detect it, and a control that prevents it. None of the five is fixed by a better model.

9.1 Near-duplicate flooding

This is the most common visible failure — the slate fills with the same chair twenty times — and it is a data-modelling error rather than a model error. The slate below is what a photo-level index returns — 920M vectors, one per catalog image. Retrieval ranking and the arithmetic that sizes them sizes the shipped index at 200M because the first of the two fixes here is already applied to it; read this as the system you get if you index the thing you have rather than the thing the user is shopping for.

QUERY   a photo of a mid-century walnut lounge chair

top 20 returned by raw ANN:
  ranks 1-9    SKU 88213, nine catalog photos of the same chair
  ranks 10-14  SKU 88213 listed by four other sellers, plus one crop
  ranks 15-20  SKU 90447, six photos

distinct physical products in the top 20:   2
distinct product_ids in the top 20:         7   <- 1 SKU + 4 sellers + 1 crop + 1
user-visible outcome:              nothing to compare, no purchase
CTR@1:                             0.41   <- UP. The metric approves.

Read the second line before anything else, because it is the failure inside the failure. The slate holds two chairs and seven product ids: a product id is per listing, so four sellers listing the same chair are four ids, and a re-cropped photo uploaded as its own listing is a fifth. A dedupe keyed on product_id collapses nine catalog photos into one row and leaves the other six untouched, and the guardrail metric — “distinct products in the top 20” — then reads 7 against a target of 20 and raises nothing. The dedupe key and the guardrail metric have to be the same key, and it has to be a key on the content: the perceptual-hash / 0.97-cosine cluster id already built in Mining strategies and the trap in the best one for mining.

The index contains photos; the user is shopping for products. Two fixes, and they are not alternatives:

Index at the product level. Pool a SKU’s photos into one vector. A simple mean of L2-normalized embeddings works, and view-clustered centroids work better for products photographed from very different angles.

That phrase names a two-step operation:

1. Cluster a SKU’s 4.6 photos in embedding space, which separates the front-on shots from the three-quarter and the detail shots — those are exactly the groups the encoder places far apart. Keep one centroid per cluster.
2. Score a query against the best-matching centroid, rather than against an average of all of them.

It helps because a mean over genuinely different viewpoints lands between them and matches none of them well. It costs a SKU 2-3 index entries instead of 1, so use it for the categories that need it (furniture, footwear) rather than everywhere.

Either way, product-level indexing removes the intra-SKU flood entirely and shrinks the index by 4.6x — from 920M photo vectors to 200M product vectors, which is where Retrieval ranking and the arithmetic that sizes them’s 200M comes from.

Over-retrieve and dedupe across sellers. Pooling fixes duplicates within a SKU, but the same physical chair listed by five sellers is still five entries, so the collapse has to happen again after retrieval. That means asking the index for more results than you need — k' of them — and collapsing down to the k you will show.

How much more depends on how many listings a distinct product typically occupies. Measure that on real result sets: the mean is 2.9 listings per product and the 95th percentile is 5.4. Sizing k' on the mean gives you the number you need on a typical query; sizing it on the p95 gives you the number you need on almost every query:

measured listings-per-distinct-product among the top-k' results:
   mean 2.9, p95 5.4

to guarantee 20 distinct products at p95:  k' = 20 × 5.4  =   108
to fill the reranker's 200 at the mean:    k' = 200 × 2.9 =   580
shipped:                                   k'             =   800

800 is the mean requirement plus headroom for filter attrition, not a p95 guarantee at k = 200 — that would cost 200 × 5.4 = 1,080. Say which one you bought: the p95 rule is affordable at k = 20 and is not at k = 200.

The last step is diversity in the final slate, because two genuinely different SKUs can still look nearly the same. Maximal marginal relevance (MMR) picks the slate one item at a time, and at each step it scores a candidate by its relevance minus its similarity to whatever has already been picked, with lambda setting how much weight goes to each side:

score(i) = lambda · rel(i)  -  (1 - lambda) · max_{j in selected} sim(i, j)

And MMR does not rescue a broken dedupe key — the crossover is far below anything shippable. To show that, solve for the lambda at which MMR is exactly indifferent between one more clone of the item already selected and a genuinely different SKU.

Call the clone the high-relevance, high-redundancy option (rel_hi, sim_hi) and the different SKU the low-relevance, low-redundancy one (rel_lo, sim_lo). Set their MMR scores equal and solve:

lam·rel_hi - (1-lam)·sim_hi  =  lam·rel_lo - (1-lam)·sim_lo

lam·(rel_hi - rel_lo)        =  (1-lam)·(sim_hi - sim_lo)
lam·(rel_hi - rel_lo) + lam·(sim_hi - sim_lo)  =  sim_hi - sim_lo

lam*  =  (sim_hi - sim_lo) / ((rel_hi - rel_lo) + (sim_hi - sim_lo))

Now substitute Near duplicate flooding’s own slate. The clones sit at rel 0.95 and sim 1.00 (they are the same image, so redundancy is exactly 1); SKU 90447 sits at rel 0.62 and cos 0.846 to the chair already selected:

rel_hi - rel_lo  =  0.95 - 0.62   =  0.33
sim_hi - sim_lo  =  1.00 - 0.846  =  0.154

lam*  =  0.154 / (0.33 + 0.154)   =  0.154 / 0.484  =  0.318

at lambda = 0.75 (shipped):  clone 0.75(0.95) - 0.25(1.000)  =  0.4625
                             90447 0.75(0.62) - 0.25(0.846)  =  0.2535

Every clone outranks the one different product until lambda drops below 0.318 — and lambda = 0.318 means 68% of the weight goes to diversity, which shreds relevance on the slates that were never redundant in the first place.

The reason the crossover always lands down there is structural: relevance gaps between near-identical candidates are small (0.33 here) and redundancy gaps are order 1 (0.154 here only because the different SKU is itself visually close). lambda trades distinct-clusters-in-top-20 against the fast proxy metric and is worth tuning, but it is not a substitute for deduping on content. Tune it against both metrics jointly, at a value that leaves relevance intact.

import math


def dedupe_and_diversify(candidates, k=20, lam=0.75):
    """Collapse content-level duplicates, then MMR for visual diversity.

    `candidates` is an ordered list of dicts with keys: content_key,
    product_id, rel, vec. `vec` must be L2-normalized so the dot product is a
    cosine.

    **The dedupe key is `content_key`** -- the perceptual-hash / 0.97-cosine
    cluster id built in section 3.3 -- and NOT `product_id`. A product id is
    per-listing, so the same physical chair sold by five sellers carries five
    product ids and survives a product-id dedupe intact: section 9.1's top 20
    is 2 physical products and 7 product ids, so the guardrail metric reads 7
    and passes while the user sees two chairs.

    Two distinct mechanisms in sequence: dedupe removes the *same* product
    appearing many times (an indexing artifact), MMR removes *different*
    products that look the same (a genuine ranking choice). MMR does not
    substitute for the dedupe -- see `mmr_crossover`.
    """
    seen, pool = set(), []
    for c in candidates:
        if c["content_key"] not in seen:
            seen.add(c["content_key"])
            pool.append(c)

    selected = []
    while pool and len(selected) < k:
        best, best_score = None, None
        for c in pool:
            redundancy = max(
                (sum(a * b for a, b in zip(c["vec"], s["vec"])) for s in selected),
                default=0.0,
            )
            score = lam * c["rel"] - (1 - lam) * redundancy
            if best_score is None or score > best_score:
                best, best_score = c, score
        selected.append(best)
        pool.remove(best)
    return selected


def mmr_crossover(rel_hi, rel_lo, sim_hi, sim_lo):
    """The lambda below which MMR prefers the *different* item over the clone.

    Solve  lam*rel_hi - (1-lam)*sim_hi  =  lam*rel_lo - (1-lam)*sim_lo:

        lam* = (sim_hi - sim_lo) / ((rel_hi - rel_lo) + (sim_hi - sim_lo))

    Relevance differences are small and redundancy differences are order 1, so
    lam* sits far below the values anyone ships. That is the whole reason the
    dedupe key has to be right: diversity weighting cannot rescue it.
    """
    d_sim = sim_hi - sim_lo
    return d_sim / ((rel_hi - rel_lo) + d_sim)


# --- section 9.1's own worked failure, executed --------------------------
_CLONE = (1.0, 0.0)                                  # SKU 88213, every listing
_OTHER = (0.846, math.sqrt(1.0 - 0.846 ** 2))        # SKU 90447, cos 0.846
_TOP20 = (
    [{"content_key": "hash:88213", "product_id": "88213",
      "rel": 0.95, "vec": _CLONE} for _ in range(9)]
    + [{"content_key": "hash:88213", "product_id": "88213-seller-%d" % i,
        "rel": 0.95, "vec": _CLONE} for i in range(4)]
    + [{"content_key": "hash:88213", "product_id": "88213-crop",
        "rel": 0.95, "vec": _CLONE}]
    + [{"content_key": "hash:90447", "product_id": "90447",
        "rel": 0.62, "vec": _OTHER} for _ in range(6)]
)

assert len(_TOP20) == 20
assert len({c["product_id"] for c in _TOP20}) == 7    # what the shipped key saw
assert len({c["content_key"] for c in _TOP20}) == 2   # what the user sees

assert len(dedupe_and_diversify(_TOP20, k=20)) == 2

# MMR alone, on the product-id-deduped pool that shipped: six clones first.
_as_shipped = [dict(c, content_key=c["product_id"]) for c in _TOP20]
_slate = dedupe_and_diversify(_as_shipped, k=20, lam=0.75)
assert len(_slate) == 7
assert _slate[-1]["product_id"] == "90447"
assert all(s["vec"] == _CLONE for s in _slate[:6])

_lam_star = mmr_crossover(rel_hi=0.95, rel_lo=0.62, sim_hi=1.0, sim_lo=0.846)
assert abs(_lam_star - 0.3182) < 5e-4
assert dedupe_and_diversify(_as_shipped, k=2, lam=0.75)[1]["product_id"] != "90447"
assert dedupe_and_diversify(_as_shipped, k=2, lam=0.30)[1]["product_id"] == "90447"

9.2 The red dress that returns red curtains

This failure teaches the most transferable lesson in the chapter: a model uses whatever shortcut the training task leaves available, so the way to fix a bad representation is usually to change the task rather than the architecture.

QUERY   a red bodycon dress, photographed indoors

rank  item                          cos    category
  1   red midi dress                0.81   dresses
  2   red wrap dress                0.79   dresses
  3   crimson blouse                0.77   tops
  4   red velvet curtain panel      0.76   home/window
  5   red satin pillowcase          0.74   home/bedding
  7   red table runner              0.71   home/dining

The mechanism is not “the model is confused.” It is that the training task never required shape, in three steps that each look harmless:

1. The augmentation set had crops, flips, and blur, but no color jitter.
2. The negatives were random items from a 200M catalog, and a random item differs from the anchor in color 97% of the time.
3. So a representation that encodes only the dominant color histogram already solves the contrastive task to within a rounding error — and gradient descent, the procedure that finds the model’s weights by repeatedly stepping downhill on the loss, finds the cheapest sufficient solution rather than the most meaningful one.

The diagnostic is a linear probe on the frozen embedding: freeze the encoder, train a one-layer classifier on top of it to predict some property, and see how well that property can be read straight off the vector.

All three targets here are nominal — a colour bucket, a leaf category, a SKU id — so the probe metric is top-1 accuracy against the probe set’s chance rate, plus normalized mutual information (NMI), and not R^2.

NMI measures how much knowing the prediction tells you about the truth, scaled to sit between 0 and 1. Unlike accuracy, it is comparable across targets with different numbers of classes.

R^2 is the share of variance explained, which needs an interval scale where the distance between two label values means something. On a nominal target it is a function of the integer codes somebody assigned: permuting those codes moves R^2 while leaving the model, the predictions and the grouping untouched. The Python block at the end of this subsection executes exactly that permutation and asserts the three metrics behave as claimed.

Here is what the probe returns. Read each row against its own chance rate — that is the only way the three are comparable, and it is why the chance column is there at all:

linear probe on the frozen embedding

target                 classes   chance   top-1 accuracy    NMI
dominant colour bucket      12    0.083            0.87    0.71
leaf category            4,800    0.0002           0.52    0.44
SKU identity             1,000    0.001            0.22    0.19

State that result carefully, because the tempting phrasing is not a claim the numbers support. “The embedding is 87% a color descriptor” is a share-of-variance sentence — the thing R^2 means and the thing it cannot mean on a nominal target. The defensible version: a linear probe reads the dominant colour bucket off the frozen embedding 87% of the time against an 8.3% chance rate, and reads the SKU 22% of the time; NMI orders them the same way, 0.71 against 0.19. Note why NMI is on the table at all — 0.52 accuracy over 4,800 categories is a far larger lift over chance than 0.87 over 12, so accuracy is not comparable across targets of different cardinality and the conclusion needs a metric that is.

The fixes follow directly from the trace, and they are both about the task, not the architecture:

• Color-jitter the augmentations. This makes color an invariance, which removes it from the set of features that can solve the task. The model is then forced onto shape and texture. Watch the exact-product metric for products that differ only in colorway — this is the Framing similar is four different questions tension made concrete, and the resolution is to keep color as an explicit reranker feature (Retrieval ranking and the arithmetic that sizes them) where it is a decision rather than a shortcut.
• Mine attribute-stratified negatives. Negatives that share the anchor’s dominant color but differ in category. Now a color-only representation gets the answer wrong, so color-only is no longer a local optimum.

After both, on the same eval set and the same probe sets: colour accuracy 0.31 (NMI 0.24), category 0.74 (NMI 0.61), SKU 0.61 (NMI 0.55), and recall@100 on the low-light user-photo slice — Metrics’s 0.631 row, not the 0.703 traffic-weighted number and not the 0.6127 user-side one — goes 0.631 -> 0.752. That 0.752 is where 4a training the split the correction and the cadence’s decay series starts, which is why the cadence is derived on this slice rather than on the pipeline.

The general lesson, and the one to state in the interview: augmentations define what the model must ignore, and negatives define what it must distinguish. Any property that appears in neither is a property the model is free to use as a shortcut.

import math
from collections import Counter


def probe_accuracy(y_true, y_pred):
    """Top-1 accuracy. Report it against the probe set's chance rate, because
    0.52 over 4,800 classes and 0.52 over 12 are not the same claim."""
    n = len(y_true)
    return sum(1 for a, b in zip(y_true, y_pred) if a == b) / n if n else 0.0


def probe_nmi(y_true, y_pred):
    """Normalized mutual information, I(Y;Yhat) / sqrt(H(Y) H(Yhat)).

    The comparable-across-targets metric: it is in [0, 1] like R^2, it is
    defined for nominal targets unlike R^2, and unlike accuracy it does not
    reward a 12-way probe over a 4,800-way one for free.
    """
    n = len(y_true)
    if not n:
        return 0.0
    joint = Counter(zip(y_true, y_pred))
    py, pyh = Counter(y_true), Counter(y_pred)

    def entropy(counts):
        return -sum((c / n) * math.log(c / n) for c in counts.values())

    h_y, h_yh = entropy(py), entropy(pyh)
    mi = sum((c / n) * math.log((c / n) / ((py[a] / n) * (pyh[b] / n)))
             for (a, b), c in joint.items())
    return mi / math.sqrt(h_y * h_yh) if h_y > 0 and h_yh > 0 else 0.0


def r2_on_codes(y_true, y_pred):
    """R^2 -- the share of VARIANCE explained. Defined only when the target is
    on an interval scale, i.e. when the arithmetic distance between two label
    values means something. Applied to SKU ids or leaf-category ids it is a
    function of the integer codes somebody happened to assign."""
    n = len(y_true)
    mean = sum(y_true) / n
    ss_res = sum((a - b) ** 2 for a, b in zip(y_true, y_pred))
    ss_tot = sum((a - mean) ** 2 for a in y_true)
    return 1.0 - ss_res / ss_tot if ss_tot else 0.0


# --- why the metric had to change: R^2 is not relabeling-invariant -------
_TRUE = [0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2]
_PRED = [0, 1, 2, 0, 2, 1, 0, 1, 2, 1, 1, 2]       # a probe that gets 9 of 12
_RELABEL = {0: 0, 1: 2, 2: 1}                       # same partition, new ids
_TRUE_R = [_RELABEL[y] for y in _TRUE]
_PRED_R = [_RELABEL[y] for y in _PRED]

assert probe_accuracy(_TRUE, _PRED) == probe_accuracy(_TRUE_R, _PRED_R)
assert abs(probe_nmi(_TRUE, _PRED) - probe_nmi(_TRUE_R, _PRED_R)) < 1e-12
assert abs(r2_on_codes(_TRUE, _PRED) - r2_on_codes(_TRUE_R, _PRED_R)) > 0.10

# and the shape of the diagnostic: an embedding that is only a colour
# histogram reads out colour perfectly and identity at chance.
_COLOUR = [c for c in range(4) for _ in range(6)]
_SKU = list(range(24))
_FROM_COLOUR = list(_COLOUR)
assert probe_accuracy(_COLOUR, _FROM_COLOUR) == 1.0
assert probe_nmi(_COLOUR, _FROM_COLOUR) == 1.0
assert probe_accuracy(_SKU, [c * 6 for c in _FROM_COLOUR]) < 0.20

9.3 Embedding drift on reindex

This is the worst failure in the chapter, because it is both total and silent: search returns nothing useful, and every dashboard stays green. It is also the one an interviewer is most likely to ask about, so be able to walk through the incident.

INCIDENT
  encoder v2 deployed to the query path at 14:02
  index still holds v1 vectors for 62% of shards (rolling backfill)

  cos(v1(x), v2(x)) for the SAME image, measured on 1,000 items:  0.31

  recall@100 on the affected shards:   0.71 -> 0.04
  zero-result rate:                     0.3% -> 22%
  alert that fired:                     none for 41 minutes
                                        (latency normal, error rate 0, QPS normal)

A rolling backfill means replacing the index’s contents machine by machine while the system keeps serving, which is how almost every other kind of data migration is safely done and is exactly wrong here. Two encoder versions are two different vector spaces, not two nearby versions of one space. A retrain started from a different random seed lands on an unrelated set of axes; dimension 7 of v2 has nothing to do with dimension 7 of v1, and there is no rotation you can apply to reconcile them. The mechanics are identical to Embedding model version skew the nastiest, and the controls are the same three:

1. The model version is part of the index identity and part of every cache key. A query embedded with v2 must be structurally unable to reach a v1 shard.
2. Build the new index beside the old one and swap an alias atomically across all shards, or not at all. A rolling backfill of an embedding index is a bug, not a deployment strategy.
3. Alert on recall against a frozen probe set, evaluated continuously. 200 queries with known answers, run every minute against production. This is the only monitor that fires here — latency, error rate, and QPS all look perfect during the incident.

9.4 Cold start for new items

Cold start is the problem of serving something the system has no history for — and visual search only half has it. Retrieval is fine, ranking is not, and the fix belongs in the features rather than in the ranker.

Content-based retrieval has a genuine structural advantage worth naming: a new SKU is fully retrievable the moment its photo is encoded, which is not true of a collaborative-filtering system, meaning one that recommends by “people who liked this also liked that” and therefore knows nothing about an item nobody has interacted with yet. The cold start here is not in retrieval, it is in the reranker, whose most predictive features are engagement counters that are null for a new item. Feature gain below is how much of the model’s total predictive power each feature accounts for:

reranker feature importance (gain):
   ctr_30d              0.24     null for new items
   purchase_rate_30d    0.19     null
   return_rate          0.11     null
   embedding cosine     0.14
   attribute overlap    0.09
   ...
   -> 54% of the model's gain is unavailable on a new listing

A GBDT sends nulls down a default branch learned from training data, and that default is “behaves like a low-CTR item” — so new items are systematically ranked below where they belong, they get no impressions, they accumulate no engagement, and the cold start becomes permanent. The ch 01 feedback loop, in miniature.

Both fixes attack the null rather than the ranker, because the ranker is behaving correctly on the data it was given.

Shrink toward a category prior instead of imputing null. Shrinkage means blending an item’s own thin evidence with the average of the group it belongs to, weighted so that the group average dominates when the item has little data and fades out as the item accumulates its own:

ctr_hat = (clicks + alpha · ctr_category) / (impressions + alpha)

with alpha = 200 and ctr_category = 0.031, so alpha · ctr_category = 6.2:

   new item,  0 impressions      ->  ctr_hat = ctr_category exactly
   new item, 50 impr, 4 clicks   ->  (4 + 6.2)/(50 + 200)     = 0.041
   old item, 5k impr, 210 clicks ->  (210 + 6.2)/(5,000 + 200) = 0.042

The two shrunk numbers land within 0.001 of each other and the raw ones are 2x apart — the new item’s observed CTR is 4/50 = 0.080 against the established item’s 210/5,000 = 0.042. Fifty impressions is not evidence, and shrinkage is the estimator saying so in the same units the ranker consumes. alpha is exactly the number of impressions at which the item’s own data outweighs the prior — pick it by cross-validation, not by taste (Categorical encoding has the same shrinkage in target encoding). And these counters are the leakage trap 4a training the split the correction and the cadence’s temporal split is built around: computed at training time rather than as-of impression time, ctr_30d contains the click it is supposed to predict.

Reserve an exploration slot. One position in the top 20 goes to a randomly chosen eligible new item, with the propensity — the probability that item was chosen for the slot — written into the log. Because the choice was random rather than made by the ranker, this is the only data about new items that is not contaminated by the ranker’s own opinion of them, and it costs about 0.4% of conversion.

9.5 Query-side failures

The four failures above all live in the catalog or the model. This last group lives in the photograph itself, and it is where the largest single recall win in the chapter is available:

Summary

Every failure in this section reduces to the same three columns: what causes it, what would show it, and what stops it.

10. Alternatives considered and rejected

Every design has alternatives, the reasonable things you could have built instead. Here are ten, each with why it is attractive and the specific number that rules it out.

Four names in the table are worth defining first:

• CLIP is a widely available pretrained model trained on images paired with their captions. That gives it an excellent sense of what a picture is about and no sense at all of which particular chair it is. Zero-shot means using it with no training of your own at all.
• SIFT and ORB are classical, pre-neural methods that find distinctive keypoints in two images and check whether they line up geometrically.
• A cross-encoder is a model that reads the query and the candidate together in one pass. It is far more accurate than comparing two independently computed vectors, which is exactly why it cannot be indexed — there is no per-item vector to precompute.
• nprobe is IVF’s search-time dial: how many of its clusters to actually scan. More clusters means better recall at more time, and it costs no memory.

11. Interviewer pushback

The chapter closes as dialogue. Each question is one an interviewer asks, the italic line names what it is testing, and the answer is the shortest complete version of the relevant derivation above.

“You said you route to an exact-product lane above a threshold. What cosine do you ship, and where did it come from?” Testing: whether the architecture’s load-bearing number is a number or a Greek letter. 0.83, and it comes from a cost matrix rather than from a sweep. The decision is “declare this the same physical item,” so I need the two costs. A wrong one means the shopper buys, the box holds a different chair, they return it and stop trusting the feature — call it $14 of margin. A missed one means a real match goes to the style lane, where the shopper still sees it a row lower in a slate that converts worse — call it $0.60. C_fp / (C_fp + C_fn) is 0.959, so the system may not say “same item” below 96% confidence, and the 23:1 asymmetry is why there are two lanes at all: everything it refuses on still has to go somewhere. Then 0.959 is a probability and a cosine is not, so I bin the 3,000-query gold set by top-1 cosine and read off where the verified same-product rate crosses it — the 0.83-0.86 bucket sits at 0.962 and the one below it at 0.934, so 0.83. Two things I would volunteer. It is a property of the encoder, not the problem, so every model version reships its calibration table and the version pinning in Embedding drift on reindex covers the threshold too. And it is insensitive: halving the $14 moves the shipped cosine one bucket, to 0.80. I need the cost ratio to be tens rather than ones, not to three digits.

“Walk me through training. How do you split, and what does negative sampling do to your scores?” Testing: whether the loss you derived earlier ever becomes a run. Temporally and by content cluster, because there are two different leaks. Train on weeks 1-8, validate on 9, test on 10, gold set frozen out of all three. The content-cluster grouping is because a SKU with nine photos yields 36 pairs, and a random split puts one pair in train and a pair sharing an image in test, which measures memorization. The temporal part is for the reranker’s counters: ctr_30d computed today for a row from six weeks ago contains the click that is that row’s label. Rebuilding those point-in-time takes offline NDCG@10 from 0.71 to 0.58, and the 13 points that disappear were never real. On negatives, the correction I would not skip is logQ — subtract log Q(i) from the logit, where Q is the item’s rate of appearing as an in-batch negative, estimated as a decayed streaming count. Head-to-tail impression share spans about 1,000x here, so the correction spans log(1000) = 6.9 nats, and at temperature 0.07 that is 0.48 of cosine — six times the entire 0.83-to-0.75 band the router lives in. Without it the same cosine means different things for a head and a tail item and one global threshold is not a legal object. What logQ does not fix is a tail item with too few positive pairs to have learned an embedding; that is a separate failure with a separate fix on the positive side. Cadence I would derive: recall decays about 0.0012 a week, a point of recall is worth about $4,200 a week, one retrain-and-deploy is $1,136 of which only $297 is training, so k* = sqrt(2Y/Vr) is 2.1 weeks. Biweekly — though monthly costs only $221 a week more and halves the number of atomic index swaps, which is the operation Embedding drift on reindex is an incident about.

“Why does hard-negative mining matter? Isn’t a bigger batch the same thing?” Testing: whether you can do the gradient arithmetic. No, and the gap is four orders of magnitude. In an InfoNCE batch at temperature 0.07 with the positive at cosine 0.80 and 255 random negatives at 0.10, the positive’s softmax share is 0.9886, the loss is 0.0115, and each negative carries a gradient weight of 4.5e-5. Add one mined negative at cosine 0.75 and its weight is 0.33 — about 7,300 times a random one — while the loss jumps 35x to 0.406. A bigger batch multiplies the negative-side gradient sublinearly in the count — 0.0114 to 0.1568, about 14x — so 256 to 4,096 buys 14x for 16x the memory, against 29x for the one mined negative. Mining buys thousands per negative for the price of an index rebuild. The catch I would raise unprompted is that the hardest mined negative is the same product about 16% of the time, and it carries the biggest weight in the batch — so I sample from a semi-hard band and dedupe the catalog before mining.

“Users say a red dress query returns red curtains. Fix it.” Testing: whether you diagnose or guess. First I would run a linear probe on the frozen embedding: predict dominant colour, leaf category, and SKU. All three are nominal targets, so the metric is top-1 accuracy against the probe set’s chance rate — not R^2, which is a share of variance and is undefined here; permuting the SKU integer codes changes an R^2 and changes nothing about the model. If the colour bucket comes back at 0.87 accuracy against an 8.3% chance rate while SKU comes back at 0.22, and NMI orders them the same way at 0.71 against 0.19, the embedding is mostly a colour histogram and the reason is in the task, not the model. Random negatives from a 200M catalog differ in color 97% of the time, so a color-only representation already solves the contrastive objective — and there was no color jitter in the augmentations forcing invariance. So the fix is two changes to the task: color-jitter the positives, which makes color an invariance and forces the model onto shape, and mine negatives that share the anchor’s color but differ in category, which makes color-only wrong. Then I put color back as an explicit reranker feature, so it is a decision I control rather than a shortcut the model found. The general form: augmentations define what the model ignores, negatives define what it distinguishes, and anything in neither is available as a shortcut.

“How big is the index and what does it cost?” Testing: arithmetic and where the money is. 200M products, 512-dim fp16, HNSW at M=16 is 1,172 bytes per vector — 1,024 of payload, 128 for layer-0 links, and change for upper layers and bookkeeping — so 234 GB. The number people stop at is four shards of 59 GB, and that layout has three weeks of life in it: at 3% weekly delisting, HNSW tombstones fill the 9% headroom on a 64 GB machine in 3.1 weeks, and a year uncompacted is 61% dead nodes and 600 GB. So it is five shards of 47 GB replicated three times, fifteen machines at $0.62 an hour, $6,700 a month, with one shard compacted behind the alias every five days — 25 minutes on three spare machines, about 78 cents. Serving is 25 QPS at peak, so the machines exist entirely for RAM, and 88% of the bill is holding an index that answers 25 queries a second; the rest is a reserved half-H100 at $2.50 a GPU-hour for $900, and about two dollars of churn re-encode. That tells me the optimization: binary quantization with an exact rescore of the top 200. It compresses the payload only — links are incompressible — so 1,172 goes to 212 bytes, 42 GB, a single shard on three machines instead of fifteen. I would price the term people drop, which is that the rescore needs the full vectors: 205 GB of fp16 on local NVMe, $16 a month and about 2 ms of random reads. So $2,258 against $7,603, a 3.4x win, and it costs two points of index recall — 0.96 rescored against 0.98 — not zero.

“You deployed a new encoder and search broke. What happened and why did nothing alert?” Testing: whether you know embedding versions are not backward compatible. Two encoder versions are two different vector spaces, not nearby versions of one. Cosine between v1 and v2 embeddings of the same image is about 0.31 — no rotation relates them. So during a rolling backfill, queries embedded with v2 hit shards holding v1 and recall goes from 0.71 to 0.04 on those shards. Nothing alerted because latency was normal, the error rate was zero, QPS was normal, and the system returned results — just wrong ones. The three controls: the model version is part of the index identity and every cache key, so a v2 query cannot structurally reach a v1 shard; the new index is built alongside and swapped by an atomic alias flip across all shards or none; and a frozen probe set of 200 queries with known answers runs every minute, which is the only monitor that fires during this incident.

“CTR went up 6% and the team wants to ship. Any concerns?” Testing: whether you know the metric can improve while the product gets worse. Yes, and I would check one thing first: distinct content clusters in the top 20 — clusters, not product ids, and that distinction is the whole check. Near-duplicate flooding raises CTR@1, because twelve catalog photos of one appealing chair make a very clickable first result, and it destroys the session because the user has nothing to compare. I have seen the shape where two physical products occupy twenty slots and CTR@1 reads 0.41 — and where the guardrail read 7, because four sellers listing that chair are four product ids and a re-crop is a fifth, so the metric and the dedupe both have to key on an image hash. So CTR is a guardrail here, not the target. The primary is attributed purchase within seven days, and the fast proxy I would actually ramp on is the rate of sessions with a click and no reformulation inside 60 seconds — that reads out in about seven hours at 600k searches a day, against roughly twenty times that for the purchase metric.

“Recall@100 is 0.91 offline. What is it in production?” Testing: whether you know your eval set is not your traffic. Probably around 0.70, and the difference is query provenance. A gold set built from catalog images measures 0.912; user phone photos in good light are 0.784, low light or motion is 0.631, and an uncropped photo with three objects in it is 0.417. Weighted by 70% user photos, that is 0.703 — so the offline number overstates the shipped system by 21 points. Two fixes: train on the user-photo-to-purchase pairs, which are the real query distribution and cost nothing to harvest, and put an object detector and crop in front of the encoder, which is 18 ms and worth 24 points on the cluttered slice. And I would never report a single recall number again — the report is a table sliced by provenance.

“How do you get labels without paying anyone?” Testing: whether the data plan is real. The workhorse is same-SKU multi-photo pairs: 200M products at about 4.6 catalog photos each is 920M photos, and 4.6 × 3.6 / 2 = 8.3 pairs per SKU is roughly 1.7 billion positive pairs — a lower bound rather than an estimate, since the pair count is convex in the photo count, so the spread across SKUs only adds. The probability of a pair actually being the same product is 0.99, and it teaches viewpoint, lighting, background, and scale. Then user photos linked to a purchase, which is the only signal in the real query distribution. I would deliberately not use co-view pairs as positives here — measured, P(same product | co-viewed) is 0.34 and P(same category) is 0.79, so co-view is a category signal, correct for the style lane and two-thirds wrong for the exact-product lane. Human labeling goes into exactly one place: 3,000 exhaustively verified query photos as the eval set, about $9k, because recall is uncomputable without a complete denominator.

“New listings never get shown. Why, and what do you do?” Testing: whether you can see a feedback loop. Retrieval is fine — a content embedding is available the moment the photo is encoded, which is the structural advantage of visual search over collaborative filtering. The problem is the reranker: 54% of its gain sits in 30-day CTR, purchase rate, and return rate, all null for a new listing. A GBDT sends nulls down a learned default branch, and that default behaves like a low-CTR item, so the listing ranks low, gets no impressions, accumulates no engagement, and stays new forever. Two fixes. Shrink toward a category prior instead of imputing null — (clicks + 200·ctr_cat)/(impressions + 200) degrades smoothly and equals the prior at zero impressions. And reserve one slot in the top 20 for a random eligible new item with the propensity logged, which costs about 0.4% of conversion and is the only unbiased data about new items I will ever have.

Next: 03 — Street View Blurring — an offline pipeline where the operating point falls out of a legal cost asymmetry, and mAP is the wrong number to report.