06 — Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) answers a question by first retrieving relevant passages from your own documents, then including those passages in the model’s prompt. The answer is built from text the model reads at query time, rather than from what it learned during training.

This chapter takes the system view: not the retrieval algorithm, but the running service around it:

• the pipeline that turns a changing set of files into a searchable index,
• what that index costs in memory and in dollars,
• and the failures that produce fluent, confidently wrong answers while raising no error.

After this chapter you should be able to:

1. Size a retrieval index in gigabytes and dollars.
2. Derive the time between saving a document and that document becoming findable.
3. Choose between the two dominant index structures on the columns that decide it.
4. Keep one customer’s documents out of another’s answers by construction, not by remembering to pass a filter.
5. Name the detector for each silent failure.

Every failure in this chapter is silent: no exception, no error log, no latency spike. A system with no detector for them looks healthy until a customer finds their contract text inside someone else’s answer.

Terms are defined at first use. Where a mechanism from elsewhere in the series matters, it is restated briefly and linked.

How this chapter differs from the agent chapter

Two chapters cover retrieval without overlapping. It is worth knowing which one a given question belongs to.

Chapter 05 of the agents track covers retrieval from the agent-loop view — retrieval as a tool the model chooses to call, possibly several times with refined queries and possibly not at all. It answers “what should the model do?”: when a lookup is worth doing, how to merge two rankings, how to cut documents into passages, and when to reach for GraphRAG.

(GraphRAG is an index built over an extracted entity graph rather than over passages. It exists so that questions whose answer lives in no single passage — “what themes recur across all incident reports?” — can be answered by aggregating over the graph instead of by fetching a passage that does not exist.)

This chapter answers a different question: “what does the service look like, and who gets paged when it breaks?” That means the indexing pipeline as a running system, how fast a change becomes visible, what is physically inside a vector index, how to split one across machines, how to keep customers apart, and the cost model in dollars.

The test for which chapter a question belongs to is whether it has an on-call rotation attached to it. Chapter 05 never opens the index; this chapter never decides whether to search. Where the same idea appears in both, this chapter gives the price and the failure mode and links back for the mechanism.

What goes in and what comes out

This system has two paths, running on different clocks:

• Index time runs once per document version. It is allowed to take seconds.
• Query time runs once per question. It must complete within a human’s patience budget.

Almost every design decision in this chapter belongs to one path or the other.

Index time: a document in, searchable rows out

The block below shows one document going through the slow path. One input file becomes eleven independent rows, and each row carries bookkeeping fields that later sections depend on.

IN    doc_id "refund-policy", version 7, 40 KB of PDF exported from Confluence

OUT   11 rows in the index, one per chunk of the document:

      refund-policy#0   "Refunds are available within 30 days of..."   [0.031, -0.114, ...]
      refund-policy#1   "To start a refund, contact support and..."    [0.088,  0.002, ...]
      ...
      refund-policy#10  "Enterprise agreements may override this..."   [-0.007, 0.240, ...]

      Each row carries: the chunk id, the chunk text, a 1,024-number vector,
      and the stamps that make the rest of this chapter possible —
      corpus_version, tenant_id, ingested_at, source_modified_at,
      and the id of the embedding model that produced the vector.

Query time: a question in, a cited answer out

The fast path. Retrieved passages come back scored and ranked, and the final answer carries a bracketed chunk id. That citation makes several of this chapter’s failure detectors possible.

Two numbers, both derived later: retrieval on this path is 1.5% of the response time (Reranking in the serving path) and 11% of the cost (Query time per 1000 queries). Generation takes the rest, which is the opposite of where most teams spend their attention.

IN    "how long do I have to request a refund on an annual plan?"

      retrieved and ranked (scores are cosine similarities, defined below):
        refund-policy#3   0.71
        billing-faq#2     0.64
        refund-policy#8   0.58

OUT   "Annual plans may be refunded within 30 days of the renewal date,
       minus any usage [refund-policy#3]."

Everything in this chapter is one of three things: how those 11 rows are written and kept correct as the source document changes, what the structure holding those vectors costs, or how the system fails silently.

The vocabulary, defined once

About twenty terms do the work below. They are defined here and can be referred back to as needed.

The data

• Corpus — the collection of documents you are searching. The wiki, the PDF archive, the ticket history.
• Token — the unit models count text in, roughly four characters of English. A 500-token chunk is about 375 words.
• Chunk — one passage cut out of a document, a few hundred tokens long. Documents are too long to retrieve or read whole, so at index time each one is cut into pieces. That cutting is chunking, and it matters that retrieval returns chunks, never whole documents.
• Tenant — one customer of a system that serves many. The defining requirement: one tenant’s chunks must never reach another tenant’s answer.

Dense retrieval: searching by meaning

An embedding is a fixed-length list of floating-point numbers — 1,024 of them for a typical model. The model that produces it was trained on pairs of related texts, so that texts a human would call related come out pointing in similar directions.

The dimensions have no names. Nothing guarantees that dimension 47 means anything you could describe. It is called dense because nearly all 1,024 slots hold a nonzero number.

The procedure is three steps:

1. Embed every chunk once, at index time, and store the vector.
2. Embed the incoming question the same way, at query time.
3. Return the chunks whose vectors point most nearly the same direction as the question’s.

“Most nearly the same direction” is cosine similarity: the cosine of the angle between two vectors. It is 1.0 when they point the same way and 0 when they are unrelated, and it ignores vector length entirely. The store that holds those vectors and answers direction queries against them is a vector index.

(Chapter 05’s vocabulary section works cosine similarity through an arithmetic example if you want to see the substitution.)

Sparse retrieval: searching by words

The other family scores a chunk on which of the query’s exact terms appear in it, and on how rare those terms are across the corpus. It is called sparse because the natural representation is one slot per vocabulary word, and almost every slot is zero.

BM25 is the standard formula for that score. The name is “Best Matching 25”, the twenty-fifth ranking function in a line of information-retrieval research from the 1990s, and it is the default in Lucene, Elasticsearch and OpenSearch.

The two families fail in opposite directions, which is the whole reason both exist:

• Dense search answers “how do I get my money back” from a page that only ever says refund. Sparse search cannot, because no query word appears.
• BM25 finds ERR_4021 or tier-2 — exact strings that a dense embedding averages into noise. Dense search cannot, because averaging a passage into 1,024 numbers destroys rare literals.

Running both and merging the rankings is hybrid search. The standard merge is reciprocal rank fusion (RRF): score each result by 1/(k + its rank) in each list, then add the two scores. Notice what RRF does not use — the raw scores. It uses only positions, so it never has to make a cosine similarity and a BM25 score comparable, which they are not.

Approximate search, and the two things called “recall”

Scoring the query against all N stored vectors is exact, and its cost grows linearly with N. An ANN index — approximate nearest neighbour — is a data structure that inspects only a small fraction of the corpus and returns almost the true nearest vectors. At the scale priced in Why recall below 10 is usually fine that is about 340 times faster.

“Approximate” is literal, and two different metrics measure two different things. Confusing them is common, so separate them now.

Index recall is the fraction of the true nearest neighbours the approximate search actually returned. Recall 0.96 means 96 of every 100 true nearest neighbours were found and four were missed — with no error raised anywhere.

Recall@k is the fraction of test questions whose gold passage appears somewhere in the top k results. A gold passage is the chunk a human labelled as containing the answer.

An index can sit at 0.96 index recall while Recall@5 is 0.40. Returning the true nearest neighbours is no help when the nearest neighbours are the wrong passages.

Two further labelled metrics appear in Evaluation three layers and nobody builds the third:

• nDCG@k — normalized discounted cumulative gain. Sum each relevant result’s usefulness, discounted by how far down the list it sits, then divide by the score of the perfect ordering. 1.0 is perfect, and position is penalized.
• MRR — mean reciprocal rank. The average of 1 / (rank of the first relevant result). It cares only about how fast the user hits something useful.

Reranking: the second, slower pass

The retrieval stage has to run over millions of chunks, so it uses a bi-encoder: the query and the chunk are embedded separately and compared by cosine. That separation is what lets chunk vectors be computed once and stored forever.

Reranking is a second pass over the shortlist the first stage produced — take the top 50, score them properly, keep the best 5.

The scorer is a cross-encoder: it reads the query and one chunk concatenated together, in a single forward pass, so every query word can attend to every chunk word. Two consequences follow from that one difference:

• It is more accurate, because it sees the pair rather than two independently-made summaries of the pair.
• It cannot be precomputed, because the pair does not exist until the query arrives. That is why it is slow, and why it runs on 50 candidates rather than 10 million.

The pipeline and where it breaks explains the mechanism in more detail.

The generation side

• Grounding — every claim in the answer traces to a retrieved passage rather than to the model’s memory.
• Hallucination — a claim that does not so trace. Fluent, plausible, unsupported.
• Citation resolution — the check that each [refund-policy#3] the model emitted is a chunk id that (a) exists and (b) was actually in the context you gave it.

An unresolvable citation is the cheapest hallucination detector there is. That is why Half swapped alias after a reindex treats a citation lookup that quietly fails as a defect rather than an edge case.

The two version stamps, which are not the same stamp

Two version counters run through the whole system. They answer different questions, and swapping them breaks a cache in a way nothing reports.

corpus_version is what makes an alias flip atomic, and what Half swapped alias after a reindex asserts on at fusion time. Two rows with different corpus_version values came from two different builds and must never be ranked together.

A full rebuild moves both counters. One editor saving one page moves only index_version.

The retrieval cache needs index_version, and it needs the per-write one specifically. That cache is safe only because its key contains a value that changes whenever a cached result could become wrong (Caching three caches three risk profiles), and a single edited document is exactly such a change.

Key it on corpus_version instead and the cache goes on serving the pre-edit ranking until the next full rebuild — which is the answer cache’s failure with a different label on it. You can check which one your system does in five minutes: edit one document and watch whether the counter moves.

The operational words

• SLO — service level objective, the target you promise. “A new document is findable within ten seconds.”
• SLI — service level indicator, the measured number you compare against the SLO.
• p50 / p99 — percentiles of a latency distribution. p50 is the median; p99 is the value only 1% of requests exceed.
• Idempotent — doing it twice leaves the same result as doing it once. This is what makes retrying safe.
• Upsert — write a row, replacing it if it already exists.
• CDC — change data capture. Read a database’s own write log, so you learn about every insert, update and delete in order, instead of periodically re-scanning and guessing what moved.

1. Two numbers decide the architecture

The shape of the whole system follows from two facts about the corpus:

1. How fast must a new document become findable? This is the freshness SLO — the promise you are making about the gap between a write in the source system and that write being retrievable.
2. How big is the corpus, and how fast does it change?

Everything else is a consequence.

The table below maps answers to architectures. Read it as four rungs: you climb one only when the corpus or the SLO forces it, and each rung adds a system to operate.

Two of those rows use shorthand worth unpacking.

pgvector is a PostgreSQL extension that stores embeddings in an ordinary table column and searches them with ordinary SQL. The vector lives in the same database and the same transaction as the row it describes, which turns out to kill an entire bug class (Alternatives considered and rejected).

Quantized vectors means storing each number in fewer bits than a full 32-bit float. That is the subject of Hnsw memory per vector derived, and it turns out to be the second-largest cost lever in the system.

The first row is a real answer. Below about 100,000 tokens the entire corpus fits inside one model prompt. The cheapest correct retrieval system is no retrieval system.

Why the SLO decides more than the size does

A one-hour SLO lets you rebuild the index nightly and diff. That is one batch job. You need no streaming ingest, no online insert, and no version-stamped read path. (A read path is “version-stamped” when every row carries the corpus version it belongs to and queries filter on one version, so a query can never mix rows from two builds.)

A ten-second SLO forces all three of those into existence, as separate systems with separate failure modes.

Corpus size tells you how many machines. The SLO tells you how many systems.

A third question applies only sometimes: how many tenants? At 50,000 tenants the isolation model stops being a policy choice and becomes a memory-budget calculation (Sharding and multi tenancy).

What must be true for this table to be the right table

The table assumes three things: documents whose text is the thing being searched, arriving at a rate one pipeline can absorb, in a corpus where a single passage usually contains the answer. Each assumption breaks a different row.

If the questions are aggregate — “what are the recurring themes across all incident reports?” — then no single passage contains the answer. Top-k retrieval cannot construct it out of passages that do not contain it. You need a second index over an entity graph, not a bigger one over passages (Advanced variants).

If the answers must be exact and current — order status, account balance — the right design is a database query against the system of record. Not an approximate search over a copy that goes stale on every write.

If the corpus is small but the query volume is enormous, the cost model in The cost model derived inverts: generation rather than indexing becomes the entire bill.

2. The indexing pipeline is a streaming system

The index-time path is the slow half of the two-clock system from the opening. Its central claim: the loop most people write first is the wrong shape, not merely a slow version of the right one. What matters is the unit of work, why one storage operation must be atomic, and what goes wrong when it is not.

Every tutorial writes this:

for doc in corpus:
    for chunk in split(doc):
        index.add(embed(chunk))

That is a batch job, and it is correct exactly once — at t=0. The moment the corpus changes, three things become true that a loop over a list cannot express:

1. You must know what changed. A delete is not the absence of an insert. A full re-scan cannot distinguish “this document was removed” from “the crawler missed it.”
2. You must be idempotent. Queues are at-least-once, workers crash mid-document, and a retry must not double-insert.
3. You must be ordered per document. Two edits to the same document in flight means the older one can land last.

So ingest is a stream processor. The unit of work is a document version; the operations are upsert and delete; the key is doc_id, with chunk ids derived deterministically from it.

Two terms the diagram assumes

Both are load-bearing for the rest of the chapter.

An index alias is a name that queries resolve through. Searches go to vectors-live, and vectors-live is a pointer that currently names the concrete index v9. Repointing that name is an alias flip, and the flip is atomic: every query before it reads v8, every query after it reads v9, and no query ever sees a half-built index.

A shadow index is a complete second index built quietly off to the side while the live one keeps serving. You build it, audit it, then flip the alias to it. The flip is the only moment the live system changes — which is what makes a bad rebuild something you discard rather than something you repair under pressure.

The pipeline

The diagram has two lanes and two destinations. Work enters at the top and splits into a green incremental lane and an orange backfill lane; the lanes stay separate until the shared parse-and-embed stages.

flowchart TD
    SRC[("Source of truth<br/>CMS · S3 · Confluence · DB")] -->|CDC / webhook| BUS["Change bus<br/>doc_id · version · op"]
    BACK["Backfill / reindex<br/>job"] --> Q2

    BUS --> Q1["Incremental queue<br/>priority lane"]
    BACK -.->|never shares a lane| Q1

    Q1 --> W1["Ingest workers<br/>pool A · 40"]
    Q2["Backfill queue<br/>rate-limited"] --> W2["Ingest workers<br/>pool B · 200"]

    W1 --> P["Fetch · parse · chunk"]
    W2 -->|"targeted re-ingest<br/>(a few thousand docs)"| P
    P --> E["Embed<br/>cache: sha256 text + model_id"]
    E --> U["Atomic replace by doc_id<br/>one write, not two"]
    U --> LIVE[("Live index<br/>corpus_version N")]
    W2 -->|"full reindex<br/>(whole corpus)"| SHADOW[("Shadow index<br/>corpus_version N+1")]
    SHADOW -->|recall audit passes| FLIP{{"Atomic alias flip<br/>vectors AND bm25"}}
    FLIP --> LIVE

    style Q1 fill:#2d6a4f,color:#fff
    style Q2 fill:#bc6c25,color:#fff
    style FLIP fill:#1d3557,color:#fff

Walk one document through it

An editor saves refund-policy in Confluence. Follow it node by node.

1. Source of truth. Confluence — or a CMS, an S3 object-storage bucket, or a production database, which is what the top node’s CMS · S3 · Confluence · DB lists — is where the document actually lives. The index is only ever a derived copy of it.
2. Change detection. A connector reading the database’s write log, or a webhook (an HTTP request the source system fires at you the moment something changes), emits one small message. That is the edge marked CDC / webhook.
3. Change bus. The message lands on a durable, ordered message log. It carries three fields and no content: doc_id, version, and op (upsert or delete). Keeping content off the bus is why this stage costs half a second rather than seconds.
4. Incremental queue (the priority lane). One of the 40 ingest workers in pool A picks the message up.
5. Fetch · parse · chunk. 2 seconds here, because the document is a PDF. HTML would be 0.05 s.
6. Embed cache. Keyed on sha256(chunk text) + model_id. It finds 8 of the 11 chunks already computed from the previous version, so only 3 chunks are actually embedded.
7. Atomic replace by doc_id. One write, not two — the reason is derived immediately below — into the live index, stamped with corpus version N.

The two things the diagram asserts that a box cannot say

First: the dotted arrow that means “never”. The dotted edge from the reindex job to the incremental queue is labelled “never shares a lane”. It is an arrow drawn to assert an absence: no backfill message may ever be enqueued into Q1, by construction rather than by convention. That is the fix derived in Freshness and the staleness window derived, drawn as negative space.

Second: the pool sizes look backwards, and are not. Pool A has 40 workers; pool B, the backfill pool, has 200. Five times as many workers — and yet it is the backfill queue that is rate-limited, and the limit is applied at the embedding tier, not at the worker tier.

The reason is which resource is actually scarce. Workers are cheap and spend most of their time waiting on network calls and PDF parsing. Embedding capacity is the shared resource both lanes contend for. Throttling workers would idle hardware without protecting anything; throttling embedding throughput protects exactly the thing that is contended.

The two destinations

Pool B has two outgoing edges because it serves two different jobs, and each edge says which.

A targeted re-ingest — a few thousand documents whose parser was wrong — goes down the shared parse/embed/upsert path into the live index. There is nothing to rebuild.

A full reindex builds a shadow index stamped corpus version N+1 instead. It does not go live until the recall audit of Evaluation three layers and nobody builds the third passes against it — the gate labelled “recall audit passes”. Only then does one atomic alias flip move the vector index and the BM25 lexical index together, never one and then the other. Half swapped alias after a reindex prices what happens when they move separately, in wrong answers.

The lexical index is invisible in the drawing because it shares the parse stage entirely: it is built from the same parsed text, against the same corpus version, and lives behind the same flip.

The assumptions this pipeline is built on

Three, and each one is a real deployment somewhere.

1. The source of truth can tell you what changed, through CDC or a webhook. If it cannot, t_detect becomes a full re-scan interval — the freshness arithmetic in Freshness and the staleness window derived picks up a term two orders of magnitude larger than every other term combined. Worse, deletes stop being detectable at all, because a re-scan cannot distinguish a removed document from one the crawler missed.

2. Documents are individually re-processable. If a “document” is really a 400-page manual whose chunks only make sense with the whole thing in hand, per-document atomic replace is still right, but the parse cost stops amortizing and the unit of work has to become a section.

3. The embedding function is pure and stable for a given model id. That is what makes the cache free money and the retries safe. It stops being true the moment a hosted provider silently updates a model behind an unchanged name — which is Embedding model version skew the nastiest arriving with no migration to blame.

What each stage costs, and how each one fails

The table prices every stage of the pipeline. Read the last two columns first. The bolded failures are the ones with no exception attached; everything else throws, gets retried, and never reaches a customer.

Three things in that table need reading keys.

P/2 in the detect row is the expected wait when you poll rather than subscribe. If you re-scan every P seconds and the write lands at a uniformly random moment inside a cycle, the average wait is half a cycle. Freshness and the staleness window derived substitutes P = 15 minutes = 900 s and gets 450 seconds.

The idempotency key column is the value that makes a retry of that stage a no-op rather than a duplicate. Retry the fetch, and the content hash tells you it is the same bytes. Retry the embed, and (chunk_hash, model_id) hits the cache instead of paying the model again.

The three bolded failures raise nothing. A missed delete leaves a document retrievable forever. A scanned PDF that parses to an empty string indexes zero chunks and reports success. Version skew ranks perfectly happily. Nothing in your error budget moves.

The rule that removes an entire bug class

Chunk ids are a deterministic function of (doc_id, chunk_index), and a document update replaces every chunk under that doc_id atomically.

Atomic here means the write either fully happens or does not happen at all, with no state in between that a query could observe.

The rule is emphatically not “upsert the chunks I produced this time.” That difference produces the chapter’s cheapest silent failure: an orphan chunk, a row still sitting in the index whose text no longer exists in any live document.

The trace below shows what the wrong version costs when a document gets shorter. Watch the chunk indices: v1 had 11 chunks, v2 has 9, and nothing ever removes the two left over.

t0  doc "refund-policy" v1  ->  11 chunks: refund-policy#0 .. #10
t1  doc edited and shortened ->  9 chunks: refund-policy#0 .. #8
    upsert overwrites #0..#8.  #9 and #10 are never touched.

t2  query "how long do I have to request a refund?"
    top-3:  refund-policy#9   0.74   (v1 text: "...within 60 days...")
            refund-policy#3   0.69   (v2 text: "...within 30 days...")
            billing-faq#2     0.61

    -> the model receives two contradictory passages carrying the SAME
       source id, from the SAME document, and picks one.
       No exception. No log line. The citation looks perfect.

Why the replace must be one call, not two

The obvious implementation of “replace everything under this doc_id” is delete_by_doc followed by insert. That is the right semantics and the wrong number of calls.

It is two writes. A worker that dies between them leaves the document absent — not stale, gone.

Compare the two failures:

Absent is strictly worse. A stale passage at least answers the question, with old text. A missing document answers nothing and reports no error, and it is indistinguishable from a crawler that never saw the file.

Queues retry, so the hole does close eventually. But the staleness window for that document is however long the retry takes — unbounded if the worker died with the message still in flight.

So the replace has to be one operation. That is a requirement on the storage layer, not something the caller can arrange.

The argument, made executable

The listing below proves the claim instead of asserting it. It has three parts:

1. ingest() — the real function, with the at-least-once guard and the one atomic write.
2. FakeIndex — a deliberately fake store whose only job is to let a worker crash at the precise instant the new chunks are being written.
3. The assertions at the bottom — the actual claims.

Read the assertions first. They say that after the crash the document is stale rather than absent, that the retry re-runs, and that replaying the same version a third time changes nothing. Everything above them is scaffolding that makes those three lines executable.

import hashlib


def chunk_id(doc_id: str, index: int) -> str:
    """Deterministic and stable across re-ingests of the same document."""
    return f"{doc_id}#{index}"


def content_key(text: str, model_id: str) -> str:
    """Embedding cache key. The model id is IN the key, not beside it."""
    h = hashlib.sha256(text.encode("utf-8")).hexdigest()
    return f"{model_id}:{h}"


def parse(raw: bytes) -> str:
    return raw.decode("utf-8")


def structural_chunks(text: str) -> list:
    """Split where meaning already breaks, never mid-sentence (§10.3)."""
    return [p.strip() for p in text.split("\n\n") if p.strip()]


def ingest(doc_id: str, version: int, raw: bytes, store, index,
           embed, cache, model_id: str) -> int:
    """At-least-once safe. Replaying this with the same version is a no-op."""
    if store.applied_version(doc_id) >= version:
        return 0                                    # stale or duplicate delivery

    chunks = structural_chunks(parse(raw))
    vectors = []
    for i, text in enumerate(chunks):
        key = content_key(text, model_id)
        vec = cache.get(key)
        if vec is None:
            vec = embed(text)
            cache.set(key, vec)                     # pure function, safe forever
        vectors.append((chunk_id(doc_id, i), text, vec))

    # ONE call, and it must be atomic. Replace-by-doc, not a bare upsert (the
    # old version may be longer) and not delete-then-insert (a crash between
    # them deletes the document). mark_applied comes after: a crash in the gap
    # replays a replace that is idempotent for this version.
    index.replace_doc(doc_id, vectors, corpus_version=store.corpus_version)
    store.mark_applied(doc_id, version)
    return len(vectors)


# --- the crash, injected at the instant the new chunks are landed ----------

class Crash(RuntimeError):
    """A worker dying before its write is durable."""


class FakeIndex:
    """Both storage contracts, so the fault can be injected at one point.

    `_write` is the only mutation. The fault fires on the write that lands the
    new chunks — the same logical instant under either contract. The only
    difference is whether the old rows are already gone by the time it fires.
    """

    def __init__(self, rows, crash_on_landing=False):
        self.rows = dict(rows)
        self.crash_on_landing = crash_on_landing

    def _write(self, rows, landing: bool):
        if landing and self.crash_on_landing:
            raise Crash("worker died before the new chunks were durable")
        self.rows = rows

    def _without(self, doc_id):
        return {k: v for k, v in self.rows.items()
                if not k.startswith(f"{doc_id}#")}

    def replace_doc(self, doc_id, vectors, corpus_version):
        staged = self._without(doc_id)
        staged.update({cid: text for cid, text, _ in vectors})
        self._write(staged, landing=True)           # all of it, or none of it

    def delete_by_doc(self, doc_id):                # the non-atomic pair,
        self._write(self._without(doc_id), landing=False)      # kept to show

    def insert(self, vectors, corpus_version):      # what it costs
        staged = dict(self.rows)
        staged.update({cid: text for cid, text, _ in vectors})
        self._write(staged, landing=True)


class FakeStore:
    corpus_version = 4471

    def __init__(self):
        self.applied = {}

    def applied_version(self, doc_id):
        return self.applied.get(doc_id, 0)

    def mark_applied(self, doc_id, version):
        self.applied[doc_id] = version


class FakeCache(dict):
    def get(self, key):
        return dict.get(self, key)

    def set(self, key, value):
        self[key] = value


def embed(text):
    return (len(text) / 100.0, 0.2)


V1 = {"refund-policy#0": "Refunds are available within 60 days.",
      "billing-faq#0": "Invoices are issued monthly."}
V2 = b"Refunds are available within 30 days.\n\nContact support to start one."

store, cache = FakeStore(), FakeCache()
store.mark_applied("refund-policy", 1)
index = FakeIndex(V1, crash_on_landing=True)

try:
    ingest("refund-policy", 2, V2, store, index, embed, cache, "embed-v2")
    raise AssertionError("the fault injector never fired")
except Crash:
    pass

# STALE, which is the failure this section is defending against. Not ABSENT,
# which is what delete-then-insert produces at this same instant.
assert index.rows["refund-policy#0"] == "Refunds are available within 60 days."
assert store.applied_version("refund-policy") == 1       # so the retry re-runs

index.crash_on_landing = False
assert ingest("refund-policy", 2, V2, store, index, embed, cache, "embed-v2") == 2
assert index.rows["refund-policy#0"] == "Refunds are available within 30 days."
assert index.rows["billing-faq#0"] == "Invoices are issued monthly."          # untouched
assert ingest("refund-policy", 2, V2, store, index, embed, cache, "embed-v2") == 0

Revert that one call to the delete_by_doc / insert pair and the first assertion raises KeyError: 'refund-policy#0' — the crash took the document out of the index entirely.

What the storage layer has to give you

The rule above is only enforceable if the store underneath you offers the right primitive — so here is what to ask a vendor for, and what to do when the answer is no.

replace_doc is not a convenience wrapper. It is the interface where the atomicity requirement lives, and what it costs depends entirely on what is underneath it.

The table gives the three cases you will actually meet. GC in the middle row is garbage collection — a background job that reclaims the storage of the old version once no reader can still be looking at it.

If you can only pick the ordering, pick insert-then-delete. Stale-and-duplicated is a ranking problem you can detect and repair; absent is a silent hole that looks exactly like “the crawler never saw this document.”

3. Freshness, and the staleness window derived

“How fresh is the index?” can be turned from a promise into a number you can derive. One of the seven terms in that sum dominates all the others combined.

Staleness window W: the elapsed time from a write in the source system to the moment a query can retrieve the new content.

It is a sum, because each stage must finish before the next begins:

W = t_detect + t_queue + t_parse + t_chunk + t_embed + t_upsert + t_refresh

Six of those terms are the pipeline stages of The indexing pipeline is a streaming system, in order.

The seventh, t_refresh, is the one people forget: the delay between a write being durable in the index and that write being visible to a search. It is real in every system that batches writes into segments before making them searchable, and it is a hard floor you cannot optimize below.

Substituting real numbers

Two configurations below, running identical code, differing only in how the change is detected. Every term after t_detect is the same number in both columns — that is the point of the layout, so read across the rows.

PUSH  (webhook or CDC)                 PULL  (poll every 15 min)
  t_detect    0.5 s                      t_detect   450 s   (P/2 expected)
  t_queue     0.3 s                      t_queue      0.3 s
  t_parse     2.0 s   (PDF)              t_parse      2.0 s
  t_chunk     0.01 s                     t_chunk      0.01 s
  t_embed     0.6 s   (0.5 batch wait)   t_embed      0.6 s
  t_upsert    0.05 s                     t_upsert     0.05 s
  t_refresh   1.0 s   (visibility)       t_refresh    1.0 s
              ------                                ------
    W_p50  =  4.5 s                        W_p50 = 454 s  (7.6 min)
                                           W_max = 904 s  (15 min)

The polling interval dominates every other term by two orders of magnitude: 450 seconds of detection against 3.96 seconds of everything else combined.

Shaving 200 ms off embedding latency is meaningless while you poll every 15 minutes. Freshness is one architectural decision, push or pull, not a tuning exercise.

What has to be true for those columns to hold

The push column assumes the source emits an event for every write, including deletes, and that the event is durable. A webhook dropped during a deploy is a document that is silently never updated, and the only way you find out is the staleness measurement in Evaluation three layers and nobody builds the third.

It also assumes the average document is a 2-second PDF. A corpus of HTML pages parses in 0.05 s, and the whole push column collapses toward t_refresh — at which point index visibility, not parsing, is your freshness floor.

The pull column assumes writes arrive uniformly at random inside a polling cycle. That uniformity is what makes the expected detection wait P/2. A corpus edited in a burst every morning is much worse than the average suggests, because most of the day’s writes land right after a scan has finished.

The queue term is the one that explodes

Every term above is a steady-state number. One term is not bounded in steady state, and it is the term that actually breaks the SLO in production.

Under steady state a queue is a non-issue, and the standard model says so.

M/M/1 names the simplest queueing model: random arrivals, random service times, one server. Three symbols:

• lambda — the arrival rate, in documents per second.
• mu — the service rate the pipeline can sustain, in documents per second.
• rho = lambda/mu — utilization, the fraction of capacity in use. At rho = 0.95 the pipeline is running at 95% of what it can handle.

The expected waiting time is W_q = rho / (mu · (1 - rho)). Substitute two utilizations at the same service rate:

W_q = rho / (mu · (1 - rho))

rho = 0.20, mu = 20 docs/s   ->  0.20 / (20 x 0.80)  =  0.0125 s
rho = 0.95, mu = 20 docs/s   ->  0.95 / (20 x 0.05)  =  0.95 s

Both are fine: 12 milliseconds at a fifth of capacity, and under a second even at 95% utilization.

The (1 - rho) in the denominator is what makes the number explode as rho approaches 1. Push rho to 1.0 and the wait goes to infinity, because work arrives exactly as fast as it can be cleared and the backlog never drains. A reindex drives rho past 1 on purpose.

Now someone triggers one, against a system that has not been split into two lanes — a single shared pool of 40 workers serving both incremental updates and the backfill. That is the configuration everybody starts with.

bulk reindex enqueues            2,000,000 documents
40 workers x 0.5 doc/s (PDF-heavy)  =  20 docs/s
drain time = 2e6 / 20  =  100,000 s  =  27.8 hours

For those 27.8 hours, the staleness window for every incremental update is also 27.8 hours, because incremental updates sit in the same queue behind two million backfill messages.

Measure that against the 4.5-second push median derived above: 100,000 s / 4.5 s ≈ 22,000. This is not missing the target by 1%. It is missing it by a factor of 22,000.

And the symptom users report is “search is stale,” not “the backfill is running.” So it gets diagnosed slowly, by the wrong team.

The fix is structural, not a bigger worker pool — doubling the workers halves 27.8 hours to 13.9, which is still four orders of magnitude past the target:

• Two queues with separate worker pools. Incremental never waits behind backfill, by construction. This is the two-lane split drawn in The indexing pipeline is a streaming system, and it is why the backfill pool there can be 200 workers while incremental keeps its own 40: once the lanes cannot mix, making the backfill faster no longer makes freshness worse.
• A rate limit on the backfill lane so it cannot consume more than a fixed share of embedding capacity, which is the resource the two lanes still share even after the worker pools are separated.
• Backfills write to a shadow index and swap (Half swapped alias after a reindex) — the build-audit-flip sequence defined in The indexing pipeline is a streaming system — so a full rebuild never contends with the live index at all, which also means a bad backfill can be discarded instead of repaired.

4. Vector index internals

Now open the box the previous sections have been writing into. Three structures compete:

• Flat — an exhaustive scan (Flat the baseline nobody prices).
• IVF — a partitioned scan (Ivf partition and probe).
• HNSW — a graph walk (Hnsw memory per vector derived).

By the end you should be able to price each one in bytes per vector, choose between them on the two columns that decide it, and say when approximate search is the wrong tool. The choice, derived in The comparison with the columns that actually decide it, reduces to:

flowchart TD
    A["Choosing a vector index"] --> B{"Fewer than about 1M vectors?"}
    B -->|Yes| FLAT["Flat: exact scan, recall 1.0, no tuning"]
    B -->|No| C{"Continuous writes and deletes?"}
    C -->|Yes| HNSW["HNSW: cheap online insert, plus scheduled compaction for deletes"]
    C -->|"No, rebuilt periodically"| IVF["IVF: cheap rebuild via k-means"]

Two symbols run through the whole section. N is the number of vectors in the index. d is the dimension — the count of numbers in each vector. The running example is N = 10M and d = 1024 throughout, so every figure below is comparable to every other one.

4.1 Flat — the baseline nobody prices

The simplest index is no index at all, and its price is the baseline for everything that follows.

A flat index just stores the vectors in a list and, at query time, computes cosine similarity against every one of them. There is nothing to build and nothing to tune.

Its cost is therefore not compute but memory traffic. The machine has to read every stored byte in order to score it, so:

latency = (bytes to read) / (how fast memory can be read)

Substitute the running example. fp32 means each number is stored as a 32-bit floating-point value, which is four bytes:

N = 10M, d = 1024, fp32
bytes scanned per query  =  10e6 x 1024 x 4  =  41 GB
at 100 GB/s effective bandwidth              =  410 ms per query

Scale that down and it stops being alarming. At 100k vectors — a hundredth of the corpus — the same arithmetic gives 410 / 100 = 4.1 ms, which is completely fine.

Flat is not a toy. It is the correct answer below roughly 1M vectors, and it is the only index whose recall is exactly 1.0. Reaching for HNSW at 200k vectors buys you a build step, a tuning surface, and a delete problem, in exchange for latency you did not need.

The assumption underneath it is the bandwidth figure: 100 GB/s of effective memory bandwidth. Every flat latency in this chapter scales inversely with it, so it is the number to challenge first.

Effective, not peak. A random-access scan on a machine whose specification sheet claims 400 GB/s will not get 400. And if the vectors do not fit in RAM at all, so the scan touches disk, the figure changes by an order of magnitude and flat stops being viable far below 1M vectors.

4.2 IVF — partition and probe

The first real index buys speed by looking at only part of the corpus. The question is how much it skips.

IVF stands for inverted file index, a name inherited from text search.

Build time. Run k-means — the standard clustering algorithm, which finds k centre points and assigns every vector to its nearest one — over the corpus. That produces nlist cells, and each vector is stored in the cell whose centre it is closest to.

Query time. Two steps:

1. Compare the query against the nlist centres only. That is a few thousand comparisons, not ten million.
2. Exhaustively scan the nprobe cells whose centres came out nearest. Ignore every other cell entirely.

Insert time. A new vector is assigned to its nearest existing centroid — one comparison against nlist centres, then a list append. That is what “cheap online insert” means in the comparison table below.

Now price it. Two parameters matter: nlist (how many cells exist, fixed at build time) and nprobe (how many you open per query).

nlist = 4·sqrt(N) = 4 x 3,162 ≈ 12,650
vectors scanned = N · nprobe / nlist

nprobe = 32  ->  10e6 x 32/12650  =  25,300 vectors  =  0.25 % of the corpus
latency      ≈  0.0025 x 410 ms   =  1.0 ms

Read the last line as a fraction applied to flat’s cost. You scan 0.25% of what flat scans, so you pay 0.25% of flat’s 410 ms. That is 400 times less memory traffic, and the 1.0 ms falls straight out of it.

nprobe is free, and that is IVF’s real advantage

nprobe is a pure speed/recall dial with zero memory cost. That is why IVF survives in memory-constrained deployments, and the reason is worth stating rather than asserting.

nprobe is not a stored structure. It is a query-time count of how many cells to open. Raising it changes how much work one search does and changes nothing about what is on disk — so you can turn it up for a compliance sweep and back down for interactive traffic without rebuilding anything.

What IVF assumes is that the cell boundaries still fit the data. The centroids were fitted to the corpus as it looked when k-means last ran. As new documents arrive in regions the old centroids do not cover, cells grow lopsided and recall at a given nprobe drifts downward. The fix is re-running k-means: cheap, but still a rebuild.

4.3 HNSW — memory per vector, derived

The whole chapter’s cost model rests on one number: how many bytes one vector occupies in the graph index that most production systems use. This is the derivation worth being able to reproduce.

HNSW is a hierarchical navigable small-world graph. Ignore the name and picture the structure.

Every vector is a node. Each node is linked to a few of its nearest neighbours. Search is a walk: start somewhere, hop to whichever neighbour is closer to the query, repeat until no neighbour improves.

“Small world” is the property that makes the walk short. A graph of mostly-local links plus a few long ones has short paths between any two points — the way six degrees of separation works in a social network.

“Hierarchical” is the speed-up on top. Nodes are stacked into layers: a sparse top layer with few nodes and long links, down to layer 0, which contains every point. Search enters at the sparse top, greedily descends toward the query, and at layer 0 does a beam search of width efSearch — meaning it keeps the efSearch best candidates alive at once instead of committing to a single path.

The three parameters

The derivation below is unreadable without these.

• M — the maximum number of neighbour links each node keeps on the upper layers. M = 16 is the common default.
• Layer 0 gets 2M links per node, twice the connectivity of every other layer. This is a real property of the algorithm, not a fudge: layer 0 holds every point and is where the final beam search happens, so it needs more links to stay navigable. At M = 16 that is 2M = 32 links, which is where the 32 in the second line of the budget comes from.
• efSearch — the beam width at layer 0. A query-time dial with no memory cost, exactly like IVF’s nprobe and for exactly the same reason.

How many upper layers a node lives on

One more input, and it is the only line of the budget that needs real work.

Each node’s level is drawn from an exponentially decaying distribution. That is what makes higher layers geometrically sparser, and it gives:

P(level >= l) = M^-l

Substitute M = 16 and read it in words:

P(level >= 1) = 16^-1 = 1/16    = 0.0625      one node in 16 reaches layer 1
P(level >= 2) = 16^-2 = 1/256   = 0.0039      one node in 256 reaches layer 2
P(level >= 3) = 16^-3 = 1/4096  = 0.00024     one node in 4,096 reaches layer 3

The expected number of levels a node has above layer 0 is the sum of those probabilities — a geometric series:

E[levels above 0] = 1/16 + 1/256 + 1/4096 + ...
                  = sum_{l>=1} M^-l
                  = 1/(M-1)
                  = 1/15
                  = 0.067

So on average a node carries 0.067 upper layers, each holding M links of 4 bytes each. That is where the tiny 4-byte line below comes from: upper layers are nearly free, and almost the entire graph cost is layer 0.

The budget

Now add it up, with ids stored as 4-byte integers:

vector payload          d x 4 bytes            1024 x 4        =  4,096 B
layer-0 neighbours      2M ids x 4 bytes       32 x 4          =    128 B
upper-layer neighbours:
    level assignment gives P(level >= l) = M^-l, so
    E[levels above 0] = sum_{l>=1} M^-l = 1/(M-1) = 1/15 = 0.067
    0.067 x M x 4 bytes                        0.067 x 16 x 4  ≈      4 B
per-node bookkeeping (level, offset, external id)              ≈     16 B
                                                                  -------
                                              M = 16, d = 1024   4,244 B
                                                                ≈ 4.24 KB

Every KB and GB in this chapter is decimal — 1,000 and 1e9 bytes, the units storage and RAM are sold in. So 4,244 B is 4.24 KB. Dividing by 1,024 instead gives ~4.15, and mixing the two conventions is a 2.4% error per vector that compounds straight into the RAM bill. The 424 GB figure and every row of the quantization table below are decimal.

Now multiply the per-vector cost by the corpus:

10M chunks  x 4,244 B  =   42.4e9 B  =   42 GB
100M chunks x 4,244 B  =  424e9 B    =  424 GB

42 GB fits on one machine. 424 GB does not. That single computation is the entire reason quantization exists.

Quantization: the same vectors in fewer bits

Quantization means storing each of the d numbers in fewer bits and accepting a small error in the resulting similarity scores.

Four options appear in the table below. Read each one first, so the rows mean something:

• fp16 — a 16-bit float. Half the bytes of fp32, with a shorter mantissa (fewer significant digits).
• int8 scalar — maps each dimension onto 256 integer steps. To map the integers back to real values you need the minimum and maximum that dimension takes across the corpus, which is the “per-dimension min/max” note in the table.
• Matryoshka — not a compression scheme at all, but a training scheme, named after the nesting dolls. The model is trained so the first 256 of its 1,024 dimensions are already a usable embedding on their own. That is why truncating a Matryoshka embedding is legitimate and truncating an ordinary one is destructive: in an ordinary model the information is spread across all dimensions with no privileged prefix.
• Binary — keeps one bit per dimension, just the sign. Comparisons become bit operations, which is why it is fast, and most of the magnitude information is gone, which is why the raw recall is poor.

Every “Bytes/vector” figure below is the same budget re-run with a different payload line. The binary row, for instance, is 1024 bits / 8 = 128 B of payload plus the same 128 + 4 + 16 = 148 B of graph and bookkeeping, giving 276 B.

Every figure in the “100M chunks” column is bytes/vector x 100e6 at d = 1024. The binary row is 276 B x 100M = 27.6 GB, shown as 28 GB.

Both factors matter and neither is a constant of nature. Halve the dimension or halve the corpus and every number in that column halves. Carry the bytes/vector across problems, never the GB.

How to read the recall column

Read it as a set of differences, not as absolute numbers — and never against The comparison with the columns that actually decide it.

Every row here is measured at one fixed graph configuration: M = 16 held constant, and a single efSearch held constant across all five rows. That is deliberate. The only thing this table isolates is what changes when you change the representation.

That fixed beam width is not either of the two settings priced in §4.4. Those are M = 16, ef = 64 giving 0.96, and M = 32, ef = 128 giving 0.99. The 0.98 baseline here sits between them, which is the tell that it is a third operating point rather than a copy of one of theirs.

So the honest reading is a set of deltas against this table’s own baseline:

• fp16 costs nothing (0.98 to 0.98).
• int8 costs about one point (0.98 to 0.97).
• binary plus rescore costs about two points (0.98 to 0.96).

Those deltas transfer to other systems. The 0.98 does not. Comparing a number from this table against a number from §4.4 is comparing two different indexes, and it will mislead you.

Binary plus rescore, the interesting row

15x less RAM, and the recall mostly comes back. The mechanism is that the binary pass is only a candidate generator, not the final ranking.

Rescoring is the second step: take the shortlist the cheap representation produced, fetch the full-precision vectors for just those 200, and re-sort by their exact scores. That is roughly 2 ms of random reads from NVMe — the fast flash-storage interface modern SSDs use.

The point of the row is that the full vectors still have to live somewhere. You moved them out of RAM and onto storage that is far cheaper per gigabyte. You did not delete them.

The trade, stated plainly: 424 GB - 28 GB = 396 GB of RAM given back, in exchange for 2 ms of added latency and two points of recall.

4.4 The comparison, with the columns that actually decide it

Everything above priced one structure at a time. Put them side by side and the surprise is that the two columns everyone reads are the two that do not decide anything.

Two reading keys before the table.

A tombstone is a node marked deleted and filtered out of results, but still physically present in the graph and still used as a routing hop by searches passing through. It is the standard way graph indexes handle deletion, because genuinely removing a node would break the links of every neighbour that pointed at it. Deleted documents that stay retrievable prices what happens when tombstones accumulate.

O(N log N) in the build column is the usual notation for how build cost scales: proportional to the number of vectors times the logarithm of that number.

The two columns to actually read are the last two — online insert and delete. The latency column is nearly identical across every non-flat row, which is exactly why it decides nothing.

Choose HNSW when writes are continuous; choose IVF when the corpus is rebuilt periodically.

The reason is the build and delete columns, not the latency column:

• HNSW inserts online at 1.2 ms/vector, which is what continuous ingest needs — but it cannot truly delete without a rebuild (Deleted documents that stay retrievable).
• IVF needs a fresh k-means when the distribution drifts — but it rebuilds cheaply, and its deletes are no worse.

Published benchmarks plot recall against QPS — queries per second, the throughput the index sustains — and hide both of these. That is why people pick from the wrong two columns.

The one assumption to check with your vendor

“HNSW cannot truly delete” is stated above as a property of the algorithm. It is really a property of most implementations.

Whether a deleted node’s slot is ever reclaimed, and whether segment merges quietly compact the graph for you, differs between hnswlib, FAISS, Lucene’s HNSW and every managed service.

• If the service you are buying compacts on merge, the tombstone arithmetic of Deleted documents that stay retrievable does not apply to you, and you should stop budgeting for it.
• If it does not, that budget is mandatory.

This is the single question managed vector databases document least well. Ask it explicitly.

4.5 Why recall below 1.0 is usually fine

The moment you say “approximate”, an interviewer will object — and the objection deserves arithmetic rather than reassurance: losing 4 points of index recall does not cost 4 points of answer quality, because the ANN stage is not the last stage and the stage after it is also lossy.

The end-to-end quantity that matters is not “did the index return the true nearest neighbours”. It is whether the gold passage survives all the way into the handful of passages the model is shown.

Two stages have to not drop it: the ANN search has to put it in the top 50, and the reranker has to keep it in the top 5. Two survivals in a row, so the probabilities multiply.

P(gold passage in the final 5)
   = P(gold in ANN top-50) x P(reranker places it in top-5 | it was in the 50)

exact flat:    1.000 x 0.87  =  0.870
HNSW ef=64:    0.960 x 0.87  =  0.835      -3.5 points end to end
HNSW ef=128:   0.990 x 0.87  =  0.861      -0.9 points end to end

Where each factor came from

The 0.87 is the reranker’s measured Recall@5 at 50 candidates, from the rerank-depth table in Reranking in the serving path. It is the ceiling: even with a perfect front end, the reranker itself loses 13% of gold passages.

The 340x is flat’s 410 ms against HNSW’s 1.2 ms, from The comparison with the columns that actually decide it: 410 / 1.2 = 342.

Be honest about which recall the ANN factors are. 0.960 and 0.990 are the Recall@10 figures from The comparison with the columns that actually decide it, substituted into a term that actually asks for Recall@50.

Recall@50 is strictly higher than Recall@10 on the same index — the gold passage only has to survive into a pool five times deeper. So the numbers above are upper bounds on the loss, and the true gap is smaller than shown.

There is a second reason they are upper bounds: that 0.87 was itself measured with an approximate front end already in place. Using it as the conditional probability of the reranker succeeding charges part of the ANN loss twice.

The conclusion

A 4-point ANN recall loss becomes at most a 3.5-point end-to-end loss, in exchange for a 340x latency reduction.

Buying the 0.9-point version instead costs 2.5 ms and 3% more RAM per vector. The 0.990 comes from the M = 32 row of The comparison with the columns that actually decide it, which is 4.37 KB/vector against M = 16’s 4.24 — 4.37 / 4.24 = 1.03.

That is the argument, and it is arithmetic rather than folklore.

One more softener is worth saying out loud: many questions are answerable from several passages, so losing the single annotated “gold” chunk frequently costs nothing measurable at all.

The assumption underneath all of this is that ranking is the deliverable — that the system’s output is a ranked shortlist a later stage will filter, and that being slightly wrong about ordering degrades an answer rather than invalidating it. The next subsection is the list of jobs where that assumption is false.

4.6 When it is not fine — say these unprompted

Three situations invert the argument above. Volunteer them unprompted.

1. Exact-neighbor semantics. Deduplication, near-duplicate detection, “have we already indexed this,” entity resolution — deciding that two records refer to the same real-world person or company — and plagiarism detection. In all of these the nearest neighbour is the answer, so a missed neighbor is a wrong answer, not a slightly worse ranking. Use flat, or ANN with exact rescoring and a generous candidate pool.

2. Recall is the deliverable. Legal discovery, compliance sweeps, safety audits. “We searched and found nothing” has to mean it, and 0.96 recall means one document in 25 was never looked at.

3. Selective filters — and this is the big one.

Selectivity is the fraction of the corpus that survives a filter. A filter that admits one row in ten thousand has 0.01% selectivity.

Post-filtering means retrieve the top k by vector first, then drop whatever fails the filter. The arithmetic of that ordering is brutal, and the block below works it for a small tenant:

tenant T owns 0.01 % of the corpus
retrieve k = 50, then filter
E[survivors] = 50 x 0.0001 = 0.005

-> 99.5 % of tenant T's queries return ZERO results, and the system
   reports "no relevant documents found." Which is a lie, and it is
   indistinguishable from an empty corpus.

The two numbers in that block connect like this. Each of the 50 retrieved chunks independently has a 0.0001 chance of belonging to tenant T, so the chance that none of the 50 does is (1 - 0.0001)^50 = 0.995. Hence 99.5% of that tenant’s queries come back empty.

Pre-filtering is the obvious repair: restrict the traversal so it only ever visits eligible nodes. It fixes the count, and it breaks HNSW a different way.

The graph was built over all nodes. Its links encode nearness in the full corpus, not in the eligible subset. So a greedy traversal keeps hopping to neighbours that turn out to be ineligible, learns nothing from them, stalls, and degenerates toward a linear scan. You pay graph overhead and get scan performance.

Below roughly 1% filter selectivity, HNSW’s graph structure stops helping, and you must partition instead of filter. Partitioning means physically separating the eligible vectors into their own index, so the predicate chooses which structure to search rather than what to discard afterwards.

That derivation is what makes the next section a consequence rather than an opinion.

5. Sharding and multi-tenancy

Sooner or later the index no longer fits on one machine, and sooner or later it holds more than one customer’s documents. Those look like separate topics and are one: both are decisions about which vectors are physically stored together, and When it is not fine say these unprompted already proved that physical grouping — not query-time filtering — is the only thing that works below 1% selectivity.

Sharding

Sharding means splitting one logical index across several machines, each holding a disjoint slice called a shard. The choice is the splitting rule — and the tail-latency bill that arrives with every rule that requires asking more than one shard.

Fan-out in that table means sending the query to every shard and merging the results, because any shard might hold the best match.

Its cost is counterintuitive, so derive it rather than asserting it. The merge itself is cheap. The waiting is not, because a fan-out query is only finished when its slowest shard is finished.

one shard: p50 = 1.2 ms, P(latency > 5 ms) = 0.01
16 shards, query latency = max over shards
P(at least one shard > 5 ms) = 1 - 0.99^16 = 0.149

Read the middle line carefully. 0.99^16 is the probability that all sixteen shards stay under 5 ms; one minus it is the probability that at least one does not. That works out to 0.149.

Your p99 becomes your shards’ p85. A one-in-a-hundred event on one machine becomes a one-in-seven event once you need sixteen machines to all behave. Fan-out converts a rare per-shard tail into a common per-query tail.

Two mitigations:

• Fewer, larger shards. Fewer chances to draw a slow one.
• Hedged requests. When a shard has not answered by the time it usually would, fire a duplicate of the same query at a replica and take whichever response arrives first. You pay a few percent extra load to cut the tail.

The derivation assumes shard latencies are independent, which is what lets the probabilities multiply. Correlated slowness — a garbage-collection pause, or a noisy neighbour affecting a whole rack at once — makes the real number better than 0.149, not worse, because the slow events coincide on one query instead of spreading across many.

Multi-tenancy

Where do each customer’s vectors physically live? There are three arrangements, two of them wrong, and the reason the third wins is the filter arithmetic of When it is not fine say these unprompted rather than anything about cost.

The hybrid is the answer, and the reason is When it is not fine say these unprompted rather than economics.

Tenants above ~0.5% of the corpus get a dedicated index, because they are large enough to justify one.

Tenants below it must live in a shared index that is physically partitioned by tenant — meaning each tenant’s vectors are stored in their own separately-searchable region. Then the tenant predicate (the true-or-false test tenant_id = "acme") selects a partition rather than filtering results after the fact.

Get that backwards and small tenants get empty answers while large tenants are fine. That is a bug which reproduces only for the customers least able to report it precisely.

The 50-200 MB figure is the assumption to check

The whole 50,000-tenant rejection rests on one number, and it is the number you should not take from me.

The ~50-200 MB per-index overhead is a band, not a measurement. It is what an index process typically costs before it holds a single vector:

• the graph’s own metadata and free lists,
• the segment and doc-id maps,
• the BM25 dictionary,
• the JVM or runtime heap floor,
• whatever the service reserves per shard replica.

Every one of those is implementation-specific. None is a property of HNSW. The spread across real systems is wider than one order of magnitude: a bare hnswlib index in a shared process sits at the very bottom of that band or below it, while a managed service that provisions a replica set per index can sit far above the top.

The conclusion is entirely linear in that number, so substitute both ends:

at 200 MB:  50,000 x 200 MB  =  10 TB  of pure overhead   -> obviously impossible
at   2 MB:  50,000 x   2 MB  = 100 GB  of pure overhead   -> merely expensive

At 2 MB the hybrid stops being forced. So measure it: create one empty index, measure resident memory, create a hundred, divide. That is a five-minute experiment, and it is the same class of vendor question as the delete semantics in The comparison with the columns that actually decide it.

The shape of the argument survives either way — a fixed per-index cost times a large tenant count eventually dominates. The tenant count where it bites is yours to measure, not mine to assert.

The two thresholds are the same threshold

When it is not fine say these unprompted found that filtering stops working below about 1% selectivity. The dedicated-index cutoff above is 0.5% of the corpus — deliberately half of it.

The gap is not arbitrary. Setting the cutoff below the breakdown point means every tenant left in the shared index is unambiguously in the regime where filtering fails, which removes the judgement call entirely. There is no band of tenants for whom a query-time filter is “probably fine.”

Anyone in the shared index is partitioned, full stop. Anyone big enough that a filter might have survived already has their own index.

A cutoff above 1% would create exactly that ambiguous band — and the tenants inside it would be the ones whose empty results nobody can reproduce.

The leak

This is the failure that actually ends careers. The trace below is one argument going missing, six months apart.

# the original, correct call
retriever.search(query_vec, top_k=10, filter={"tenant_id": ctx.tenant})

# six months later, a new "global knowledge base" feature ships
retriever.search(query_vec, top_k=10)          # filter omitted; defaults to None

RESULT: chunks from 340 tenants, ranked by cosine similarity.
        No exception. No error log. Latency is NORMAL - slightly better.
        The agent cites them with source ids that look internal and correct.
        Discovered three weeks later by the customer whose contract text
        appeared inside another customer's answer.

The control is the same one the support agent uses for customer_id (case study 06): make the unscoped call unrepresentable rather than validated.

The same shape closes a second hole: a query embedded by one model being searched against an index built by a different one (Embedding model version skew the nastiest). But only if you put the check where the vector is.

A constructor that compares two model-id strings, and then hands you a search(query_vec) that accepts any vector from anywhere, is decoration. The constructor never sees a vector. So an embed-v1 vector passed to a retriever “verified” against embed-v2 ranks happily and returns results.

The fix is that vectors are never bare. An Embedding carries the id of the model that produced it, and the check runs on every query.

The listing below implements both controls and then attacks them. Read the three numbered tests at the bottom first:

1. A foreign-space vector must be refused on the search path.
2. An untagged bare vector must not be searchable at all.
3. Subclassing the constructor check away must buy nothing — this is the test that proves the control does not live in the constructor.

import inspect
from typing import NamedTuple


class Embedding(NamedTuple):
    """A vector that carries the id of the model that produced it.

    This is the structural half of the control: an untagged vector cannot be
    checked against anything, so it is not accepted at all.
    """
    model_id: str
    vec: tuple


class QueryEncoder:
    """The only thing that mints an Embedding, and it stamps its own id."""

    def __init__(self, model_id: str, encode):
        self.model_id, self._encode = model_id, encode

    def __call__(self, text: str) -> Embedding:
        return Embedding(self.model_id, tuple(self._encode(text)))


class ScopedRetriever:
    """Tenant scope and embedding space are both constructor arguments."""

    def __init__(self, index, tenant_id: str, encoder: QueryEncoder):
        if not tenant_id:
            raise ValueError("refusing to construct an unscoped retriever")
        if index.embed_model_id != encoder.model_id:
            raise ValueError(
                f"index was built with {index.embed_model_id!r} but the query "
                f"encoder is {encoder.model_id!r} - the vectors are not comparable"
            )
        self._index = index
        self._tenant = tenant_id
        self._encoder = encoder

    def search(self, query, top_k: int = 50):
        # Control 1: there is no parameter that widens the tenant scope.
        # Control 2: the model-id check, and it has to be HERE. In __init__ it
        # compares two strings once and never sees a vector, so it cannot stop
        # a v1 vector from reaching a v2 index — which is the actual failure.
        q = self._encoder(query) if isinstance(query, str) else query
        if not isinstance(q, Embedding):
            raise TypeError(
                "search takes query text or an Embedding; a bare vector carries "
                "no model id, so nothing can check which space it lives in")
        if q.model_id != self._index.embed_model_id:
            raise ValueError(
                f"query vector from {q.model_id!r} against an index built with "
                f"{self._index.embed_model_id!r} - see §10.2, and note it would "
                f"rank happily if this line were deleted")
        return self._index.search(q.vec, top_k=top_k, partition=self._tenant)


class FakeIndex:
    embed_model_id = "embed-v2"

    def search(self, vec, top_k, partition):
        return [(f"{partition}#3", 0.81)]


v2 = QueryEncoder("embed-v2", lambda t: (0.1, 0.2))
v1 = QueryEncoder("embed-v1", lambda t: (0.3, 0.4))
r = ScopedRetriever(FakeIndex(), "acme", v2)

assert r.search("what is the refund window?") == [("acme#3", 0.81)]
assert "tenant" not in inspect.signature(ScopedRetriever.search).parameters

# 1. a foreign-space vector is refused ON THE SEARCH PATH
try:
    r.search(v1("what is the refund window?"))
    raise AssertionError("an embed-v1 vector was ranked against an embed-v2 index")
except ValueError:
    pass

# 2. an untagged vector is not searchable at all
try:
    r.search([0.3, 0.4])
    raise AssertionError("a bare vector reached the index unchecked")
except TypeError:
    pass


# 3. and subclassing the constructor check away buys nothing, because the
#    control is not in the constructor
class Unchecked(ScopedRetriever):
    def __init__(self, index, tenant_id, encoder):
        self._index, self._tenant, self._encoder = index, tenant_id, encoder


try:
    Unchecked(FakeIndex(), "acme", v1).search("what is the refund window?")
    raise AssertionError("skipping the constructor check bought a working search")
except ValueError:
    pass

Delete the two lines in search that compare q.model_id and test 1 returns [('acme#3', 0.81)] — a ranked result list, from a v1 vector, against a v2 index, with no error. That is the entire Embedding model version skew the nastiest incident reproduced in four lines, and it is why the check cannot live in __init__.

Three tests, and the third one runs in production forever:

• search exposes no tenant parameter — the control is that the field does not exist.
• A foreign-space vector raises, and an untagged one is not accepted at all. Assert this by constructing the bad call, not by constructing a bad retriever; the retriever was never the thing that ranked anything.
• A canary tenant holding poison documents with unique sentinel strings, plus a nightly job asserting those strings never appear in any other tenant’s results. A canary here is a fake customer that exists only to be watched, and a sentinel string is a nonsense token like zq7-canary-4418 that could not occur naturally, so finding one anywhere else is proof rather than suspicion. This failure has no other detector: no error, no anomalous latency, no unusual log line.

What this control assumes

State it plainly, because this is a control that can be walked around. It assumes two things.

1. The tenant scope is fixed when the retriever is constructed. A wildcard tenant id like "*" satisfies the non-empty check and constructs cleanly. If your storage layer treats that as “all partitions,” the guard has been passed, not defeated.

2. Constructing a retriever is the only way to reach the index. The underlying index object still has a wide API and is one attribute access away, so the isolation holds only as long as nothing else in the process is handed the raw index.

The structural control raises the cost of the mistake enormously. The canary tenant is what covers the cases where somebody paid it.

6. Reranking in the serving path

The query-time path, the fast clock from the opening, end to end with every stage priced. It settles the most common false economy in retrieval systems: that reranking is too slow to afford.

One framing note before the diagram. This is the classic single-pass path: embed once, retrieve once, rerank once, generate once.

In the agentic variant that chapter 05 describes, the model may call retrieval zero, one, or several times per question. To price that, multiply the retrieval subtotal below and the per-query cost of Query time per 1000 queries by the expected number of calls. Chapter 05 derives that multiplier and quotes a band of roughly 0.6x to 6x, depending on how often the model searches at all.

Every number below is for one pass.

The diagram traces one question from arrival to answer. Each node carries its own millisecond cost; the argument of this section is visible in the numbers.

flowchart LR
    Q(["Query"]) --> QC{"Query-embedding cache<br/>saves 18 ms on a hit"}
    QC -->|hit 15-40%| RC
    QC -->|miss| EMB["Embed · 18 ms"]
    EMB --> RC{"Retrieval cache · saves 73 ms on a hit<br/>key: query · tenant<br/>filters · index_version"}
    RC -->|hit 20-35%| CTX
    RC -->|miss| PAR["ANN 1.2 ms<br/>BM25 8 ms<br/>in parallel · max = 8 ms"]
    PAR --> RRF["RRF fusion · 0.1 ms"]
    RRF --> RR["Cross-encoder rerank<br/>50 candidates · 45 ms"]
    RR --> CTX["Assemble 3-5 passages<br/>2 ms"]
    CTX --> LLM["Generate · 4,650 ms<br/>TTFT 650 ms · decode 4,000 ms"]
    LLM --> A(["Answer + citations"])

    style RR fill:#2d6a4f,color:#fff
    style LLM fill:#9d0208,color:#fff
    style RC fill:#2d6a4f,color:#fff
    style QC fill:#2d6a4f,color:#fff

Green marks a stage you should spend on; red marks the stage that already has all the money.

Follow one question through it

1. Query-embedding cache. The query hits this first. On a hit — 15 to 40% of the time — its vector is already known and the embed stage is skipped entirely.
2. Embed, 18 ms. Only on a miss.
3. Retrieval cache. Keyed on four fields: the query, the tenant, the filters, and the index_version. That last field is what makes this cache safe, as Caching three caches three risk profiles derives. On a hit — 20 to 35% of the time — the ranked list is already known and the pipeline jumps straight to context assembly.
4. ANN 1.2 ms and BM25 8 ms, in parallel. “In parallel” is the load-bearing word: dispatching them together means the pair costs the slower of the two, max(1.2, 8) = 8 ms, rather than their sum.
5. RRF fusion, 0.1 ms. Merges the two rankings by position.
6. Cross-encoder rerank, 45 ms. Re-scores the surviving 50 candidates properly.
7. Assemble context, 2 ms. Packs three to five passages into a single prompt block.
8. Generate, 4,650 ms. Split into 650 ms of TTFT — time to first token, how long until the first word appears — plus 4,000 ms of decoding to write the remaining 400 tokens.

Generation is 4,650 ms of the 4,723 ms total, and 89% of the cost. That is the red node.

The three green nodes — the reranker and both caches — are together about 1% of the latency. Between them they hold the largest quality lever in the pipeline and its two largest latency refunds. Those two facts are the whole section.

The bill of latencies

The block below is the same path written out, assuming every cache missed. That is the worst case, and it is the right case to design an SLO against.

query embed (hosted API)                18 ms    network-dominated
query embed (local 0.1B model)           4 ms
ANN search (HNSW, ef=64)               1.2 ms
BM25 (Lucene, 10M docs)                  8 ms    runs parallel with ANN, so max(1.2, 8) = 8
RRF fusion                             0.1 ms
rerank 50 candidates x 250 tokens       45 ms    <- the "expensive" stage
assemble context                         2 ms
                                       -------
retrieval subtotal            18+8+0.1+45+2  =  73.1 ms

LLM time to first token (3.5k prompt)  650 ms
LLM decode, 400 tokens               4,000 ms
                                       -------
end to end                           4,723 ms    retrieval = 1.5 %

Where the 45 ms comes from

The rerank figure is the one people flinch at, so derive it rather than quoting it.

The standard estimate for a transformer’s forward pass is about 2 floating-point operations per parameter per token — one multiply and one add. So the cost is 2 x parameters x tokens.

Three units: FLOP is one floating-point operation, TFLOP is a trillion of them, and TFLOP/s is the rate the accelerator sustains in practice — well below its headline number.

50 candidates x 250 tokens  =  12,500 tokens
300M-param cross-encoder:  2 x 3e8 x 12,500  =  7.5 TFLOP
at 300 TFLOP/s effective                     =  25 ms
plus batching overhead and one network hop   ≈  45 ms

Reranking costs 45 ms against a 4.7-second end-to-end response. It is 1% of latency and the single largest quality lever in the pipeline (ch 05 explains why a cross-encoder beats a bi-encoder). The standard objection — “reranking is too slow” — is almost always measuring the wrong denominator. The real questions are whether the reranker batches across concurrent queries, and whether it fits before the first streamed token.

The dial that actually matters is depth

The parameter people do not name is rerank depth, not rerank presence — how many candidates the reranker is given, rather than whether it exists at all.

Depth is the parameter with a real curve behind it. Read the table for where the curve bends, not for the absolute numbers:

Read the two steps either side of 50:

• 50 to 100 doubles the cost (45 ms to 88 ms) and buys 2 points of Recall@5.
• 50 to 20 halves the cost (45 ms to 18 ms) and gives back 6 points.

Fifty is not a magic number. It is where the curve bends for this corpus, and you find it by plotting it.

Three legitimate reasons to skip reranking entirely: nDCG@5 measures identical with and without it (your ANN ordering is already good enough), latency-critical autocomplete, or a UI that only ever shows three results and a large top-1 margin.

Two assumptions, both checkable in an afternoon

1. Generation dominates the response — 650 ms to first token and 4,000 ms of decode. That is what makes 45 ms of reranking 1% rather than 30%. It stops being true for a system that answers with a single extracted sentence rather than a written paragraph, and the whole latency argument then has to be re-run.

2. The recall column above was measured on your corpus. Every number in it is corpus-specific. The bend at 50 is a property of how many near-misses your ANN stage puts between rank 20 and rank 50, which is a property of your documents and your queries and nothing else. Import the method from this table, never the numbers.

7. Caching: three caches, three risk profiles

Four things in this pipeline can be cached and three of them should be. Telling which is which turns out to be a single property of the cache key rather than a judgement about staleness tolerance.

Three terms before the table:

• Cache key — the value you look an entry up by.
• Hit rate — the fraction of requests that find an entry already there.
• sha256 — a hash function. It turns any text into a short fixed-length fingerprint, so identical text always produces the same key and different text essentially never does.

Read the Key column against the Risk column. That pairing is the entire section.

The two embedding caches are free money

embed(text, model) is a pure function — one whose output depends only on its inputs, with no hidden state and no clock. So two identical chunks have identical vectors forever, as long as the model id is in the key.

On a re-ingest where 5% of documents changed, the cache turns a full re-embed into a 5% re-embed. That is a 20x reduction (1 / 0.05 = 20) on the dominant ingest compute.

The other two caches can go wrong, and one field decides which

index_version is a counter that increments every time the index changes. It is in the retrieval key and not in the answer key, and that single difference is the whole rest of this section.

State the asymmetry in one sentence:

A retrieval cache entry is invalidated by the very thing that would make it wrong, because index_version is in its key. An answer cache entry is not, because the answer’s correctness depends on the documents, and the documents are not in its key.

t0   query "what is the refund window?"
     retrieval -> [refund-policy#3 @ index_version 4471]
     answer    -> "30 days from the renewal date."
     both cached.

t1   legal updates the policy to 14 days. Ingest runs.
     index_version  4471 -> 4472.

t2   same query, same user.
     retrieval cache: key holds 4471, current is 4472  ->  MISS
                      re-retrieves, gets the new passage.       CORRECT
     answer cache:    key is query_hash alone           ->  HIT
                      returns "30 days."                        WRONG
                      ...with a citation to a document that now says 14,
                      which makes the wrong answer look verified.

The answer cache does not merely go stale. It goes stale while continuing to emit a citation that makes it look checked.

You can repair it by adding index_version to the answer key — at which point the hit rate collapses to near zero on any live corpus. That collapse is the honest signal that the cache was never viable in the first place.

The one answer-cache key that does work

Key the answer on the retrieved chunk ids plus their content hashes. Then it invalidates exactly when the evidence changes, and survives index rebuilds that did not change any content. Lower hit rate, correctness preserved by construction.

Replay the timeline above under that key:

• t0 — the key is built from refund-policy#3 plus the hash of the text saying 30 days.
• t1 — legal edits that chunk, so its content hash changes.
• t2 — the key computed from the freshly retrieved evidence no longer matches the stored one. The cache misses, and the answer is recomputed as 14 days. Correct.

Now the converse case, which is where this key earns its keep. A reindex bumps index_version without changing a word of any document. Every content hash is identical, so every entry survives — where an index_version-keyed cache would have thrown all of them away for nothing.

Two assumptions decide whether those hit rates are yours

1. Queries repeat verbatim after normalization — lowercasing, whitespace collapsing, punctuation stripping. That holds on an interface where users type short questions and many type the same one. It is much weaker where questions are long, and it can vanish entirely in an agentic system, because a model that rewrites the user’s question before searching produces a different string almost every time (The pipeline and where it breaks).

2. index_version advances on every change. If it is only bumped on full rebuilds, the retrieval cache silently inherits the answer cache’s problem — and there is no signal that it has.

8. The cost model, derived

There are two bills — what it costs to index a corpus once and what it costs to answer a question — and reading them shows where the money actually is, which is not where anyone expects. Prices are US dollars at the rates listed; the method survives the rates changing, so carry the derivations rather than the totals.

8.1 Ingest, per 1M documents

Pricing the index-time path of The indexing pipeline is a streaming system produces the chapter’s most counterintuitive result.

Start with the assumptions, out loud. They are half the answer, and an interviewer is listening for whether you state them before you quote a total.

average document   8 pages ≈ 4,000 words ≈ 5,300 tokens
structural chunking at ~500 tokens        ->  11 chunks/document
1M documents  ->  11M chunks  ->  5.5 B tokens

Five reading keys for that table:

• Parsing — turning a file into plain text. OCR, optical character recognition, is the expensive branch of it: reading characters out of an image, which is what a scanned page is.
• MTok — one million tokens, the unit hosted model APIs bill in.
• GPU-hour — one hour of one accelerator. TF/s is trillions of floating-point operations per second, so the self-hosted line is the same 2 x parameters x tokens estimate from Reranking in the serving path turned into hardware time and then into money.
• GB-month — one gigabyte held for one month.

Two of that table’s inputs are implicit, so surface them. The 8M pages are 1M documents x 8 pages. The 22 GB of extracted text is 5.5e9 tokens x 4 bytes, using the same “a token is about four characters” rule the chapter opened with.

The result that surprises everyone

Parsing is 98% of ingest cost. Embedding is 1%.

That inverts what almost everyone expects, and it redirects the optimization entirely. The win is “do not OCR the 60% of the corpus that is already digital text,” which drops hosted parsing from $12,000 to about $12,000 x 0.4 = $4,800. The win is not “find a cheaper embedding model.”

On the self-hosted embedding line: $13 against $110 is an 8.6x gap, and it does not matter. Five GPU-hours of work does not justify operating a service. Self-host embeddings when you re-embed continuously — a 100M-chunk corpus with a monthly model refresh — not when you ingest once.

How much of this rests on the corpus being PDFs

Almost all of it. The 98% figure is a statement about document format, not about retrieval.

It holds for scanned archives, contracts and manuals. It collapses for a corpus of wiki pages or source code, where parsing is nearly free and embedding becomes the dominant line.

So in an interview, state the format assumption before the number, because the number is worthless without it. And if the corpus is 60% digital text, the fix is not a cheaper OCR vendor — it is a branch that never calls one.

8.2 Storage, recurring

Ingest is a one-time bill. The one that arrives every month is where quantization stops being a tuning detail.

11M chunks x 4.24 KB (HNSW fp32, d=1024)     =  46.6 GB RAM

256 GB instance at $2.02/hr, ~200 GB usable
effective RAM price = $2.02 x 730 / 200      =  $7.37 / GB-month

46.6 GB x $7.37                              =  $343 / month
x2 replicas for availability                 =  $687 / month

with int8 quantization (1.17 KB/vector):
12.9 GB x $7.37 x 2                          =  $190 / month     3.6x cheaper
                                                                 (= 4,244 / 1,172 bytes, exactly)

Two lines in that block need explaining.

The 730 converts an hourly instance price into a monthly one. It is the average number of hours in a month (365 x 24 / 12 = 730).

The replicas are copies of the whole index on separate machines, so one machine failing does not take search down with it. Two is the usual minimum, and it doubles the bill exactly.

Now put the recurring bill next to the one-time bills from §8.1:

storage, replicated        $687 / month
embedding bill  (once)     $110           -> passed inside the first week
whole ingest bill (once)   $12,150        -> passed inside 18 months (12,150 / 687 = 17.7)
                                             of which $12,000 is parsing

Quantization is not a micro-optimization. It is the second-largest cost lever in the system, behind only the generation context size.

The assumption that makes RAM the recurring bill is that the index must be resident in memory to hit its latency target. That is what an in-memory graph index requires, and it is what the ~200 GB usable figure prices.

A disk-backed index changes this whole line item by an order of magnitude, and changes the latency budget of Reranking in the serving path by rather more. That is exactly the trade the binary-plus-rescore row of Hnsw memory per vector derived makes deliberately.

8.3 Query time, per 1,000 queries

Pricing one question end to end produces the priority list that should govern every optimization decision in the system.

The assumptions: 5 passages at 500 tokens, a 1,000-token system prompt, a 50-token question, 400 output tokens, on claude-sonnet-5 at $3/$15 per MTok.

That price pair is input and output respectively. Output tokens cost more everywhere, because they are produced one at a time rather than read in parallel.

The input total adds those assumptions up:

system prompt                     1,000 tokens
5 passages x 500 tokens           2,500 tokens
question                             50 tokens
                                  -----
input                             3,550 tokens

3,550/1e6 x $3   +   400/1e6 x $15   =   $0.01065 + $0.006   =   $0.0167 per query

One row derives rather than quotes. The self-hosted rerank line is the 25 ms of accelerator time from Reranking in the serving path, priced at the chapter’s $2.50/GPU-hour:

0.025 s x ($2.50 / 3,600 s)  =  $1.74e-5 per query

It is under a tenth of a percent of the bill and moves nothing. It is derived here only so that no number in the table is asserted.

Generation is 89% of query-time cost. The entire retrieval stack is 11%, and the vector search itself is 0.1%.

Four consequences follow, in priority order:

1. Tuning efSearch to save 0.5 ms optimizes 0.1% of the bill. Do not spend the week.
2. Cut from 5 passages to 3. That removes exactly 1,000 input tokens — the prompt goes 3,550 -> 2,550 — for 1.0 x $3/1000 = $0.003/query = $3.00/1k, 16% of the total bill from one config change, and frequently better answers, because the marginal passages were diluting attention (Why quality degrades in long contexts).
3. Self-host the reranker above roughly 5M queries/month: $2.00/1k -> $0.0174/1k, worth 10.7%. The threshold is a judgement rather than a crossover point — the raw arithmetic breaks even far sooner, and the gap is the cost of running, monitoring and keeping redundant a service you did not previously operate.
4. Prompt-cache the system prompt — which, as specified above, saves nothing at all. The next subsection is why.

Consequence 4, and why it is worth zero as written

Prompt caching is a feature of hosted model APIs. Mark a stable prefix of the prompt, and repeat requests beginning with exactly those tokens are billed at a fraction of the input rate — 0.1x here — because the provider reuses the work it already did on that prefix. The mark is called the breakpoint.

Two rules govern whether it fires:

• Rule 1: exact prefix. Everything before the breakpoint must be byte-identical across requests.
• Rule 2: minimum length. Every model has a minimum cacheable prefix, and a prefix below it silently does not cache.

Rule 2 is the one that bites here, and the floors are not even monotonic across a vendor’s own generations: 512 tokens on claude-opus-5, 1,024 on claude-sonnet-5, 4,096 on claude-haiku-4-5 (Prompt caching derived).

The model priced in this table is claude-sonnet-5. The stable prefix assumed above is 1,000 tokens — 24 short of the 1,024 floor. So nothing caches, cache_creation_input_tokens comes back 0, no error is raised anywhere, and the real saving is $0.00:

claimed:  0.9 x 1,000 tok x $3/MTok  =  $0.0027/query  =  14.5%
actual :  1,000 < 1,024 -> no cache  =  $0.00         =   0%
padded past the floor (1,024 tok)    =  $0.0028/query  =  14.8%

The arithmetic was right conditional on caching happening. The premise failed.

Pad the stable prefix past the floor — 24 tokens of boilerplate is enough — and the 14.8% is real.

One more constraint: retrieved passages vary per query, so they can never be part of the cached prefix. Put them after the breakpoint. Put them before it and they invalidate the cache on every single request, and you are back to zero for a different reason.

The optimization order at query time is context size, then rerank vendor, then never the ANN parameters. Note what that ordering does not contain: prompt caching, which as specified is worth nothing and only enters the list once the prefix clears the floor. Say the order and you have shown you know where the money is, which is the whole point of building the table.

This is the cheapest instance in the chapter of a control that looks like it is working and is not — a config that raises no error, returns a well-formed response, and delivers exactly none of the saving the design assumed. It belongs in the same family as Embedding model version skew the nastiest and the cross-tenant leak of Sharding and multi tenancy, and it has the same detector shape: assert the thing you assumed, on every call. Here the assertion is one field — alert if cache_creation_input_tokens is 0 on a request whose prefix you believe is cacheable.

Two assumptions that move the whole table

Every share above assumes one retrieval and one generation per question, priced at $3/$15 per million tokens. Both assumptions move the conclusion rather than the arithmetic.

An agentic loop that searches three times pays the generation line roughly four times over. That makes the retrieval stack a smaller fraction still, and it makes consequence 2 stronger — a passage you did not fetch is also a passage not re-billed on every subsequent call (Classic vs agentic).

A cheaper generation model compresses the 89% toward the retrieval stack. At a small enough model, hosted reranking becomes the largest line and consequence 4 moves to the top of the list.

Re-run the table whenever either changes. Do not carry the percentages.

9. Evaluation: three layers, and nobody builds the third

Measurement comes in three layers, and the order matters. The first two are standard. The third is the one that separates a design from a demo, because every failure in Failure modes is invisible to the first two.

The generation-layer metrics are the ones most likely to be assumed rather than defined, so define them:

• Faithfulness — the fraction of claims in the answer that are actually supported by the passages the model was given. This is the direct measurement of grounding, and the direct inverse of the hallucination rate.
• Answer relevance — whether the response addresses the question that was asked, rather than a nearby one.
• Citation accuracy — the citation-resolution check from the vocabulary section, scored. Each [chunk-id] the model emitted must exist, must have been in the context, and the passage must in fact support the sentence it was attached to.

Diagnose retrieval first, because it is a dependency and not a preference. If Recall@10 is 0.4, the right passage is absent from context 60% of the time. No prompt change makes a model cite what it cannot see.

Tuning generation first is optimizing a downstream stage against a broken input.

The system layer, concretely

Four measurements. Each one is the only detector for a failure in Failure modes, which is why this layer is the one that gets skipped and the one that matters.

1. Nightly ANN recall audit. Sample 1,000 production queries. Run each one twice: against the live HNSW index, and against a flat index over the same vectors. Then score each query as

|top10_ann & top10_exact| / 10

which is the number of chunk ids the two top-10 lists have in common, divided by ten. A query whose approximate and exact top-10 share nine ids scores 0.9. Average that over the 1,000 sampled queries and you have last night’s index fidelity — one number to trend against the previous run.

This is the only way you will learn that your index degraded: after a rebuild with different parameters, after months of deletions, after a corpus distribution shift. Nothing else in the stack raises a signal.

Gate the alias flip on it. A shadow index that does not clear the bar does not go live, and the fact that it did not is the alert.

2. Staleness p99. Stamp ingested_at on every chunk, and carry source_modified_at from the origin — when the source system last changed the document. The p99 of the difference is your freshness SLI, the measured counterpart of the staleness window derived in Freshness and the staleness window derived.

The point of measuring it: a backfill starving the incremental lane shows up here and nowhere else. 3. Model-id assertion on the search path. Every query vector carries the id of the model that produced it, and the retriever refuses a foreign one on every call — not once at construction, where nothing has a vector yet and the check is decoration (Sharding and multi tenancy, Embedding model version skew the nastiest).

4. Orphan and tombstone counts. Orphans are chunks whose doc_id no longer exists upstream, or whose chunk_index exceeds the current version’s chunk count — the leftovers of The indexing pipeline is a streaming system. The tombstone ratio is the fraction of graph nodes that are deleted-but-resident, which Deleted documents that stay retrievable turns into a memory bill and a compliance failure.

What the whole evaluation layer assumes is a labelled set that looks like production. Recall@k, nDCG and MRR all need questions with known-correct passages. If those questions were written by the team, they will be the questions the team would ask — which is systematically not the traffic.

The system-layer metrics are the exception, and that is precisely their value. Index fidelity, staleness, orphan count and tombstone ratio need no labels at all, which is why they keep working on the day the labelled set goes stale.

10. Failure modes

Every failure below produces a fluent, well-formatted, confidently wrong answer with a citation that looks correct. None of them raises an exception, none of them shows up as a latency anomaly, and none of them is caught by the retrieval or generation metrics of Evaluation three layers and nobody builds the third. Each comes with the mechanism, the detector, and the control.

10.1 Half-swapped alias after a reindex

The first failure is what happens when the atomic alias flip from The indexing pipeline is a streaming system is applied to two indexes one at a time.

Recall that queries reach vectors-live and bm25-live, which are names pointing at concrete builds. In the timeline below the two names briefly point at different corpus versions — watch the clock between 04:12 and 05:40.

02:00  backfill builds vector index v9         (completes 04:12)
04:12  alias  vectors-live  ->  v9             [flipped]
04:12  BM25 rebuild still running              (completes 05:40)
       alias  bm25-live     ->  v8             [not flipped]

04:30  query "annual plan refund window"
       dense  -> refund-policy#3   (ids from corpus v9)
       BM25   -> refund-policy#9   (ids from corpus v8; deleted in v9)
       RRF fuses both rankings.
       The v8 id is looked up in the v9 content store -> KeyError,
       swallowed by a try/except in the assembler, passage dropped.

       Recall for that query: half of what it should be.
       No alert. The answer is fluent and cites the surviving passage.

A KeyError is what Python raises when you look up a key that is not there. The try/except around it in the assembler is the whole defect: the code that fetches passage text for the ids the ranker returned treats a missing id as a normal, ignorable event.

It is not. Every id in that list came out of your own index moments ago. An id that does not resolve means the two halves of the system disagree about what exists.

Two indexes over one corpus must flip atomically, or ids from different corpus versions will be fused into one ranking. Controls: a single corpus_version that both indexes are built against, an alias flip that moves both or neither, and a hard assert at fusion time that every candidate id carries the same corpus version. Never a swallowed KeyError in the assembler — a missing document is a page, not a shrug.

10.2 Embedding-model version skew — the nastiest

This is the failure with the widest blast radius and the weakest symptoms: an entire corpus disappears from search results while every metric on the dashboard stays green.

The setup is ordinary. You upgrade from embed-v1 to embed-v2. Same dimension, so nothing raises. New documents enter with v2 vectors; old documents keep their v1 vectors. Same index.

The mechanism, stated first and unpacked immediately after:

Two independently trained embedding models share no coordinate system. Dimension 47 of v1 and dimension 47 of v2 encode unrelated features. A dot product between a v2 query and a v1 document is a sum of products of unrelated coordinates — a random variable with mean ≈ 0 and standard deviation ≈ 1/sqrt(d) ≈ 0.031 for d=1024 on unit vectors.

Unpack that in three steps.

1. What a dot product is. It is what cosine similarity computes once both vectors have been scaled to length one: multiply the two vectors coordinate by coordinate, then add the 1,024 results.

2. Why the sum lands near zero. When the two coordinate systems are unrelated, each of those 1,024 products is as likely to be positive as negative. They cancel.

3. Where 0.031 comes from. The spread of a sum of d independent such terms shrinks like 1/sqrt(d), and 1/sqrt(1024) = 1/32 = 0.031.

Nothing about this is an error condition. It is two vectors of the right length and the right dimension, producing a perfectly well-formed number that happens to mean nothing.

So cross-version similarities pile into a tight band around zero, while same-version similarities spread across 0.3-0.9. The consequence is not “slightly worse ranking” — compare the two rows below and then read the last line:

query encoded with v2; index holds 8M v1 vectors + 400k v2 vectors

cos(q_v2, d_v1):  mean 0.002, sd 0.031, max observed 0.14
cos(q_v2, d_v2):  mean 0.31,  sd 0.12,  max observed 0.89

top-10 for EVERY query: 10 of 10 documents ingested after the migration date.

The pre-migration corpus behaves exactly as if it had been deleted, and nothing raises an error. Recall on the 95% of the corpus that is old falls from 0.94 to about 0.05, and the system returns confident, well-formatted, wrongly-scoped answers with valid-looking citations.

Detection is a distribution check, never an exception

Two signals, and the second is much sharper.

p50 top-1 cosine. The median similarity of each query’s best match. It drops when queries stop finding good matches.

Age distribution of retrieved documents. Plot how old the returned documents are. A corpus that has silently vanished shows up as every result being newer than one particular date — the migration date. This costs one histogram and it is unmistakable.

Controls, and only the first is a real control

1. The model id is part of the index identity, and part of every vector. The index is named chunks_embed-v2, vectors travel as (model_id, vec) rather than bare, and the retriever compares the two on every search (see ScopedRetriever, Sharding and multi tenancy). Checking it once at construction feels like the same control and is not: the constructor never sees a vector, so it cannot stop the one that matters. A silently-comparable-looking vector must be impossible to pass, not merely impossible to declare.
2. Blue/green. The name comes from deployment practice: run two complete environments, one live (“blue”) and one being prepared (“green”), and cut over in one step. Here it means building the v2 index completely, running the Evaluation three layers and nobody builds the third recall audit against it, and only then flipping. Never migrate in place.
3. Dual-write during the build — every incoming document is indexed into both the old and the new index while the migration runs — so the v1 index stays live and correct right up to the flip, and the flip has nothing left to catch up on.

Price the migration so the decision is easy:

11M chunks x 500 tokens  =  5.5B tokens  =  $110 hosted  /  $13 self-hosted
plus one extra index copy for the duration of the build

The re-embed is cheap. The second index copy and the operational care are what cost anything — which is precisely why teams migrate in place, and precisely why they should not.

10.3 Chunk-boundary loss

This failure is created at index time and paid at query time, and it is the reason chunking is a design decision rather than a preprocessing detail. The trace below shows a fact split across two chunks, and a retriever that behaves perfectly and still loses it.

document "Billing FAQ", section 4
  chunk 7 (ends):    "... Annual plans may be refunded within"
  chunk 8 (begins):  "30 days of the renewal date, minus any usage."

query: "how long do I have to get a refund on an annual plan?"

dense top-3:
  1. billing-faq#7   0.71   has "annual", "refunded", "plans"
  2. billing-faq#2   0.64   general refund overview
  3. billing-faq#8   0.58   has "30 days", "renewal", but NOT "refund"

context budget admits 2 passages  ->  chunk 8 is dropped

answer: "Annual plans are refundable, though the specific window is not
         stated in the documentation available to me."
   ...or, on a less-grounded model: "within 14 days."

The retriever did not fail. It ranked the chunk containing the query’s vocabulary first, exactly as designed.

The failure was created at index time, by a splitter that cut a sentence in half. The mechanism: the answer’s lexical signal (“annual”, “refunded”) and the answer’s content (“30 days”) ended up on opposite sides of a chunk boundary.

The four fixes below are ordered by what to reach for first. Read the Cost column — the first one is free and the last one is 18% of the parsing bill.

The four fixes, in the order the table lists them:

• Structural splitting cuts at headings and sentence ends rather than at a fixed token count, so a boundary never lands mid-fact.
• Overlap repeats the last 50-100 tokens of each chunk at the start of the next, so a fact straddling a boundary appears whole in at least one of them.
• Sentence-window (also called parent expansion) retrieves the chunk but feeds the model its neighbours too. This fixes the trace above directly: chunk 7 wins the ranking, and chunk 8 arrives with it.
• Contextual chunking prepends a sentence of document-level context to each chunk before embedding it, so a chunk that begins “This limit was raised to 1000 in v3” carries the subject that the split destroyed (The pipeline and where it breaks works that mechanism through in detail).

That last cost line is worth carrying in your head. Contextual chunking is the largest reported single-technique retrieval win, and $2,200 / $12,000 = 18% of the parsing bill. That makes it an easy yes rather than a research project.

10.4 Deleted documents that stay retrievable

The last failure is the one with legal consequences rather than quality consequences, and it accumulates silently over months rather than breaking at a moment.

HNSW has no true delete. A deleted node is tombstoned — still physically resident in the graph and still traversed as a routing hop on the way to live neighbours, merely filtered out of every result set.

Filtering it out of results is cheap. Removing it would orphan the links of everything that pointed at it, which is why implementations do not.

The arithmetic below assumes a corpus that is only deleting, not growing — the ratio is deletions against the original point set, with no new documents diluting it. A corpus that also grows has a lower tombstone fraction and a correspondingly smaller penalty, so treat this as the pessimistic bound it is.

Read 1 - 0.95^12 as “the fraction of the original points that survived twelve months of losing 5% each month, subtracted from one”:

5% of the corpus deleted per month, no compaction
after 12 months:  1 - 0.95^12  =  46% of graph nodes are tombstones

memory       1.85x     you are paying RAM for deleted vectors
latency      +~40%     traversal walks through dead nodes
recall       drifts    the degree distribution was optimized for the
                       original point set, not the surviving 54%
compliance   FAILED    the vector still exists in RAM and in every snapshot

The 1.85x memory figure is 1 / 0.54 — you are holding 100% of the original vectors in RAM to serve the 54% that are still live.

The degree distribution in the recall row is how many links each node has. The graph builder chose those links to make the original point set navigable, and deleting nearly half the points leaves a link structure fitted to a population that no longer exists.

The last row is the one that matters. A right to erasure — the legal obligation, under regimes such as the GDPR, to actually delete a person’s data on request — is not satisfied by a tombstone. The vector is still in RAM, still in every backup snapshot, and still reconstructible.

The control is scheduled compaction: for a graph index, rebuilding the shard from only its live vectors. Tie it to an SLA — a service level agreement, the contractual version of an SLO — matching your erasure commitment.

If you promise 30 days, every shard holding an erasure request is rebuilt within 30 days. Track requests to specific shards, so you rebuild the two that matter rather than all forty.

Summary

The whole section in one table. Read the middle column first — for every one of these, the detector is the design decision, because none of them announce themselves.

11. Alternatives considered and rejected

The design’s negative space: the choices that were on the table, why each is attractive, and the specific fact that rules it out.

Several are rejected only at this scale and are the right answer at another. So the “why rejected” column names the crossover wherever there is one — an interviewer is testing whether you know the boundary, not whether you memorized a verdict.

12. Interviewer pushback

Ten questions an interviewer actually asks, each with what it is testing and an answer at the length you would speak it.

Use them as a self-check. If you can produce the substance of each one without looking back at the section it came from, the chapter has done its job. If you cannot, the italic line under each question names the section to re-read.

“Walk me through the indexing pipeline.” Testing: whether you know it is a stream and not a for-loop. The tutorial version is a loop over documents, and it is correct exactly once — at t=0. As soon as the corpus changes I need three things a loop cannot express: knowledge of what changed, because a delete is not the absence of an insert; idempotency, because queues are at-least-once and workers die mid-document; and per-document ordering, because two edits in flight can land backwards. So it is CDC into a queue into stateless workers, the unit of work is a document version, and chunk ids are doc_id#index computed deterministically. An update replaces every chunk under that doc_id — never a bare upsert, because if the new version has nine chunks where the old had eleven, chunks nine and ten stay in the index forever, get retrieved, and get cited under the same source id as the current text. And that replace has to be one atomic storage operation: delete-then-insert is two writes, and a worker that dies between them deletes the document instead of updating it, which is a worse and quieter failure than the one I was fixing. In Postgres it is a transaction; in an ANN service it is writing the new chunks under a new doc version and flipping one visibility pointer.

“How fresh is the index?” Testing: whether you can derive a number instead of promising one. It is a sum of terms, and one term dominates. With webhooks: 0.5 s detection, 0.3 s queue, 2 s parse for a PDF, 0.6 s embed including batch-fill wait, 0.05 s upsert, 1 s index visibility — about 4.5 s p50. With 15-minute polling every term is identical except detection, which becomes 450 s expected and 900 s worst case. So polling is 100x everything else combined, which means freshness is one architectural decision, push or pull, and not a tuning exercise. And the term that actually breaks the SLO is the queue: a 2M-document backfill at 20 docs/s drains in 27.8 hours, and every incremental update queued behind it inherits that. Separate lanes, separate pools, rate-limit the backfill.

“HNSW or IVF?” Testing: whether you pick on the columns that matter or the ones in the benchmark chart. It depends on write pattern, not latency — they are both about 1 ms at 10M vectors. HNSW inserts online at ~1.2 ms per vector, which is what you need for continuous ingest, but it cannot truly delete; deletes are tombstones that stay in the graph as routing hops, so at 5% monthly deletion you are 46% tombstones after a year, paying 1.85x memory and 40% latency, and failing right-to-erasure. IVF rebuilds cheaply but needs a fresh k-means when the distribution drifts. So: HNSW plus scheduled compaction for a streaming corpus, IVF for one rebuilt periodically. And below about a million vectors, flat — 4 ms, recall exactly 1.0, no tuning surface, no delete problem.

“How much RAM for 100 million chunks?” Testing: whether the number is derived or remembered. At d=1024 fp32 with HNSW M=16: 4,096 bytes for the vector, 128 for 32 layer-0 neighbour ids, about 4 for upper layers since the expected number of levels above zero is 1/(M-1) = 0.067, and roughly 16 bytes of bookkeeping. That is 4.24 KB per vector, so 424 GB — no longer one machine. int8 takes it to 1.17 KB and 117 GB, which fits, for about a point of recall. Binary quantization takes it to 276 bytes, so 28 GB for the same 100M chunks at d=1024, with raw recall around 0.71, but rescoring the top 200 against full vectors on NVMe brings it back to 0.96 for about 2 ms. So the real answer is that 424 GB is a choice, not a requirement, and the lever is representation.

“Your ANN recall is 0.96. Is that a problem?” Testing: whether you can reason about a pipeline instead of a stage. Usually not, and I can show why. The ANN stage feeds a reranker that itself only places the gold passage in the top 5 about 87% of the time, so exact search gives 0.87 end to end and HNSW at ef=64 gives 0.96 x 0.87 = 0.835 — at most a 3.5-point loss for a 340x latency reduction, and less than that in truth, since 0.96 is Recall@10 standing in for Recall@50. Pushing to the M = 32, ef=128 row costs 2.5 ms and 3% more RAM per vector, and gets me to 0.861. Where 0.96 is not acceptable: exact-neighbour semantics like dedupe or entity resolution, where a miss is a wrong answer rather than a worse ranking; legal discovery, where recall is the deliverable; and selective filters, where it is not a recall question at all — with a 0.01% tenant and post-filtering, 50 candidates yield 0.005 expected survivors, so 99.5% of that tenant’s queries return nothing.

“How do you isolate tenants?” Testing: whether isolation is a control or a convention. Hybrid, and the reason is the filter math rather than cost. Tenants above about 0.5% of the corpus get a dedicated index. Everyone below lives in a shared index physically partitioned by tenant, so the tenant predicate selects a partition instead of filtering results — because below roughly 1% selectivity HNSW’s graph traversal stops helping and post-filtering returns empty. Then the isolation itself: tenant scope is a constructor argument on the retriever, and search has no tenant parameter at all. A missing filter has to be unrepresentable, not validated, because the failure is silent — normal latency, no exception, plausible-looking source ids, discovered weeks later by the other customer. I also run a canary tenant with sentinel documents and assert nightly that they never appear anywhere else. There is no other detector.

“Where does the money go?” Testing: whether you optimized the thing you measured. Two separate bills. At ingest, per million documents: parsing is $12,000 and embedding is $110 — parsing is 98%, which surprises everyone, and the fix is not OCRing the 60% of the corpus that is already digital text, not a cheaper embedding model. At query time, per thousand queries: generation is $16.65 of $18.68, hosted reranking is $2.00, and the vector search is two cents. So the optimization order is context size, then rerank vendor, then never the ANN parameters. Cutting from five passages to three is 16% of the bill from one config change and often improves answers, because the marginal passages were diluting attention. Tuning efSearch optimizes 0.1%. And storage is the sleeper — $687/month for 11M chunks replicated, which passes the entire embedding bill in the first week, so quantization is the second-biggest lever in the system.

“We upgraded the embedding model and quality collapsed for old documents. What happened?” Testing: the failure that has no error message. Two independently trained embedding models share no coordinate system — dimension 47 of v1 and dimension 47 of v2 encode unrelated features. So a v2 query dotted with a v1 document is a sum of products of unrelated coordinates: mean about zero, standard deviation about 1/sqrt(1024) = 0.031. Same-version similarities spread 0.3 to 0.9. The old corpus is therefore uniformly pushed below every new document, and it behaves exactly as if it had been deleted — while the system returns fluent answers with valid citations and raises nothing. The detector is a distribution check: alert on the age distribution of retrieved documents, and on p50 top-1 cosine. The control is that the model id is part of the index name and part of every vector — query embeddings travel as (model_id, vec) and the retriever compares them on every search. I would specifically not put that check only in the constructor, which is the version I have shipped and been wrong about: it compares two strings before any vector exists, so a v1 vector still ranks fine against a v2 index and the guard reads as if it protects you. Then blue/green with dual-write, never in place. The re-embed itself is $110 of compute; it is the second index copy people are trying to avoid, and that is exactly the wrong thing to economize on.

“Answers are wrong. Where do you look first?” Testing: whether you diagnose in dependency order. Retrieval, always, and it is a dependency rather than a preference: if Recall@10 is 0.4 then the right passage is absent from context 60% of the time and no prompt change makes a model cite what it cannot see. So Recall@k and nDCG first, then faithfulness and citation accuracy conditional on retrieval succeeding. But I would also check the system layer, which is where the failures with no other detector live — the nightly ANN-versus-flat recall audit on a thousand sampled queries, staleness p99 from ingested_at minus source_modified_at, orphan chunk count, and tombstone ratio. If retrieval metrics look fine and answers are still wrong, I check whether the assembler is silently dropping passages on a KeyError, which is what a half-swapped index alias looks like from the outside.

“Could you just cache answers? Our queries repeat constantly.” Testing: whether you know which caches are safe and why. Cache retrieval, not answers, and the asymmetry is precise: a retrieval cache entry is invalidated by the very thing that would make it wrong, because index_version is in the key. An answer cache entry is not, because its correctness depends on the documents and the documents are not in the key. So when legal changes the refund window from 30 days to 14, retrieval misses and re-fetches correctly while the answer cache serves “30 days” — with a citation to a document that now says 14, which makes the stale answer look verified. Adding index_version to the answer key fixes it and drops the hit rate to near zero on a live corpus, which tells you the cache was never viable. The one variant that works is keying the answer on the retrieved chunk ids plus their content hashes, so it invalidates exactly when the evidence changes.

Next: 07 — Face Generation.