05 — RAG for Agents

Retrieval-augmented generation (RAG) means looking a question up in your own documents and pasting the passages you find into the model’s prompt. The model then builds its answer from text it can read directly, rather than from what it absorbed during training.

This chapter covers the agentic form: retrieval as a tool the model chooses to call, rather than a step that always fires. It answers four questions:

• How does one retrieval run, end to end?
• What does it cost?
• How does each stage of the pipeline fail?
• Which questions deserve a lookup at all?

By the end you should be able to design a retrieval pipeline, identify the stage responsible for a given wrong answer, and justify skipping retrieval with a cost calculation.

Where a mechanism from elsewhere in the series matters here, it is restated briefly and then linked.

What goes in and what comes out

A question goes in as ordinary text. A short ranked list of document passages comes back out, each with a score and an identifier. That is the entire input and output of a retrieval step.

The component that performs it is the retriever: the piece of your system that takes a query string, searches the documents, and returns that ranked list. Everywhere below, “the retriever” means exactly that — search and ranking, nothing else. It never writes prose. The model does that afterwards, from what the retriever handed it.

The documents being searched are the corpus. Here is one retrieval against a corpus of product documentation. Read it as two halves: the user’s raw sentence going in, and two scored, identified passages coming back out, best match first.

INPUT — the question, exactly as the user typed it

  "we're on tier 2 — how many requests can we send before we start
   getting throttled?"

OUTPUT — the passages the retriever hands back, best match first

  [doc:api-limits]  (score 0.84)
  Tier 2 accounts are limited to 1,000 requests per minute per API key.
  Bursts to 2,000 rpm are absorbed for up to 10 seconds.

  [doc:api-errors]  (score 0.42)
  Requests over the limit return HTTP 429 with a Retry-After header.

Those two passages are then pasted into the prompt above the user’s question. The model writes its answer out of them and cites the identifiers it was shown. Notice that every fact below appears verbatim in one of the two passages:

"Tier 2 is 1,000 requests per minute per API key, with bursts to 2,000 rpm
 absorbed for up to 10 seconds [doc:api-limits]. Past that you'll get an
 HTTP 429 with a Retry-After header [doc:api-errors]."

That is retrieval-augmented generation in full. Everything else in this chapter is detail on three questions: how those passages got scored and ranked, who decided a lookup was needed, and what it cost. Retrieval as a tool shows this same exchange again as raw traffic against the model’s API — the application programming interface, the HTTP endpoint you send requests to — message by message.

The key structural difference. Classic RAG retrieves once and then answers, for every question. Agentic RAG makes retrieval a tool the model calls when it decides it needs to — possibly several times with refined queries, possibly zero times. Almost every trade-off below comes from that difference.

The vocabulary, defined once

Six terms carry the chapter. Define them before going further.

One pair of terms comes first, because the other six are timed against it. Index time is the offline pass you run once per document version, before any user has asked anything: cut, embed, store. Query time is what happens while a user waits. Work you can move to index time is effectively free; work at query time sits on the latency budget.

Chunk. One passage of a document. Documents are too long to retrieve or read whole, so at index time each one is cut into pieces of a few hundred tokens — a token being the unit models count text in, roughly four characters of English. That cutting is chunking. Retrieval returns chunks, never whole documents. [doc:api-limits] above is a chunk.

Embedding. A fixed-length list of floating-point numbers — 1,024 of them for a typical model. It is produced by a model trained on pairs of related texts, so that texts a human would call related come out pointing in similar directions.

The dimensions have no names. Nothing guarantees dimension 47 means anything nameable. What training fixes is the geometry, not the axes. That is also why two different embedding models are not interchangeable: each invents its own coordinate system (Embedding model version skew the nastiest).

Vector search. The search method that follows from embeddings. Embed every chunk once, at index time, and store the vectors. At query time, embed the question the same way and return the chunks whose vectors point most nearly the same direction as the question’s. The store that holds those vectors and answers direction queries against them is a vector index.

Cosine similarity. How “most nearly the same direction” is measured. It is not the angle itself but the cosine of that angle, which is what makes it a similarity rather than a distance: two vectors pointing the same way score 1.0, two at right angles score 0, and bigger means more alike. It ignores length entirely, so a one-sentence chunk and a full page can score identically. Length carries no signal.

Pooling. The step that produces one vector for a whole chunk. The embedding model emits one vector per token; pooling collapses that variable-length sequence into a single fixed-length vector, usually by averaging over token positions. It is where information is lost, because a few hundred vectors become one.

Pooling is not the only place a pipeline breaks. Of the six stage failures tabulated in The pipeline and where it breaks, pooling causes two — the embedding and search rows — and the other four are unrelated to it. Pooling is worth understanding first because it is the least obvious of the six, not because it is the only one.

Top-k. The size of the answer: return the k highest-scoring chunks and discard the rest. k = 5 in the example above would have returned five passages instead of two.

Cosine similarity, substituted

Scores appear throughout this chapter, so work the formula once with real numbers. Two pieces of arithmetic go into it, and both are worth naming before you meet them:

• The dot product of two vectors: multiply matching coordinates and add the results. Written a·b.
• The norm of a vector, written |a|: its length, the square root of the vector dotted with itself.

Cosine similarity is cos(a,b) = (a·b) / (|a| · |b|) — the dot product divided by both lengths, which is exactly what removes length from the answer.

Below, a three-dimensional query vector is scored against two documents: one on-topic, one not. Watch the two cosines come out far apart, and watch the last line double a document’s vector without moving its score at all.

query  "annual plan refund window"        a = [0.8, 0.5, 0.1]
doc 1  "Annual plans are refundable ..."  b = [0.7, 0.6, 0.2]
doc 2  "Invoices are issued monthly."     c = [0.1, 0.2, 0.9]

step 1 — norms (each vector dotted with itself, then square-rooted)
         a·a = 0.64 + 0.25 + 0.01 = 0.90   ->  |a| = √0.90 = 0.9487
         b·b = 0.49 + 0.36 + 0.04 = 0.89   ->  |b| = √0.89 = 0.9434
         c·c = 0.01 + 0.04 + 0.81 = 0.86   ->  |c| = √0.86 = 0.9274

step 2 — dot products (multiply matching coordinates, add)
         a·b = 0.8·0.7 + 0.5·0.6 + 0.1·0.2 = 0.56 + 0.30 + 0.02 = 0.88
         a·c = 0.8·0.1 + 0.5·0.2 + 0.1·0.9 = 0.08 + 0.10 + 0.09 = 0.27

step 3 — divide by both norms
         cos(a,b) = 0.88 / (0.9487 · 0.9434) = 0.88 / 0.8950 = 0.983
         cos(a,c) = 0.27 / (0.9487 · 0.9274) = 0.27 / 0.8798 = 0.307

step 4 — now double doc 1:   2b = [1.4, 1.2, 0.4]
         a·2b = 1.12 + 0.60 + 0.04 = 1.76        (the dot product doubled)
         |2b| = √(1.96 + 1.44 + 0.16) = √3.56 = 1.8868   (the norm doubled)
         cos(a, 2b) = 1.76 / (0.9487 · 1.8868) = 1.76 / 1.7900 = 0.983

Step 3 is the ranking: 0.983 for the refund document against 0.307 for the invoicing one. That gap is the whole of what vector search runs on.

Step 4 is length being ignored. Doubling every coordinate doubles the dot product on top and doubles the norm underneath, so the two doublings cancel and the score does not move. A one-sentence chunk and a five-paragraph chunk are on equal footing.

Two anchors for reading a score

Retrieval scores throughout this chapter are cosine similarities on exactly that scale, and two reference points make them legible.

• ~0.31 is the noise floor. A query scored against a random document in the same embedding space averages about that, so 0.31 is roughly what “no relationship at all” looks like.
• ~0.89 is a strong match. That was the best match observed in one production corpus (Embedding model version skew the nastiest).

Treat both as illustrative anchors from one corpus and one embedding model rather than as constants. The floor in particular moves with the model, since each one spreads its scores differently. The ten-minute version of measuring your own: score a few hundred query-document pairs you know to be unrelated, and take the mean.

Hold onto the two numbers anyway. Retrieval as a tool asks you to interpret a 0.84 and a 0.42 against them.

Where the system view lives. This chapter is the agent-loop view: when to call retrieval, how to fuse results, how to chunk. What is inside the vector index — the graph and partition structures that make direction queries fast, and what they cost in memory — plus data freshness, splitting an index across machines, keeping one customer’s documents invisible to another, and the cost model in dollars, are all in chapter 06 of the GenAI system-design track, which assumes this one. Every dollar figure quoted below is sourced from there.

1. Classic vs. agentic

Classic RAG retrieves on every question. Agentic RAG lets the model decide — and that decision has a price, worked out below all the way to the case where retrieving is not merely wasteful but actively harmful.

Here are the two designs side by side: one straight path against a loop and a bypass.

flowchart LR
    subgraph C["Classic RAG — fixed pipeline"]
        Q1([Query]) --> E1[Embed] --> S1[Search] --> P1[Stuff top-k] --> A1[Answer]
    end
    subgraph A["Agentic RAG — retrieval is a tool"]
        Q2([Query]) --> M((Model))
        M -->|search| R[Retriever]
        R --> M
        M -->|refine + search again| R
        M --> A2([Answer])
    end

    style A fill:#f8f9fa,stroke:#333

The difference between the two halves is entirely in the arrows.

The classic path is unconditional: embed, search, Stuff top-k passages into the prompt, answer. Four boxes, no branches, one round trip, the same cost whatever was asked.

In the agentic half the model sits between the query and the answer. The only way to reach the Retriever is for the model to emit a tool call. If it emits none, the arrow straight to the answer is taken instead — that arrow is the whole point of the pattern.

The refine + search again edge is the one classic RAG structurally cannot draw, because the refined query does not exist until the first search has returned.

Six differences follow from those arrows. The bolded last row is the one this chapter is built around.

Two of those rows carry more weight than the rest.

“Can skip retrieval” matters more than people expect. Classic RAG retrieves for “hi” and for “what’s 2+2”. The passages it injects do not sit in the prompt neutrally; they consume attention budget the question needed.

Attention is the mechanism by which the model decides how much to weight each earlier token while producing the next one. Those weights are normalized to sum to 1 across all positions. That normalization is what makes it a budget: the share is fixed at 1 no matter how long the prompt gets, so every token of added text takes share away from everything already there.

Forced retrieval is therefore a quality bug before it is a cost one. The next two subsections put numbers on both halves.

“Multi-hop” is the other structural difference. A multi-hop question is one whose answer requires two lookups in sequence, where the second query is only writable once the first has returned.

“Which of our enterprise customers churned after the pricing change?” is one. You need the pricing-change date first, and only then can you write the churn query filtered by it. One retrieval pass structurally cannot do this — it would have to write both queries before either had run.

What the cost multiplier actually is

One billing fact drives everything in this subsection. Every search an agent performs is resent on every call after it. The passages join the conversation history, and the whole history is re-sent as the input of the next call, so a search you paid for once is billed again at every step that follows.

The workload being priced

Everything below is priced against the reference workload in Query time per 1000 queries:

Prices are claude-sonnet-5 rates: $3 per million input tokens, $15 per million output tokens. Every dollar figure in this chapter is at those rates.

One footnote if you price the code as well as the arithmetic: the runnable listing in Tier 3 anthropic sdk calls claude-opus-5, which ch 09 prices at $5/$25. Swap those rates in before you cost that loop.

Classic RAG: one call

Classic RAG makes exactly one model call, with everything in the prompt from the start, so it prices in one line:

classic, one pass:  in 1,000 + 50 + 2,500 = 3,550 ;  out 400
                    3,550·$3/M + 400·$15/M                  =  $0.01665

That $0.01665 is the denominator for every multiplier below.

Agentic RAG: sum over calls, not the last call

An agentic query is not one model call, so it needs an accounting rule before it can be priced. Each search costs a call to ask for it and a further call to use what came back. The figure to compare against classic is therefore the sum over every call the loop makes — not the price of the last one.

Three numbers change from the classic workload:

• The prefix is 1,200 tokens, not 1,050: the 1,000-token system prompt, the 50-token question, and ~150 tokens of tool schema, which is what describing the search tool to the model costs. That schema is in every call, because the model has to be told the tool exists each time.
• Each search emits a ~60-token tool_use block. A tool_use block is the small structured request the model produces instead of prose when it wants a tool run; One turn end to end shows one in full. Those 60 tokens are output on the call that emits them and input on every call after.
• Each search returns 2,500 passage tokens.

So one search adds 60 + 2,500 = 2,560 tokens to the history, permanently. Written out for a single search:

searches = 1, itemised

  call 1   in  1,200                        out  60   (the tool_use block)
           1,200·$3/M +  60·$15/M                        =  $0.00450
  call 2   in  1,200 + 60 + 2,500 = 3,760   out 400   (the answer)
           3,760·$3/M + 400·$15/M                        =  $0.01728
                                                  total  =  $0.02178

Every row of the table below is built by that same pattern: one 60-token call per search, then one 400-token call to answer, each billed on the whole history as it stood at that moment.

Two columns, and they are not the same thing. final prompt is the input of the last call only. total, all calls adds up every call, which is why the total is always larger than pricing the final prompt alone would suggest. vs classic divides that total by $0.01665.

  searches    final prompt    total, all calls    vs classic
     0          1,200 tok         $0.00960          0.58×
     1          3,760 tok         $0.02178          1.31×
     2          6,320 tok         $0.04164          2.50×
     3          8,880 tok         $0.06918          4.15×
     4         11,440 tok         $0.10440          6.27×

Why the growth is quadratic

Notice that the cost roughly doubles from one search to two, but the token count does not: 3,760 to 6,320 is only 1.7×. The gap is the re-billing.

Write it out. Across n model calls, with a prefix P of 1,200 tokens and a = 2,560 tokens added per search, call 1 carries no search, call 2 carries one, call 3 carries two, and so on. Total input billed over the loop is:

n·P  +  a·(0 + 1 + … + (n−1))   =   n·P  +  a·n(n−1)/2

Substitute the four-search row, which is n = 5 calls:

5·1,200  +  2,560·(5·4/2)  =  6,000  +  2,560·10  =  6,000 + 25,600  =  31,600

Check it against the table by adding the five prompts by hand: 1,200 + 3,760 + 6,320 + 8,880 + 11,440 = 31,600. They agree.

The a·n(n−1)/2 term grows with n² while the n·P term grows with n, so for large n the re-billing dominates. This is the same quadratic shape that governs any long agent conversation (Deriving the numbers), with searches standing in for conversational turns. The n(n−1)/2 form, in which the first call carries no history, is that chapter’s convention.

The commonly quoted “2–6×” figure for agentic RAG corresponds to two to four searches. The growth is faster than linear because each search’s passages are re-billed on every call that follows, so the third search costs more than the first even though it retrieves the same volume of text.

The top row is the one this chapter is about. Skipping retrieval lands at 0.58×, cheaper than classic, because the 2,500 passage tokens never enter the prompt.

The decision boundary, priced

The chapter’s thesis is knowing when not to retrieve. That is a claim about a boundary, so here are both sides of it and the line between.

Side one: where retrieval earns its cost

Consider “What’s the rate limit on tier 2 of the API?”

Nothing the model learned in training contains your tier-2 number. So it either reads the number or invents one. Retrieval moves the answer from a fabrication to the gold passage landing in the final five about 87% of the time.

Two terms in that sentence:

• The gold passage is the one chunk a human has labelled as actually containing the answer. It is the thing retrieval is trying to find.
• Recall@5 is the fraction of test questions whose gold passage appears somewhere in the top five results. The 87% figure was measured with 50 candidates passed through a reranker — a second, slower scoring model that re-orders a shortlist (Reranking in the serving path).

Now price it. Skipping costs 0.58× and searching once costs 1.31×, so 1.31 / 0.58 = 2.3: you spent 2.3× to go from roughly 0 to 0.87. Trivially worth it.

Side two: where retrieval actively hurts

Now consider “Write me a Python function that reverses a linked list.”

Classic RAG has no way to abstain, so it searches. Your corpus contains nothing relevant — and that does not produce an empty result. It produces the five least irrelevant documents in the index, scoring around 0.31, the same number named above as the floor for a completely unrelated document.

Three costs follow, and all three are measurable. Before the numbers, one accounting note: both money lines below leave the 150-token tool schema out. That is deliberate. This comparison is classic RAG with and without its retrieved text, and classic RAG has no tool to describe. The 1,200-token prompts in the table above are these same prompts plus that schema.

money      no passages:     1,050·$3/M + 400·$15/M
                            = $0.00315 + $0.00600  =  $0.00915
           five useless:    3,550·$3/M + 400·$15/M
                            = $0.01065 + $0.00600  =  $0.01665
           $0.01665 / $0.00915  =  1.82×  for zero information

attention  the question is 50 tokens of a 1,050-token prompt
                            50 / 1,050  =  4.76%
           the same question in a 3,550-token prompt
                            50 / 3,550  =  1.41%
           4.76% / 1.41%  =  3.4× dilution of the only part that matters

latency    +73 ms of embed, search, fuse and rerank, spent on nothing

Attention is the cost most people overlook. A model spreads its fixed attention budget across everything in the prompt, so 2,500 tokens of off-topic text do not sit there inertly. They compete, and the question’s share of the total falls by 3.4×.

Position makes it worse. Retrieved passages conventionally go after the system prompt and before the question, which is the middle of the window — and a model’s ability to recall a fact is U-shaped across position: strong at the very beginning, strong at the very end, weakest in the middle (Why quality degrades in long contexts). So retrieval pays for 2,500 tokens and drops them in the weakest slot on that curve.

On the 73 ms. Treat it as one pipeline’s stopwatch reading rather than a constant. It is dominated by the reranker and moves with your hardware and your candidate count. The durable point is the order of magnitude: a wasted retrieval costs tens of milliseconds, not microseconds, and every one of them sits on the user’s critical path.

The line between the two sides

Let p be the fraction of traffic whose answer depends on a corpus fact. Then:

• Always-retrieve pays $0.01665 on every query, whatever p is.
• Retrieval-as-a-tool pays $0.00960 when it skips and $0.02178 when it searches once.

Both of the agentic figures carry the 150-token tool schema, because an agent cannot choose to call a tool it was never told about. That schema is the entire gap between the $0.00915 skip figure in the block above — which was classic RAG stripped of its passages, no tool involved — and the $0.00960 skip figure here: 150·$3/M = $0.00045, and $0.00915 + $0.00045 = $0.00960.

Blend the two agentic cases by how often each happens, then solve for the p at which the blend equals $0.01665:

blended(p)   =  (1-p)·$0.00960  +  p·$0.02178
             =  $0.00960 + p·($0.02178 − $0.00960)
             =  $0.00960 + p·$0.01218

break-even:     $0.00960 + p·$0.01218  =  $0.01665
                p·$0.01218  =  $0.01665 − $0.00960  =  $0.00705
                p  =  $0.00705 / $0.01218  =  0.579

p = 0.20   ->  0.80·$0.00960 + 0.20·$0.02178  =  $0.01204   28% cheaper
p = 0.35   ->  0.65·$0.00960 + 0.35·$0.02178  =  $0.01386   17% cheaper
p = 0.579  ->  $0.01665                                     identical
p = 0.80   ->  0.20·$0.00960 + 0.80·$0.02178  =  $0.01934   16% dearer

(The percentages compare each blend against always-retrieve’s $0.01665: 1 − 0.01204/0.01665 = 28%, and 0.01934/0.01665 − 1 = 16%.)

Below roughly a 58% corpus-question rate, letting the model skip is cheaper and avoids the dilution above. Past it you are paying a premium — 16% at p = 0.80 — for the option to abstain, and the quality argument has to carry it alone.

The boundary is sensitive to one assumption. If your corpus questions typically need two searches rather than one, at $0.04164, the gap term grows from $0.01218 to $0.04164 − $0.00960 = $0.03204, and break-even falls to $0.00705 / $0.03204 = 0.22. On a multi-hop corpus you are buying capability, not savings, and you should say so rather than pretending agentic RAG is a cost optimization.

The failure you accept in exchange is a wrong skip — the model deciding, from its own background knowledge, that it already knows your refund window. Preventing that is the job of the tool description in Retrieval as a tool, and it is why the boundary belongs in the prompt rather than in a separate router you tune on the side.

The rule, stated so you can apply it without the arithmetic: retrieve when the answer depends on a fact that is (a) specific to your corpus and (b) able to change. Skip when either condition fails.

Three questions against that rule:

• “What’s the rate limit on tier 2?” — satisfies (a) and (b). Retrieve.
• “Reverse a linked list in Python.” — fails (a); the answer is in no corpus of yours. Skip.
• “What did we decide about pricing last week?” — the genuinely hard one. It fails (a) if the decision is a few turns up in this conversation, and satisfies it if the decision is written in a document. That makes it a memory question first (ch 04) and a retrieval question second.

What interviewers probe: “When is classic RAG the right answer?” High-volume, single-hop, latency-sensitive question answering over a stable corpus. The round trips buy nothing there.

2. The pipeline, and where it breaks

Every retrieval pipeline is the same six stages, and each stage fails in its own characteristic way. Two of those stages — chunking and hybrid search — repay attention far out of proportion to the effort they take.

Two chains run through the pipeline and meet at Hybrid search: one starting at Documents, which runs at index time, and one starting at Query, which runs while the user waits.

flowchart TD
    D[Documents] --> CH[Chunk]
    CH --> EM[Embed]
    EM --> IX[(Vector index)]
    Q([Query]) --> QR[Rewrite / expand]
    QR --> HY[Hybrid search<br/>dense + BM25]
    IX --> HY
    HY --> RR[Rerank<br/>cross-encoder]
    RR --> CX[Assemble context]
    CX --> GEN((Model)) --> ANS([Answer + citations])

    style RR fill:#2d6a4f,color:#fff
    style HY fill:#2d6a4f,color:#fff

The index-time path runs once per document version: chunk, embed, write to the Vector index. That index is a black box here on purpose; what is inside it, and what it costs in memory, is Vector index internals.

The query-time path runs once per question: Rewrite / expand, search, Rerank cross-encoder, Assemble context, generate Answer + citations.

Only the query-time path is on the latency budget. That is why almost all the expensive work is pushed onto the index-time path, where a user is not waiting on it.

Hybrid search and Rerank — the two shaded boxes — are the stages with the highest quality gained per unit of effort spent. Most teams tune chunk size for a week and skip both of them.

The rewrite box

Rewrite / expand is easy to overlook but does real work. It does three things:

• Resolves pronouns against the conversation. “Does it support single sign-on?” becomes “does the Pro plan support single sign-on?”
• Adds vocabulary the corpus is likely to use where the user’s differs. “It crashed” becomes “crash, exception, stack trace”.
• Splits a compound question into two queries.

In agentic RAG this box vanishes into the model, because writing the query is the tool call. That is why the agent in Retrieval as a tool searches "rate limit tier 2" rather than the user’s whole sentence.

Three terms from the shaded boxes

Hybrid search dense + BM25 and Rerank cross-encoder each get a subsection of their own below, but both appear in the failure table that follows, so take their vocabulary now.

• Hybrid search runs two different search methods over the same query and merges their results.
• BM25 is a decades-old lexical scoring formula: it ranks a document by how many of the query’s exact words it contains, weighting rare words far more heavily than common ones.
• A cross-encoder is the reranking model. It takes the query and one candidate passage and feeds them through the model together, in a single pass over the pair, so it can judge how well that specific passage answers that specific question. That is why it is accurate, and also why it is slow: it must be run once per candidate and cannot be precomputed, so it only ever sees a shortlist that cheaper search has already narrowed.

The six stages and their characteristic failures

Each row below is one stage from the diagram, the way that stage characteristically goes wrong, why, and what to do. Read the Mechanism column as the load-bearing one — the six failures are unrelated to each other, and the fixes only make sense against them.

Two of those six rows are the pooling failures named in the vocabulary section: Embed and Search. The other four have nothing to do with pooling. They come from a splitter that ignores structure, from what top-k does and does not ask for, from a context budget that forces truncation, and from prompt design.

Three of the rows are routinely misread. Take them in turn.

The Search row: why hybrid search exists at all. An identifier like ERR_4021 is split by the tokenizer — the component that cuts text into tokens before the model sees it — into several low-information fragments. Pooling then averages those fragments into a mean over hundreds of tokens, where they leave almost no trace. Vector search is not blind to the identifier; it simply cannot rank on it (Embeddings and why dense search misses err_4021).

The Rerank row: near-duplicates. Nothing in a top-k query asks for variety. It asks for the k nearest vectors, full stop.

Suppose your corpus holds the same release note in a PDF, a help-center article, a changelog entry, and two translations. All five are near the query, so all five make the cut at k = 5. You have spent the entire context budget on one fact restated five ways, and the sixth-ranked passage that actually held the answer never entered the prompt. Dedupe by source document before truncating, not after.

The Assemble row: saying how many were dropped. The recipient of that message is the model, in the prompt — a line like [3 additional passages omitted for length] after the last passage.

It matters because of how the two cases differ in behavior. A model that believes it has seen everything relevant answers confidently from a partial view. A model told the view is partial reliably hedges or asks to search again — and in an agentic loop it can act on that by refining the query. Silent truncation removes the only signal that would have triggered the retry.

Chunking

Where a document gets cut decides what retrieval can ever find: a fact split across a boundary is a fact no chunk contains whole.

Five ways to cut. The first is the common default; the two in bold are usually the best choices.

Overlap means letting consecutive chunks share their last and first few dozen tokens. It exists for exactly one reason: so that a fact split across a boundary appears whole in at least one chunk.

That is the entire justification, which means that if your splits respect structure you need much less of it. Overlap is compensation for bad boundaries, not a good in itself.

What a bad boundary costs

Here is a boundary error traced all the way to a wrong answer.

One number in the trace needs stating up front, because the whole failure hinges on it: the context budget admits two passages, not the five the workload in What the cost multiplier actually is was priced with. That is the realistic case rather than a rigged one. This is a chat product several turns into a conversation, and the earlier turns have already eaten most of the window, so what remains for retrieved text fits two passages and not five.

Why that matters: a boundary error is survivable while the budget is generous, because the passage holding the answer is still admitted somewhere down the list. It becomes an outright wrong answer the moment the budget tightens — and budgets tighten with every turn the conversation runs.

document "Billing FAQ", section 4, split at a fixed 500 tokens
  chunk 7 (ends):    "... Annual plans may be refunded within"
  chunk 8 (begins):  "30 days of the renewal date, minus any usage."

query: "how long do I have to get a refund on an annual plan?"

dense top-3
  1. billing-faq#7   0.71   has "annual", "refunded", "plans"
  2. billing-faq#2   0.64   general refund overview
  3. billing-faq#8   0.58   has "30 days", "renewal" — but not "refund"

context budget admits 2 passages   ->   chunk 8 is dropped

grounded model:      "Annual plans are refundable, though the specific
                      window is not stated in the documentation I have."
less-grounded model: "within 14 days."    <- fluent, confident, and wrong

The retriever did not fail. It ranked the chunk carrying the query’s vocabulary first, exactly as designed.

The splitter created the failure at index time, by cutting mid-sentence. The two halves of the answer ended up on opposite sides of a boundary:

• The lexical signal — the words “annual” and “refunded” — stayed in chunk 7.
• The content — the phrase “30 days” — went into chunk 8.

So the passage that scores well does not contain the answer, and the passage that contains the answer does not score well. Fifty tokens of overlap would put “may be refunded within 30 days of the renewal date” whole into chunk 7, and the failure disappears.

That trace also unpacks two table rows that otherwise just restate their own names.

• Sentence-window would retrieve chunk 8 as a single sentence and then hand the model its neighbours. The ranking unit stays small, which keeps scoring precise; the context unit becomes large, which makes the answer complete.
• Parent-document is the same trade one level up: rank on the chunk, feed the whole parent section to the model.

Both decouple “what you match on” from “what you show the model”, which is the general move.

Keep this example in mind for Evaluating retrieval. It is the chapter’s canonical plausible-but-wrong retrieval, and it is invisible to the metric most teams watch. Chunk 8 was retrieved, so Recall@10 scores this query a success, and the answer is fluent and on-topic. Faithfulness is what catches it — “14 days” is supported by no passage in the assembled context — and citation accuracy catches it too, if the model bothered to cite at all.

The same trace with the full fix table and a dollar cost per option is Chunk boundary loss.

Contextual chunking, and its caveat

One figure from ch 06 is worth carrying over. Contextual chunking costs one small model call per chunk at ingest: $2,200 per million documents against a $12,000 parsing bill, so 2,200 / 12,000 ≈ 18% on top of what you were already spending. That makes it an easy yes to try rather than a research project.

Why it works so well. A chunk reading “This limit was raised to 1000 in v3.” is nearly unretrievable, because it names no subject — nothing in it points at your product, your API, or rate limits. Prepending “From: API Rate Limits, Acme Platform v3 docs” gives the embedding the entities it needs in order to land near a query about Acme rate limits. You are repairing the information that chunking destroyed.

The caveat on “largest single-technique gain”. That ranking comes from practitioner write-ups on particular corpora, not from a controlled benchmark across many, so treat it as a strong prior rather than a measured constant.

The mechanism above says exactly where it should hold and where it should not: the gain scales with how context-dependent your chunks are. Documentation full of “this limit” and “the above endpoint” has a great deal to repair. Self-contained FAQ entries that each name their own subject have almost none, and there the technique buys you an ingest bill and little else.

Hybrid search

Two search methods with opposite blind spots beat either one alone — provided their result lists can be merged without comparing incomparable scores, and a reranker then narrows what the merge lets through.

The whole design is five boxes: the query fans out into two searches, the two result lists merge into one, and that one list is narrowed twice.

flowchart LR
    Q([Query]) --> D["Dense<br/>semantic"]
    Q --> B["BM25<br/>lexical"]
    D --> F["Reciprocal rank fusion"]
    B --> F
    F --> R["Rerank top-50"] --> K["Top-5 to context"]

    style F fill:#2d6a4f,color:#fff
    style R fill:#2d6a4f,color:#fff

One query fans out to Dense semantic search and BM25 lexical search in parallel. Reciprocal rank fusion merges the two ranked lists into one. Rerank top-50 scores the survivors properly. Top-5 to context is what the model finally sees. Fifty in, five out — the funnel is the point.

Two methods with inverse blind spots

The word dense names the vector search defined at the top of this chapter. Every chunk becomes a dense list of a thousand-odd nonzero floats, hence the name. Lexical names matching on the literal words.

The two have inverse failure modes, which is exactly why fusing them works.

• Dense search finds paraphrases and misses rare literals. Pooling into 1,024 floats is a lossy compression of meaning, and identifiers are what it discards.
• BM25 finds rare literals because they are rare, and misses paraphrase entirely. Its weighting term is inverse document frequency: a word appearing in three documents out of a hundred thousand counts as far stronger evidence than a word appearing in forty thousand of them.

So a query for “how do I reset my password” is dense search’s case, and a query for ERR_4021 is BM25’s. Run both and you cover both.

Reciprocal rank fusion

You now have two ranked lists to combine, and their scores are not comparable — a cosine of 0.84 and a BM25 score of 34.2 are on unrelated scales, and any weighted sum of the two is a guess about a conversion nobody has.

Reciprocal rank fusion (RRF) sidesteps that by throwing the scores away and using only ranks: first place, second place, third place. That is why it is the default rather than a weighted sum.

score(d)  =  Σ_i  1 / (k_rrf + rank_i(d))        k_rrf = 60

Read the formula piece by piece. For a document d, i runs over retrievers, not documents — one term per ranked list d appears in. rank_i(d) is where d placed in list i. Each placing contributes 1 / (60 + rank), and those contributions are added.

Two notes on the 60:

• It is not derived from anything about your corpus. It is the value used in the paper that introduced the method (Cormack, Clarke and Buettcher, “Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods”, SIGIR 2009), and it has been copied as the default ever since. Take it as a convention that works, and reach for it before you reach for a parameter sweep.
• Watch the name collision. This k_rrf is a smoothing constant with nothing whatever to do with the k in Stuff top-k, Recall@k, or Top-5 to context. The collision is universal in the literature, so learn to disambiguate it rather than expecting it to be fixed.

Now substitute. Three documents, their placings in each list, and the sum worked out. Watch which one wins.

B wins, and A — the document dense search ranked first — loses. That comparison is the entire argument for fusion: being decent in both lists beats being excellent in one, because a document only one retriever likes is usually a document only one retriever’s failure mode liked.

C is instructive for the opposite reason. Being absent from BM25’s list costs it nothing punitive; it simply accumulates one term instead of two. RRF has no penalty, only accumulation. That is what makes it tolerant of a retriever that returns a short list, or no list at all.

What the constant 60 is actually doing

The 60 flattens the head of each list. Compare rank 1 against rank 3 with it in place:

with k_rrf = 60    rank 1 -> 1/61 = 0.01639
                   rank 3 -> 1/63 = 0.01587
                   0.01639 / 0.01587 = 1.033, so rank 1 is worth 3.3% more

with k_rrf = 0     rank 1 -> 1/1  = 1.000
                   rank 3 -> 1/3  = 0.333
                   rank 1 is worth 3× more

Rerun the table at k_rrf = 0 and the result flips: A scores 1/1 + 1/30 = 1.033 and beats B’s 1/3 + 1/2 = 0.833. Fusion collapses into “whoever’s top hit is loudest wins”.

The constant is what converts a rank into evidence rather than a verdict, and it is why RRF is robust to one retriever being badly calibrated.

Why the reranker is the cheapest quality win

The cross-encoder is best understood against what it replaces.

Your search index uses a bi-encoder: query and document are embedded separately and compared only as two finished vectors. That separation is not a design flaw. It is the only reason an index can exist, because it lets every document vector be computed once, long before any query arrives.

The price of that separation is that the query and the document never actually see each other. The comparison is between two independent summaries, not a reading of the pair.

The cross-encoder pays that price back. Attention runs across query and passage at once, so the model can notice that this passage answers this question. It is far more accurate — and far too slow to run over a whole corpus, since nothing about it can be precomputed.

Hence the funnel: retrieve 50 cheaply, rerank those 50 expensively, keep 5. Recall comes from the wide net, precision from the reranker.

3. Retrieval as a tool

Everything so far has priced the model’s right to decide; here is the code that grants it. The same loop appears at three levels of concreteness — pseudocode, a graph framework, a real API call — and then runs for one full turn as raw message traffic.

Tier 1 — Pseudocode

The shape of the loop, with everything framework-specific removed. The one line to look at is if reply.done — that early return is the branch classic RAG does not have.

tools = [search_docs(query, filters), read_document(id)]
loop:
    reply = model(history, tools)
    if reply.done: return reply
    for call in reply.tool_calls:
        history += run(call)          # model decides IF and WHAT to search

Six lines, and the model now owns three decisions that classic RAG makes for it:

1. Whether to search at all — it may return on the first pass with no tool call.
2. What query to use — it writes the query string, rather than the user’s raw sentence being embedded.
3. Whether the results sufficed — after run(call) appends results, the loop goes round again and the model may search a second time.

Tier 2 — LangGraph

Why a graph framework belongs between pseudocode and the SDK at all: the retry loop is what it buys. Tiers 1 and 3 both do the job without one because neither retries. But “grade the passages, and if they are bad rewrite the query and go back” is a state machine, and a graph framework is the alternative to hand-rolling one.

The next step up therefore adds a grading node: retrieve, judge whether the passages actually answer the question, then rewrite the query and retry if they do not. The pattern has a name, Self-RAG — the model critiquing its own retrieval before it uses it, rather than trusting whatever the index handed back. Advanced variants lists it beside the other named variants.

Three things to know before reading the code:

• LangGraph is a library for describing an agent as an explicit graph of nodes and edges. A node is a function; an edge says which node runs next.
• State is the dictionary passed from node to node. TypedDict declares which keys that dictionary is allowed to have.
• total=False says every one of those keys is optional. A state dictionary missing tries entirely is still a valid State as far as the type checker is concerned. That is convenient, because each node returns only the keys it changed — and it is also precisely what lets the bug flagged in the comment below exist at all.

The graph has five nodes and one cycle. Watch route, which is the only place a decision gets made, and watch the end of the listing, where the graph is invoked with tries seeded and then all three of route’s branches are asserted.

from typing import TypedDict
from langgraph.graph import StateGraph, END


class State(TypedDict, total=False):
    query: str
    docs: list
    ok: bool
    tries: int          # MUST be seeded to 0 — an unset key makes give_up unreachable
    answer: str


def retrieve(s):  return {"docs": retriever.invoke(s["query"])}
def grade(s):     return {"ok": judge.invoke({"q": s["query"], "docs": s["docs"]})}
def rewrite(s):   return {"query": llm.invoke(f"Rewrite for search: {s['query']}").content,
                          "tries": s["tries"] + 1}
def generate(s):  return {"answer": llm.invoke(prompt(s)).content}
def give_up(s):   return {"answer": "I could not find this in the documentation."}


def route(s):
    if s["ok"]:          return "generate"
    if s["tries"] >= 2:  return "give_up"    # never loop forever
    return "rewrite"


g = StateGraph(State)
for name, fn in [("retrieve", retrieve), ("grade", grade), ("rewrite", rewrite),
                 ("generate", generate), ("give_up", give_up)]:
    g.add_node(name, fn)

g.set_entry_point("retrieve")
g.add_edge("retrieve", "grade")
g.add_conditional_edges("grade", route, ["generate", "rewrite", "give_up"])
g.add_edge("rewrite", "retrieve")           # the loop
g.add_edge("generate", END)
g.add_edge("give_up", END)
app = g.compile()

# Seeding tries is the caller's job, and it is not optional: total=False lets
# the key be absent, and rewrite() would then raise KeyError on s["tries"].
result = app.invoke({"query": "how long do I have to get a refund?", "tries": 0})
print(result["answer"])

assert route({"ok": False, "tries": 0}) == "rewrite"
assert route({"ok": False, "tries": 2}) == "give_up"     # the branch people forget
assert route({"ok": True,  "tries": 5}) == "generate"

Four names in that listing are yours to supply: retriever (a vector store), judge (a model asked to grade the passages — ch 08), llm (a model), and prompt (a prompt builder). Everything else is the graph: five nodes, one conditional edge, and exactly one cycle, rewrite -> retrieve -> grade -> rewrite.

The give_up branch is the part people forget. An agent that cannot find the answer must say so.

Take the edge out and follow what happens. route never returns give_up, so a bad grade always sends the graph back to rewrite. The query gets rewritten eleven times, each rewrite drifting further from the user’s actual question, and eventually the model answers from prior knowledge — the exact failure RAG existed to prevent. Note that give_up is an edge and a terminal node, which is why both appear in the listing.

The tries counter is the only thing standing between a cycle and an infinite one, so an unseeded tries is not a cosmetic bug. Two ways it bites:

• If tries is absent, rewrite() raises KeyError on s["tries"] the first time it runs. That is the lucky case, because it is loud.
• If your state class defaults it to something truthy, s["tries"] >= 2 may never fire and give_up is never reached.

That is why the listing ends by actually running the graph. app.invoke({..., "tries": 0}) is the seeding, it belongs to the caller rather than to any node, and a graph compiled but never invoked with it is the version that ships broken.

Tier 3 — Anthropic SDK

The same loop written against the real API, using Anthropic’s Python SDK — software development kit, the client library that wraps the HTTP calls.

A tool here is a JSON schema describing a function the model may ask you to run. The model emits a request to call it, your code runs it, and you send the result back as another message.

Read the listing in three parts: SEARCH_TOOL (what the model is told about the search function), SYSTEM (the grounding rules), and rag_agent (the loop that turns tool_use requests into tool_result messages). The single most important line is in the tool description — the sentence beginning “Skip it for”.

import anthropic

client = anthropic.Anthropic()

SEARCH_TOOL = {
    "name": "search_docs",
    "description": (
        "Search internal product documentation. Call this whenever the answer "
        "depends on product behavior, pricing, limits, or policy — never answer "
        "those from memory, since docs change weekly. Skip it for greetings, "
        "general programming questions, or anything already in this conversation. "
        "If the first search misses, call again with different keywords."
    ),
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {"type": "string",
                      "description": "Keywords, not a sentence. e.g. 'rate limit tier 2'"},
            "product": {"type": "string", "enum": ["api", "dashboard", "billing"]},
        },
        "required": ["query"],
        "additionalProperties": False,
    },
    "strict": True,
}

SYSTEM = """You answer from retrieved documentation only.

Rules:
- Cite the source id after every factual claim, like [doc:api-limits].
- If the passages do not contain the answer, say so and suggest what to search next.
- Never fill a gap with prior knowledge about how such products usually work."""

def rag_agent(question: str, max_steps: int = 6) -> str:
    messages = [{"role": "user", "content": question}]
    for _ in range(max_steps):
        resp = client.messages.create(
            model="claude-opus-5",
            max_tokens=4096,
            system=[{"type": "text", "text": SYSTEM,
                     "cache_control": {"type": "ephemeral"}}],   # stable → cached
            tools=[SEARCH_TOOL],
            messages=messages,
        )
        if resp.stop_reason != "tool_use":
            return next(b.text for b in resp.content if b.type == "text")

        messages.append({"role": "assistant", "content": resp.content})
        results = []
        for b in resp.content:
            if b.type != "tool_use":
                continue
            hits = retriever.search(**b.input)[:5]
            body = "\n\n".join(
                f"[doc:{h.id}] (score {h.score:.2f})\n{h.text}" for h in hits
            ) or "No matching documents."
            results.append({"type": "tool_result", "tool_use_id": b.id, "content": body})
        messages.append({"role": "user", "content": results})
    return "Could not find a grounded answer within the step budget."

The loop reads as four steps. Send the conversation plus the tool schema. If the model stopped for any reason other than wanting a tool, return its text. Otherwise run each requested search and append the results. Go round again.

The caching marker, and why it does nothing here

Marking the system prompt "cache_control": {"type": "ephemeral"} tells the API to store the processed form of that unchanging prefix, so later calls in the same conversation are billed at roughly a tenth of the normal input rate (Prompt caching derived).

As written, that discount does not arrive — and the reason is a size floor, not a bug in the marker.

A prefix only caches once it clears the model’s minimum cacheable length. Three models, three floors, and note that they are not ordered by generation, so a cheaper tier is not a safer one (Prompt caching the highest leverage lever):

Now measure this loop’s prefix, which is tools followed by system. The SEARCH_TOOL literal as written above is 867 characters and SYSTEM is 287. At the four-characters-per-token rule of thumb (Tokens):

(867 + 287) / 4  =  1,154 / 4  ≈  290 tokens
290 / 512  ≈  57% of the floor

A bit over half. Below the floor nothing caches, there is no error, and cache_creation_input_tokens simply comes back as 0.

Keep the marker anyway. A production version of this agent carries citation-format rules, refusal language and several more tool schemas in the same stable block, and crosses 512 without trying. But verify rather than assume: read usage.cache_read_input_tokens on the second call of a conversation, and treat a zero as “the prefix is too short, or something in it is changing” — not as “caching is on”.

Four deliberate choices in that code

1. The description says when not to search. Without that boundary, agentic RAG retrieves on “thanks!” — reintroducing the exact failure it exists to avoid.
2. Passages carry [doc:id] inline. Citations become copyable rather than invented. A model asked to cite without visible ids will fabricate plausible-looking ones.
3. Scores are included. The model can weigh a 0.31 hit differently from a 0.89 one — the noise floor and a strong match on the scale set out at the top of this chapter. Those two are the ends of the scale rather than this example’s values; the transcript below returns a 0.84 and a 0.42, which is what the interpretation has to work on in practice. Given the scale, the model will often say “the match is weak” instead of asserting. Include the scale in the system prompt if you want this reliably — a bare float means nothing without one.
4. The empty case returns text, not "". The model needs to see “No matching documents” in order to change strategy. An empty string reads as a malformed result.

The same loop, runnable as-is

The listing above leaves retriever for you to supply; here is the whole claim — the model decides when to search — in a form you can paste into a file and run. The corpus is four documents in a list, and search is plain keyword overlap: a production system swaps that one function body for the hybrid dense + lexical retrieval of Hybrid search and changes nothing else.

import anthropic

client = anthropic.Anthropic()

CORPUS = [
    {"id": "api-limits", "text": "Tier 2 accounts are limited to 1,000 requests per minute per key."},
    {"id": "api-errors", "text": "Requests over the limit return HTTP 429 with a Retry-After header."},
    {"id": "billing-refunds", "text": "Annual plans may be refunded within 30 days of the renewal date."},
    {"id": "sso-setup", "text": "Single sign-on is available on the Pro plan and above."},
]

def search(query: str) -> list:
    """Keyword overlap. Production swaps this body for hybrid dense + BM25."""
    terms = set(query.lower().split())
    scored = [(len(terms & set(d["text"].lower().split())), d) for d in CORPUS]
    return sorted([s for s in scored if s[0] > 0], reverse=True, key=lambda s: s[0])[:3]

SEARCH_TOOL = {
    "name": "search_docs",
    "description": "Search internal product documentation. Call this when the "
                   "answer depends on product limits, pricing, or policy. Skip "
                   "it for greetings and general programming questions.",
    "input_schema": {
        "type": "object",
        "properties": {"query": {"type": "string", "description": "Keywords, not a sentence."}},
        "required": ["query"],
    },
}

def ask(question: str, max_steps: int = 6) -> str:
    messages = [{"role": "user", "content": question}]
    for _ in range(max_steps):
        resp = client.messages.create(
            model="claude-opus-5",
            max_tokens=1024,
            tools=[SEARCH_TOOL],
            messages=messages,
        )
        if resp.stop_reason != "tool_use":               # no search wanted: done
            return next((b.text for b in resp.content if b.type == "text"), "")
        messages.append({"role": "assistant", "content": resp.content})
        results = []
        for block in resp.content:
            if block.type != "tool_use":
                continue
            body = "\n\n".join(f"[doc:{doc['id']}] matched {n} terms\n{doc['text']}"
                               for n, doc in search(block.input["query"]))
            results.append({"type": "tool_result", "tool_use_id": block.id,
                            "content": body or "No matching documents."})
        messages.append({"role": "user", "content": results})
    return "Step budget exhausted without a final answer."

print(ask("we're on tier 2 - how many requests before we get throttled?"))
print(ask("thanks, that's exactly what I needed"))

Three lines carry the argument. The model writes block.input itself, so the conversational question reaches the retriever as keywords — the query rewriting of The rewrite box, for free. The passages go back as a tool_result the model can react to: cite them, answer, or search again with different terms if the match looks thin. And the loop is the ch 01 agent loop with exactly one tool plugged in, which is the point — agentic RAG is not a new architecture, just a retriever behind a tool schema. Run it: the first print triggers a search, and the second returns in a single call with none, the 0.58× row of What the cost multiplier actually is.

One turn, end to end

Everything above asserts behavior. Here is that loop actually running, as the messages array sees it — the same tier-2 question the chapter opened with, now with the wire traffic shown.

The first block is the user’s message going in, and the model’s reply coming back. The reply contains two blocks, not one: a sentence of prose and a structured request to run a tool.

USER
  {"role": "user", "content": "we're on tier 2 — how many requests can we
                               send before we start getting throttled?"}

--- client.messages.create(...)  ->  stop_reason: "tool_use" ---

ASSISTANT
  {"role": "assistant", "content": [
     {"type": "text", "text": "Let me check the API docs."},
     {"type": "tool_use", "id": "toolu_01A", "name": "search_docs",
      "input": {"query": "rate limit tier 2", "product": "api"}}]}

The stop_reason: "tool_use" field is the API saying the model has stopped mid-answer because it wants a tool run. Your code is now responsible for running it.

Now look at input. The user wrote fifteen words of conversational English:

"we're on tier 2 — how many requests can we send before we start getting throttled?"

The model searched four keywords and set a filter:

{"query": "rate limit tier 2", "product": "api"}

That is the Rewrite / expand box from The pipeline and where it breaks, executed by the model instead of by a separate pipeline stage. It is exactly what the schema’s “Keywords, not a sentence” description bought.

Next, your code runs the search and hands the passages back. Note the role on that message — it is user, which surprises people.

USER   (tool results are user-role blocks — this is required, not stylistic)
  {"role": "user", "content": [
     {"type": "tool_result", "tool_use_id": "toolu_01A", "content":
        "[doc:api-limits] (score 0.84)\n"
        "Tier 2 accounts are limited to 1,000 requests per minute per API key.\n"
        "Bursts to 2,000 rpm are absorbed for up to 10 seconds.\n\n"
        "[doc:api-errors] (score 0.42)\n"
        "Requests over the limit return HTTP 429 with a Retry-After header."}]}

--- client.messages.create(...)  ->  stop_reason: "end_turn" ---

ASSISTANT
  "Tier 2 is 1,000 requests per minute per API key, with bursts to 2,000 rpm
   absorbed for up to 10 seconds [doc:api-limits]. Past that you'll get an
   HTTP 429 with a Retry-After header [doc:api-errors]."

Three of the four choices above are now visible rather than asserted:

• Copyable citations. [doc:api-limits] and [doc:api-errors] in the answer were copied out of the tool result character for character. The model never had to invent an id, because it was looking at one.
• Scores used as evidence strength. The 0.84 hit is stated flatly. The 0.42 hit — closer to the 0.31 noise floor than to the 0.89 strong match — carries only a secondary detail.
• Two model calls where classic RAG makes one. That is exactly the searches = 1 row of the cost table in What the cost multiplier actually is, at $0.02178 against classic’s $0.01665.

Now the same agent, with the same tools and the same system prompt, on the next message. Count the model calls:

USER       {"role": "user", "content": "thanks, that's exactly what I needed"}

--- client.messages.create(...)  ->  stop_reason: "end_turn" ---

ASSISTANT  "Happy to help — shout if you start seeing 429s."

There is no tool_use block and only one model call. That is the searches = 0 row, at 0.58× the cost of classic. Four lines of transcript carry the chapter’s whole thesis, and classic RAG cannot produce this trace at any price, because its retrieval step has no if.

And now the case that is genuinely hard. Take “What did we decide about the pricing change last week?”

Nothing on its surface says which way to go, because “we decided” is ambiguous about where the record lives. Both branches have a real cost:

• If the decision is four turns up in this same conversation, searching is the harm case from The decision boundary priced with extra steps: off-topic passages, 1.82× the tokens, the question diluted from 4.76% of the prompt to 1.41%.
• If the decision was written into a decision log, not searching means answering from a summary the model half-remembers.

The tier-3 tool description resolves it in one clause — “or anything already in this conversation” — which imposes an order: conversation first, index second.

The boundary is not “is this a factual question”. It is “which store owns this fact”, and that is a memory-architecture question before it is a retrieval one (ch 04).

4. Advanced variants

The pipeline and the loop above are the base pattern; the variants people name in interviews each extend one piece of it, and three of them have a mechanism worth being able to defend.

Names first, mechanisms after. Each row is a one-line summary; three of the five are then unpacked below.

Why HyDE works

HyDE stands for hypothetical document embeddings. Understanding it teaches you something about the shape of the embedding space — the shared coordinate system every vector in your index lives in.

The problem it solves: questions and answers occupy different regions of that space. “How do I rotate my API key?” is syntactically nothing like “Navigate to Settings → Security and click Regenerate.” One is interrogative and generic; the other is imperative and full of interface nouns. Embedding the question puts you in the question neighborhood, and your documents are not there.

So HyDE has the model invent a plausible answer, embeds that instead of the question, and searches with it. Answer-shaped text lands near answer-shaped documents.

The invented answer does not need to be correct. If the model guesses “Go to your account settings and regenerate the key,” that guess is wrong about the exact menu path and still lands in the right neighborhood, which is all the search needs.

Why late chunking follows straight from pooling

Ordinary chunking splits first and embeds each chunk independently, so pooling runs over a window that never saw the rest of the document.

Take the chunk “This limit was raised to 1000 in v3.” It pools into a vector with no idea what “this limit” refers to, because no token inside that window ever attended to one. The subject was two paragraphs up, on the other side of a boundary.

Late chunking reverses the order. Run the encoder — the embedding model itself — over the whole document first, so every per-token vector has attended to the whole document. Only then pool per chunk.

Same boundaries, same number of vectors. But each vector now carries context it could not otherwise have had, so it obtains contextual chunking’s benefit mechanically rather than through an extra model call per chunk.

The price: you need an encoder whose context window — the most text it can take in one pass — actually fits your documents.

Why GraphRAG is the one to be able to defend

Top-k retrieval answers questions whose answer lives in some chunk.

“What are the recurring themes across all 400 incident reports?” has an answer that lives in no chunk. It is a property of the collection rather than of any passage in it. No amount of better retrieval fixes that, because there is nothing to retrieve — you could return the perfect top 5 and still not have the answer.

GraphRAG builds entity and relationship structure at index time and traverses it, so aggregate questions become answerable. The cost is a heavy indexing pass and a staleness problem on corpora that change often.

5. RAG vs. tool vs. fine-tune vs. long context

Retrieval competes with three other ways of closing a knowledge gap, and the choice among them turns on what kind of gap the system actually has — not on which technique you were hoping to use.

The whole choice is one root question and five branches:

flowchart TD
    Q{What kind of gap?} -->|Facts that change| RAG[RAG]
    Q -->|Live/transactional data| TOOL[Tool call to the system of record]
    Q -->|Format, tone, task style| FT[Fine-tune]
    Q -->|Small stable corpus < ~100k tok| LC[Just put it in context]
    Q -->|Domain vocabulary| BOTH[Fine-tune + RAG]

    style RAG fill:#2d6a4f,color:#fff
    style TOOL fill:#2d6a4f,color:#fff
    style LC fill:#2d6a4f,color:#fff
    style FT fill:#2d6a4f,color:#fff
    style BOTH fill:#2d6a4f,color:#fff

The root node is the move. Asking What kind of gap? is not the same as asking “should we use RAG”. Asking the second question is how people end up with a vector database full of order records.

Each branch in turn.

Facts that change → RAG

Knowledge that moves and must be cited. This is the case the rest of the chapter is about.

Live/transactional data → a tool call

Send it to the system of record. Order status is a database query, not a retrieval problem.

Teams get this wrong constantly by indexing their orders table into a vector database. Two things go wrong there. The copy is stale the moment a row updates. And an exact-lookup question gets answered with approximate matches.

“Approximate” is literal. Production vector indexes search only part of the space, trading exactness for a roughly 340× speedup. A recall of 0.96 — meaning 96 of every 100 true nearest neighbours are actually found — is normal, entirely fine for ranking passages, and disqualifying for “find order #88213” (Vector index internals).

Flag that word “recall” before it does damage, the same way k_rrf had to be separated from k earlier. Two different quantities share the name:

An index can sit at 0.96 recall while Recall@5 is 0.40. Retrieving the true nearest neighbours is no help when the nearest neighbours are the wrong passages.

Format, tone, task style → fine-tuning

Fine-tuning means further training of the model’s own weights on your examples. It teaches behavior, not facts.

It does not fix hallucination. It changes style, and a confidently wrong answer in the right house style is still wrong.

Small stable corpus → Just put it in context

Long context beats RAG below roughly 100k tokens: no chunking, no index, no retrieval failures. And with prompt caching, the repeated prefix bills at about 10% of the normal rate (Prompt caching derived).

That threshold deserves its arithmetic, because it is not the number people assume it is.

100k corpus, raw:      100,000·$3/M                   =  $0.30 per query
100k corpus, cached:   100,000·$3/M·10%               =  $0.03
  + the question           50·$3/M                    =  $0.00015
  + the answer            400·$15/M                   =  $0.006
                                              total   =  $0.03615

against classic RAG                                   =  $0.01665
$0.03615 / $0.01665  =  2.2× dearer

So at 100k tokens the no-index option is still about 2.2× dearer per query. The actual cost break-even is far lower. Solve for the corpus size T at which the cached total equals $0.01665:

T·$3/M·10% + $0.00015 + $0.006  =  $0.01665
T·$0.0000003                    =  $0.01050
T                               =  35,000 tokens

The ~100k line is therefore an engineering-cost judgement, not a token-cost one. Below it, the couple of cents a query buys you the deletion of a chunker, an index, an embedding-model version to track, and an entire class of retrieval bug — cheap against the salary that would otherwise maintain them.

Above it, two things turn. The per-query cost has grown large enough that the maintenance stops looking expensive by comparison. And the quality argument turns too: a corpus that large puts the answer somewhere in the middle of the window, which is the position a model is least reliable at reading a fact out of (Why quality degrades in long contexts).

Say this branch unprompted in an interview. “Before I build a retrieval stack, is the corpus small enough to cache in context?” is a strong, cost-aware opening that most candidates never consider. They hear “documents” and reach for a vector database reflexively.

Domain vocabulary → Fine-tune + RAG

This branch needs both, because the two levers act on different things.

• Fine-tuning acts on comprehension. After it, a token like ERR_4021, or an industry abbreviation your corpus is full of, is a meaningful unit in this domain rather than noise. The query is understood — and if you fine-tuned the embedding model too, it lands near the right region of the space.
• RAG acts on the fact. What ERR_4021 currently means changed last Tuesday, and no weight update will track that.

One shapes how the question is read; the other supplies the answer.

Ship RAG first regardless, for two reasons. Hybrid search and reranking close most of the vocabulary gap far more cheaply. And fine-tuning an embedding model turns every stored vector into a versioned artifact you must later migrate (Embedding model version skew the nastiest).

6. Evaluating retrieval

Whichever of those branches you took, you eventually have to measure the result — and the first move in measuring is a split. A bad answer is either a retrieval failure or a generation failure, and the fixes are unrelated.

The Layer column below tells you which stage a red number is blaming.

The three retrieval metrics, computed

MRR is mean reciprocal rank and nDCG is normalized discounted cumulative gain. Both are defined by the arithmetic below rather than by their names, so read the block, not the acronyms.

The setup: an evaluation set of 40 questions, each with one hand-labelled gold passage, run through the pipeline. The MRR and nDCG lines then zoom in on three of those 40 questions so the numbers stay small enough to follow.

Recall@10 = (questions whose gold passage appeared anywhere in the top 10)
            / (total questions)
          = 16 / 40  =  0.40

MRR = mean of 1/(rank of the gold passage), scoring 0 when it never appeared.
      Three questions, gold at ranks 1, 3, and not in the top 10:
      (1/1 + 1/3 + 0) / 3  =  (1 + 0.3333 + 0) / 3  =  1.3333 / 3  =  0.44

nDCG@5 for the middle question alone, gold at rank 3:
      DCG  = 1 / log2(3 + 1)  =  1 / log2(4)  =  1/2  =  0.50
      IDCG = 1 / log2(1 + 1)  =  1 / log2(2)  =  1/1  =  1.00
                                          (IDCG is the best possible ordering)
      nDCG = 0.50 / 1.00                  =  0.50

Two names from that last block. DCG is discounted cumulative gain: the score a ranking earns once each relevant passage is discounted by how far down it sits. IDCG is the ideal value of that same quantity — the score the perfect ordering would have earned. Dividing one by the other is what puts every question on a 0-to-1 scale.

What each metric is actually asking

The three are not three readings of the same thing.

• Recall@k asks did we get it at all. A yes or no per question. It is the only one of the three that bounds everything downstream.
• MRR asks how far down. It collapses each question to its single best hit, which makes it right when one passage suffices and wrong when the answer needs three.
• nDCG is MRR’s general form. It discounts every relevant passage by 1/log2(rank+1) instead of only the first, then divides by the score of the perfect ordering. That normalization is the whole point: it makes a question with one relevant passage comparable to a question with nine.

Compare the middle question under two of them. Its gold passage sits at rank 3, so:

reciprocal rank  =  1/3            =  0.33
nDCG@5           =  1/log2(3+1)    =  0.50

Be careful with that 0.33. It is a single question’s reciprocal rank, not an MRR — there is no mean to take over one question. The MRR in the block above is 0.44, averaged across all three.

Comparing 0.33 against 0.50 is fair because both describe the same single ranking. The whole gap between them is the choice of discount: 1/rank against 1/log2(rank+1). The logarithm punishes rank 3 less harshly.

Diagnose in order

The order is not a preference — it is a dependency. If Recall@10 is 0.4, the right passage is absent from the context 60% of the time, and no prompt change can make the model cite something it cannot see. Fixing generation first is optimizing a downstream stage against a broken input. Diagnosing a rag agent in the right order walks the same ordering as a debugging procedure.

One caution about the retrieval/generation split. The chunk-boundary failure from Chunking passes Recall@10 — chunk 8 was in the top 3 — and passes answer relevance too. What catches it is faithfulness, because “14 days” is supported by no passage in the context that was finally assembled.

Recall measures the retriever. Faithfulness measures the whole pipeline, including the truncation you did after retrieving. A healthy Recall@10 sitting next to poor faithfulness points at assembly, not at search.

The third layer nobody builds

There is a layer neither table above has, and it is the one with no other detector — nothing in retrieval or generation quality goes red when it breaks. Four metrics, worth naming, because an unnamed metric is one nobody can ask for by name in a design review.

• Index recall audit. Nightly, run a sample of queries through the approximate index and through an exact brute-force scan of every vector, and report what fraction of the exact top-k the index found. This is the 0.96-style fidelity number from Rag vs tool vs fine tune vs long context, tracked over time rather than measured once — it degrades silently as the index grows and as its structures fragment.
• Indexing lag at p99. “p99” is the 99th percentile: sort every document by how long it took to go from written to searchable, and report the value only 1% exceed. The average is useless here, because the failure you care about is the slow tail — the documents that are stale for hours while everything else updates in seconds.
• Embedding provenance assertions. On the search path, assert that the stored vectors were produced by the embedding model you believe is in use. A model version changing underneath an existing index does not raise an error; it quietly returns worse results forever (Embedding model version skew the nastiest).
• Orphan and tombstone counts. An orphan is a vector whose source document no longer exists, so it can still be retrieved and cited after the document was deleted. A tombstone is the marker written to record a deletion; if the process that sweeps them up stops running, they accumulate and the index slowly fills with entries that answer nothing. Count both and alert on growth.

Those are system metrics rather than model metrics, and they live in Evaluation three layers and nobody builds the third.

Cheat sheet

One row per symptom you can observe from outside the system, with the mechanism that produces it and the fix.

Reading this chapter against ch 06

Chapter 06 of the GenAI system-design track opens by saying it assumes all of this one. Two places where the two look like they disagree and do not.

Cost: same saving, different denominator

Query time per 1000 queries prices a query at $0.0187 and calls generation 89% of it. That is the single-pass serving path — one retrieval, one generation — and it is the right denominator for a classic pipeline.

The 0.58× to 6.27× band derived in What the cost multiplier actually is is a multiplier on ch 06’s generation line, not a competing figure. An agent that runs three searches pays that generation cost about four times over. Ch 06’s advice therefore gets stronger here, not weaker.

Here is where two different-looking percentages come from. Cutting from five passages to three removes 1,000 input tokens, which at $3 per million saves 1,000·$3/M = $0.003 per query. Divide that by each chapter’s reference query:

$0.003 / $0.0187   =  16%     ch 06's reference query
$0.003 / $0.01665  =  18%     this chapter's workload from §1

Same saving, different denominator. If you are reasoning inside this chapter, use 18%.

Either way the saving is larger in an agent than in either single-pass figure, because every passage you did not fetch is a passage that is not re-billed on every call after the one that would have fetched it.

Caching: which cache the cheat sheet means

“Cache” in the sheet above means the embedding and retrieval caches. The retrieval cache is keyed on index_version, so that the thing which would make it wrong is the thing that invalidates it.

It does not mean caching final answers, which Caching three caches three risk profiles rates “High. Do not.” An answer cache keyed on the query alone goes stale while continuing to emit the citation that makes it look verified — the citation is what makes the staleness dangerous rather than obvious. The one safe variant is keying the answer on the retrieved chunk ids plus their content hashes.

Two things ch 06 adds

One place where ch 06 goes further than this chapter, and you should follow it rather than reconcile. Both additions are code listings there, and each is one idea.

• ScopedRetriever (Sharding and multi tenancy) is a wrapper around the retriever that refuses to run a search unless the customer-scope filter and the embedding-model check are both applied. Putting them on the search path rather than in the constructor is the whole point: a check that runs once at construction never sees a vector, so it cannot stop the one that matters.
• ingest() (The indexing pipeline is a streaming system) is the index-time function that writes a document’s chunks, and it replaces all of them in one atomic operation — one that either fully happens or does not happen at all, with no visible halfway state. Deleting the old chunks and then inserting the new ones is the obvious implementation and the wrong one, because it leaves the document absent from the index if the worker dies between the two steps.

Neither is visible in the agent-loop view, and both are the kind of thing that only shows up as an incident.

Next: 06 — Multi-Agent.