00 — LLM Internals for Agent Builders

A handful of mechanisms inside a language model actually change the decisions you make as an agent builder: tokenization, attention, the key/value cache, embeddings, sampling, and the cost arithmetic that falls out of them.

One word you need before anything else. A transformer is the neural-network architecture every current large language model is built from: a tall stack of near-identical layers, each of which mixes information between word positions. This chapter is not a full tutorial on that architecture. It covers only the parts with consequences you can act on.

Once you understand these, the cost, latency, and quality claims in later chapters follow from the mechanisms rather than needing to be memorized.

The shape of the whole thing

Before any mechanism, fix what goes in and what comes out.

A model takes a string of text and turns it into a sequence of tokens (Tokens). It runs those tokens through a stack of layers (The forward pass). It emits exactly one next token.

That one token is appended to the input, and the whole process runs again to produce the token after it. Text goes in; one token at a time comes out, until the model emits a token that means “stop.”

Everything else in this chapter is detail on that single loop: how the next token is chosen, why each pass costs what it does, and what work can be cached or skipped.

The anatomy of a request

That string of text is not something you type by hand. When you call a model API you send three things, and the server concatenates them, in this fixed order, into the one long sequence of tokens the model actually reads:

1. tools — the schemas of the functions the model is allowed to call: each one’s name, a description of when to use it, and the types of its arguments. Sent as text, like everything else.
2. system — standing instructions that hold for the whole conversation, such as “you are a support agent” or “never refund an order over $50.”
3. messages — the conversation so far: an alternating list of what the user said, what the model replied, and what any tool returned.

One user message plus the model’s reply to it is one turn. A conversation is a list of turns.

An agent is a model that calls tools in a loop until a task is done. Structurally it is just a conversation whose turns are mostly tool calls and tool results rather than human sentences.

The critical property is that the API is stateless. The server keeps nothing between calls. It does not remember your last request.

So every request resends the entire conversation from the beginning: all of tools, all of system, and all of messages up to now. Turn 12 pays for turns 1 through 11 all over again.

That single fact is what makes prompt caching worth a whole section (Prompt caching derived), and why agent cost grows quadratically with turn count rather than linearly (Deriving the numbers).

1. Tokens

The model doesn’t see characters or words. It sees tokens: subword units produced by a tokenizer, a program that chops text into pieces from a fixed vocabulary. The usual algorithm is byte-pair encoding (BPE), and the vocabulary is learned once, in advance, from a large text corpus. Common words end up as one token; rare words split into several.

BPE is one rule applied repeatedly. Start with every character as its own symbol. Find the most frequent adjacent pair of symbols in the training corpus. Merge that pair into a single new symbol. Repeat until you have the target vocabulary size (typically ~10⁵ symbols).

That rule is why the splits are predictable rather than arbitrary. Sequences that occurred often enough during training got merged into single tokens. Everything else stays fragmented into the largest pieces that did get merged.

Here are four strings and what a BPE tokenizer does to them. Look at how the frequency of the string in ordinary text predicts how badly it fragments:

"unbelievable"     -> ["un", "bel", "iev", "able"]      4 tokens
"the"              -> ["the"]                            1 token
"ERR_4021"         -> ["ERR", "_", "40", "21"]           4 tokens
"def calculate("   -> ["def", " calculate", "("]         3 tokens

"the" survived every merge round because it is everywhere. "unbelievable" never appeared often enough as a unit, so it decomposes into the four fragments that did.

ERR_4021 is a string the corpus essentially never saw, so it falls apart into generic pieces. This example returns in Embeddings and why dense search misses err_4021.

Note also that " calculate" carries its leading space. The space-plus-word pair is what actually got merged, so in most tokenizers the space belongs to the word that follows it.

Rules of thumb: ~4 characters per token for English prose, worse for code (punctuation and identifiers fragment), much worse for non-Latin scripts.

Three consequences you’ll hit:

1. Token counts are model-specific. A different tokenizer gives a different count for identical text. Never reuse a count measured on one model to budget another — call count_tokens against the model you’ll actually use.

   In particular, never use tiktoken for Claude. It is OpenAI’s tokenizer, and it has a different learned vocabulary. To see the gap, run the same text through both counters and compare: on ordinary English prose tiktoken lands roughly 15–20% low, and far lower than that on code, where the two tokenizers disagree about how identifiers and punctuation split.

   Treat that range as a direction, not a constant. It is a property of whichever two tokenizers you happen to compare, so measure it on your own text if you must estimate at all.

   The right call is a real endpoint (POST /v1/messages/count_tokens). It takes the same request body you were going to send, and it costs nothing. The response is a single number:

   client.messages.count_tokens(
       model="claude-opus-5",
       system=SYSTEM, tools=TOOLS, messages=messages,
   )
   -> MessageTokensCount(input_tokens=6042)

   Pass tools and system too — they’re part of the prompt (Prompt caching derived) and a count that omits them under-reports the fixed prefix, which is exactly the term that gets multiplied by turn count in Deriving the numbers.

2. Rare identifiers fragment. ERR_4021 becomes four low-information tokens. This is the first half of why dense embedding search cannot reliably surface exact error codes — see Embeddings and why dense search misses err_4021.

3. Cost is per token, not per character. A JSON payload with verbose keys costs real money on every turn it stays in context. {"customer_identifier": ...} vs {"cid": ...} across 40 turns adds up.

2. The forward pass

Tokenization settles what the model reads. What it does next is one loop — and every cost in this chapter is a property of some step in it. Read the diagram left to right: tokens become vectors, the vectors pass through every layer in order, the last layer produces one score per possible next token, one token is drawn from those scores — and then the dashed arrow sends you back to the start for the next token.

flowchart LR
    T["Tokens<br/>[1, 847, 22, ...]"] --> E["Embedding<br/>each token -> vector"]
    E --> L1["Layer 1<br/>attention + FFN"]
    L1 --> L2["Layer 2"]
    L2 --> LN["... Layer N"]
    LN --> LG["Logits<br/>one score per<br/>vocab entry"]
    LG --> S["Sample<br/>one token"]
    S --> A["Append,<br/>run again"]
    A -.-> T

    style LG fill:#40916c,color:#fff
    style S fill:#bc6c25,color:#fff

Four labels in that diagram matter later, so here is what each one means.

Embedding — each token → vector. A lookup table maps token ID 847 to a vector of a few thousand floats. This is one vector per token, and it is not the sentence embedding of Embeddings and why dense search misses err_4021 — that one is a single vector for a whole passage, produced by a different model. Same word, two different objects; Embeddings and why dense search misses err_4021 explains what collapses the former into the latter.

Layer — attention + FFN. Each layer does two things, and this chapter only dwells on one of them.

Attention moves information between token positions (Attention and why context costs what it does). The feed-forward network (FFN) — two matrices applied to each position independently — transforms each token’s vector in place, with no communication between positions at all.

The FFN is where the model’s bulk lives. It holds most of the parameters: the numbers learned during training and then frozen, identical for every request you send. “A 70-billion-parameter model” is counting exactly these.

It also burns most of the FLOPs — floating-point operations, the individual multiplications and additions that GPU work is measured in — spent on prefill, the one pass that reads your entire input before any output token exists. (The kv cache the most important mechanism in this chapter takes prefill apart properly.)

What matters here is the shape of the two costs, not their size. Double the sequence length and the FFN’s cost doubles — it is linear. Double it and the per-layer attention cost roughly quadruples — it is quadratic.

That is why the quadratic term of Attention and why context costs what it does doesn’t dominate at short contexts and does dominate at long ones. You are watching a linear term and a quadratic term trade places.

Logits — one score per vocab entry. The last layer projects the final token’s vector onto the vocabulary, producing one raw real number per possible next token. A logit is one of those raw scores, before any conversion into a probability.

Vocabularies are on the order of 10⁵ entries, so a logit vector is ~100,000 floats. Sampling and why temperature0 isnt deterministic turns that vector into a probability distribution; Structured output is a guarantee not a request censors it entry by entry.

Sample one token, append, run again. One token is drawn from those scores, glued onto the input, and the whole thing runs again. One pass produces one token. A 500-token response is 500 passes. That loop — and the fact that each pass must wait for the previous one — is where nearly all generation latency lives.

3. Attention, and why context costs what it does

Attention is how a layer moves information between token positions — how a verb finds out who its subject is — and it is also where context gets its price. Two structural properties of the mechanism generate the caching rules and the long-context economics of everything downstream, and the fastest way to own it is to run one head by hand.

The three vectors

Inside each layer, every token computes three vectors from its own current vector:

• a Query — what am I looking for?
• a Key — what do I offer to anyone looking?
• a Value — what content do I hand over if someone picks me?

Attention for token i is then a weighted sum of the Values of every token it can see, weighted by how well i’s Query matches each Key. Written compactly, with Q, K and V being the matrices that stack every token’s Query, Key and Value:

attention(Q, K, V) = softmax( Q · Kᵀ / √d ) · V

Two symbols in there need naming before the worked example uses them.

Kᵀ is the transpose of the Key matrix — the same Keys with rows and columns swapped. That is the arrangement that makes Q · Kᵀ produce one score for every (query, key) pair rather than a single number.

d is explained in the next subsection, and √d is why Step 2 below divides. Both are easier to justify after you have seen the arithmetic once.

Heads: the same computation, run many times side by side

A layer does not run that computation once. It runs it h times in parallel, and each independent copy is called a head.

Every head gets its own learned recipe for turning a token’s vector into a Query, a Key and a Value, so the heads specialize: one may track grammatical subjects, another may track which earlier name a pronoun refers to. Their outputs are concatenated and passed on together.

Two consequences follow immediately.

First, d in the formula above is the head dimension — the length of one head’s Query or Key vector, typically 64–128 numbers. It is not the full width of the model.

Second, every head computes its own Keys and Values. So anything you store to avoid recomputing them is stored per head, per layer. That is why the cache of The kv cache the most important mechanism in this chapter is measured in gigabytes rather than megabytes.

Everything below follows a single head. One head is the whole mechanism; the other h − 1 are the same arithmetic with different learned numbers.

A worked example, one head from start to finish

Take the three-word sentence “The cat sat”, which tokenizes to three tokens: The (position 0), cat (position 1), sat (position 2).

Follow the last one, sat. To represent a verb properly the model needs to know who did the sitting, so sat has to pull information from an earlier word. It can only look backwards — attention is causal, shown later in this section — so its candidates are The, cat, and itself. The steps below show how it identifies cat as the word it needs.

Every token carries three vectors. Here they are d = 4 numbers long, with a meaning attached to each of the four slots:

slot 0 = "is a noun"      slot 2 = "is a living thing"
slot 1 = "is a verb"      slot 3 = "is past tense"

Those four labels are a teaching fiction.

A real model has no labelled slots. Nobody decided that slot 0 would mean “is a noun,” and if you inspected a trained model you would find no slot that means anything you could name. The dimensions are whatever coordinate system training happened to settle on (Embeddings and why dense search misses err_4021 returns to this).

The mechanism is exactly what a real head does; the labels are ours, attached so the arithmetic below is followable instead of being a page of anonymous decimals.

A Query is what a token is looking for. A Key is what a token advertises about itself. So:

Query of `sat`   q  = [1, 0, 1, 0]   "I want a NOUN that is a LIVING THING"

Key of `The`     k₀ = [1, 0, 0, 0]   "I am noun-ish"       (a determiner, weakly)
Key of `cat`     k₁ = [1, 0, 1, 0]   "I am a noun AND a living thing"
Key of `sat`     k₂ = [0, 1, 0, 1]   "I am a verb, past tense"

Read those two blocks together and the answer is already visible: sat is asking for a living noun, and cat is the only word advertising both. The next four steps are how the model computes that, arithmetically, without anyone telling it.

Step 1 — score each Key against the Query with a dot product. A dot product multiplies the two vectors slot by slot and adds the results, so it is large when both vectors have the same slots switched on. That makes it a match score between what sat wants and what each word offers:

`The`   q · k₀ = (1)(1) + (0)(0) + (1)(0) + (0)(0) = 1     one slot agrees ("noun")
`cat`   q · k₁ = (1)(1) + (0)(0) + (1)(1) + (0)(0) = 2     both slots agree
`sat`   q · k₂ = (1)(0) + (0)(1) + (1)(0) + (0)(1) = 0     nothing agrees

cat scores highest because it is the only word that switched on both slots sat was asking about. sat scores zero against itself, which is correct — a verb has no subject information to offer.

Step 2 — shrink the scores by dividing by √d. With d = 4, √d = 2. This step has nothing to do with the meaning; it stops the numbers growing just because the vectors are long, and the paragraph after the example explains why that matters:

1 / 2 = 0.5        2 / 2 = 1.0        0 / 2 = 0.0

Step 3 — softmax turns those three scores into percentages. Softmax is a function that takes any list of numbers and returns that many positive numbers, which sum to 1 and keep the original order — a list of scores in, a list of percentages out. It does this by raising e ≈ 2.718 to the power of each number and then dividing each result by the total, which is the whole definition:

softmax(x)ᵢ = exp(xᵢ) / Σⱼ exp(xⱼ)

Here it converts “how well each word matched” into “what fraction of its attention sat spends on each word.” Substituting the three scaled scores from Step 2:

exp(0.5) = 1.649        exp(1.0) = 2.718        exp(0.0) = 1.000
sum      = 1.649 + 2.718 + 1.000 = 5.367

1.649 / 5.367 = 0.307      2.718 / 5.367 = 0.506      1.000 / 5.367 = 0.186

softmax(0.5, 1.0, 0.0) = 0.307, 0.506, 0.186        (sums to 1)

sat spends 51% of its attention on cat, 31% on The, and 19% on itself. Notice it is not winner-take-all: the model hedges, which is what lets it recover if cat turns out to be the wrong subject.

Exponentiating before dividing has one consequence worth holding on to: softmax turns differences between scores into ratios between weights.

The gap of 0.5 between cat and The becomes a weight ratio of exp(0.5) = 1.649, and you can read that straight off the numbers above: 0.506 / 0.307 = 1.65. It would be that same 1.649 whether the two scores were 0.5 and 1.0 or 100.5 and 101.0. Only the differences survive; the absolute size never does.

That is why widening the gaps sharpens the distribution — which is precisely the knob Sampling and why temperature0 isnt deterministic calls temperature.

Note also that a score of 0, which sounds like “no match at all,” still collects 19% of the attention. Softmax never returns a true zero, because exp of any finite number is positive. The only input that produces an exact 0 is −∞, and that is exactly the trick used for masking in Structured output is a guarantee not a request.

Step 4 — collect the Values. The Key was only the advertisement. The Value is the actual content a word hands over to whoever attends to it. Keep the slots simple — let each word’s Value carry its own identity, so we can see whose information ends up where:

v₀ from `The` = [2, 0, 0, 0]      slot 0 carries "The"-ness
v₁ from `cat` = [0, 2, 0, 0]      slot 1 carries "cat"-ness
v₂ from `sat` = [0, 0, 2, 0]      slot 2 carries "sat"-ness

The output is each Value multiplied by its Step-3 percentage, all added together — one slot at a time:

out = 0.307·v₀ + 0.506·v₁ + 0.186·v₂

slot 0:  0.307 × 2  +  0.506 × 0  +  0.186 × 0   =  0.614
slot 1:  0.307 × 0  +  0.506 × 2  +  0.186 × 0   =  1.012
slot 2:  0.307 × 0  +  0.506 × 0  +  0.186 × 2   =  0.372
slot 3:  0           +  0          +  0           =  0

out = [0.614, 1.012, 0.372, 0]

Read that result back through the slots. The largest number, 1.012, sits in the slot carrying “cat”-ness. That is the whole point: the vector that leaves this step for the word sat is now mostly information about cat, with smaller traces of The and of itself. The model started with three unrelated words and ended with sat carrying the identity of its subject.

That is one attention head, end to end: ask, score, convert to percentages, blend. Every layer does this for every token, in parallel, dozens of times over.

Why Step 2 divides by √d

The division is not cosmetic. Here is what it is protecting against.

A dot product adds up d products, so its typical size grows with d — specifically, it grows like √d. Widen the head from 4 slots to 128 and the raw scores get bigger, for no meaningful reason. Nothing about the matching improved; there are just more terms in the sum.

Watch what bigger scores do to softmax. Skip the division and the same three scores 1, 2, 0 softmax to 0.245 / 0.665 / 0.090, already sharper than the scaled 0.307 / 0.506 / 0.186.

And note how fast that sharpening compounds: doubling a gap squares the weight ratio, because exp(2g) = exp(g)². The scaled gap of 0.5 gave a ratio of 1.65; the unscaled gap of 1.0 gives 1.65² = 2.72.

At a realistic d = 128 the un-scaled scores are large enough that softmax saturates: every weight collapses toward 1 or 0, and the head stops distinguishing between candidates at all. A second-place word that was nearly as good as the winner receives effectively nothing instead of the 31% The got above.

(Only toward. As Step 3 insisted, exp of a finite number is never zero, so the runner-up’s weight becomes negligible rather than absent.)

A head whose distribution has saturated that far is no longer blending information. It is picking one token and discarding every other, which throws away exactly the hedging that makes the mechanism useful. Dividing by √d cancels the width term and holds the scores in the range where softmax stays informative regardless of head size.

Two properties that drive the rest of the chapter

Two properties of attention matter for everything that follows:

It’s causal. Token i attends to tokens 0..i only, never forward. This single fact generates the prompt-caching rules in Prompt caching derived.

It’s quadratic. With n tokens, the Q · Kᵀ matrix is n × n. Doubling context roughly quadruples attention compute.

The two properties are the same picture seen twice. Draw the n × n score matrix with queries down the rows and keys across the columns, and mask out everything above the diagonal. Each row is one token’s Step 1 — the scores it computes against every Key it is allowed to see. Notice that the rows get longer as you go down, and that the whole upper-right half is empty:

          k0   k1   k2   k3   k4
    q0     ■    ·    ·    ·    ·        row 0: 1 score
    q1     ■    ■    ·    ·    ·        row 1: 2 scores
    q2     ■    ■    ■    ·    ·        row 2: 3 scores   <- the worked example above
    q3     ■    ■    ■    ■    ·        row 3: 4 scores
    q4     ■    ■    ■    ■    ■        row 4: 5 scores

The · cells are the future, set to −∞ before softmax so they contribute nothing. That is causality, and Prompt caching derived is entirely a consequence of it.

The ■ cells number 1 + 2 + 3 + 4 + 5 = 15, which is n(n+1)/2 for n = 5, and n(n+1)/2 ≈ n²/2 for large n. That is the quadratic. Doubling n roughly quadruples the filled area.

Note also what the rows say about cost asymmetry: row 4 reads five Keys, row 0 reads one. Appending a token adds a whole new row, and that row is the longest one yet — which the decode analysis in The kv cache the most important mechanism in this chapter builds on.

Context window: linear bill, quadratic work

“Context window” is the length of that matrix’s side — the hard cap on how many tokens can appear in one request, prompt and generated output together.

It is a limit, not a budget. The provider does not charge a premium for operating near it: a token at position 199,000 is billed at exactly the same rate as the same token at position 20,000.

Two different quantities are in play here, and conflating them is the standard confusion.

What you are billed is a count of tokens, and it is strictly linear. 200k tokens cost twice what 100k tokens cost, every time.

What the provider computes on the attention term is the filled triangle above, and that is quadratic. 200k tokens is roughly four times the attention work of 100k, and roughly 100× the work of 20k — because (200/20)² = 100.

The price sheet flattens that curve into a straight line, so the quadratic never shows up on the invoice. It shows up as latency, and as the quality effects of Why quality degrades in long contexts.

The practical consequence: cost rises linearly with context while the work behind it rises quadratically, so time to first token and answer quality both degrade faster than the bill does. This is why loading everything into context stops being viable well before you reach the documented limit.

4. The KV cache — the most important mechanism in this chapter

To generate token n+1, the model needs the Keys and Values of every token 0..n. Those are the same Keys and Values it computed on the previous step, and the step before that. The KV (key/value) cache stores them instead of recomputing them — one decision that drops generation from O(n³) to O(n²) and splits inference into two phases, prefill and decode, with completely different costs.

Where the cube comes from. It is the triangle of Attention and why context costs what it does summed over time. Recomputing everything from scratch on every step:

step t re-runs attention over t tokens         -> O(t²)   (the whole mask, rebuilt)
sum over t = 1..n:  1² + 2² + ... + n²         -> O(n³)   (that sum is n(n+1)(2n+1)/6)
with the cache, step t only computes row t     -> O(t), and 1 + 2 + ... + n is O(n²)

The middle line is the one to read twice. Summing squares up to n gives a cubic, because you are redoing the entire triangle n separate times. Summing plain integers up to n gives a quadratic — that is the whole saving.

The cache doesn’t make attention cheaper. It deletes the redundant recomputation of rows the model already built. That single exponent, 3 down to 2, is why every serving stack has one.

The diagram below splits inference into the two phases that fall out of this. Read the top box as “everything you sent, done once, in parallel” and the bottom loop as “one output token at a time, each one reading the whole cache”:

flowchart TD
    subgraph P["PREFILL — the whole prompt at once"]
        P1["All input tokens processed in parallel<br/>one large matmul<br/>K,V computed and cached for every token"]
    end
    subgraph D["DECODE — one token at a time"]
        D1["Compute Q for the new token only"]
        D1 --> D2["Read K,V for ALL previous tokens<br/>from cache"]
        D2 --> D3["Emit one token, append its K,V"]
        D3 --> D1
    end
    P --> D

    style P fill:#2d6a4f,color:#fff
    style D fill:#bc6c25,color:#fff

Read the diagram as two different machines.

Prefill hands the GPU every input token at once, so the whole prompt becomes one large matrix multiply. That is thousands of tokens’ worth of work at high arithmetic intensity — a lot of computation per byte fetched from memory. Every token’s K,V lands in the cache on the way through.

Decode computes a Query for the new token only. It must read K,V for all previous tokens out of the cache in order to attend over them, then emits one token, appends that token’s own K,V, and loops. One token’s worth of arithmetic against an entire cache’s worth of memory traffic.

The two phases therefore have completely different performance characteristics. The row that matters most is “Bottleneck” — the next subsection puts a number on it:

Putting a number on “memory bandwidth”

That table cell is the most load-bearing claim in the chapter, so derive it. The cache stores a K and a V vector per token, per head, per layer:

KV bytes = 2 (K and V) × layers × kv_heads × d_head × bytes_per_value × n_tokens

Now substitute a plausible frontier configuration. Each of the five factors, with what it means:

• 80 layers — the model’s depth. Every layer keeps its own cache.
• 8 key/value heads per layer — fewer than the number of Query heads, because of grouped-query attention: an optimization where several Query heads share one set of Keys and Values, done precisely to keep this cache small.
• d_head = 128 — the head dimension from Attention and why context costs what it does.
• fp16 — 16-bit floating point, which is 2 bytes per number.
• × 2 at the front, because you store a K and a V.

per token   = 2 × 80 × 8 × 128 × 2         = 327,680 bytes  = 320 KiB
at 100k ctx = 327,680 × 100,000            = 32.8 GB

Note the first line: one token of context costs 320 KiB of GPU memory, and every decode step has to read all of it back.

Now divide by memory bandwidth — how many bytes per second the chip can pull out of its own memory. An NVIDIA H100 SXM, the data-centre GPU most frontier serving ran on when these figures were current, moves roughly 3.35 TB/s from its high-bandwidth memory (HBM), the stack of memory sitting on the GPU package itself. (Newer parts are faster — an H200 is about 4.8 TB/s — which moves the ceiling below but changes none of the reasoning.)

32.8e9 bytes / 3.35e12 bytes/s = 9.8 ms per decode step
1 second / 9.8 ms per token   -> ~102 tokens/s ceiling, from the cache read alone

Run the same arithmetic at 10k of context and the cache is 3.3 GB, the step takes 0.98 ms, and the ceiling is ~1,020 tok/s. Ten times the context, one tenth the ceiling.

And that is an optimistic ceiling: the model weights must also be streamed on every step, on top of this.

Meanwhile the actual arithmetic in that step is a handful of matrix–vector products — trivial work. The GPU spends almost all of its time waiting on memory rather than computing. That is what memory-bound means, and it is the mechanism behind the third bullet below.

Why output tokens cost more than input tokens

Prefill is an efficient parallel matrix multiply running at thousands of tokens per second. Decode is a sequential, memory-bound crawl that re-reads a growing cache on every single token, and the derivation above puts it at ~102 tok/s at 100k of context.

Be careful about how far you push that. The throughput gap is tens of times. The price gap is only ~5× (chapter 09: $5/$25, $3/$15, $1/$5 per million tokens for input/output).

The two are not the same number, and nothing here derives one from the other. Price also absorbs batching economics: a provider can run many users’ decode steps together in one pass, sharing the cost of streaming the model weights, which recovers much of the gap.

What the hardware fixes is the direction, not the magnitude. Output is intrinsically the expensive side, on every model, on every vendor.

The tell is that the ratio is exactly 5× on all three tiers just quoted, rather than drifting from tier to tier. That is what you expect from one structural fact applied uniformly, not from three independent pricing decisions.

Three practical consequences

1. Shortening output is a bigger win than shortening input. This is the whole argument for edit over write in a coding agent (case study 03).

   Source code runs about 10 tokens per line. That estimate comes from the rules of thumb in Tokens: a typical line of Python or TypeScript is 30–40 characters including its indentation, and code tokenizes at roughly 3–4 characters per token, which puts a line at about 10 tokens either way you take the range. Tokenize your own repository if you want a tighter number; the argument survives a factor of two.

   At that rate, rewriting an 800-line file is 800 × 10 = ~8,000 output tokens. A diff that changes 3 lines, with its context lines and headers, is ~200. That is a 40× difference, paid at decode speed — seconds of wall clock, not milliseconds.

2. Streaming exists because decode is slow. You can’t make the tokens arrive faster; you can start showing them immediately.

3. A long input is cheap to process but expensive to generate against. Prefill reads it once in parallel; every decode step afterwards re-reads the whole cache it produced.

5. Prompt caching, derived

The The kv cache the most important mechanism in this chapter cache has a second life: it can outlive the request that built it. Prompt caching is storing the KV cache for a prefix so prefill can be skipped.

Everything about how it behaves follows from causal attention:

Token i’s K and V depend on tokens 0..i — and nothing after.

Read that again, because four rules fall straight out of it:

Why it’s a prefix match. If tokens 0..j are identical to a previous request, their K,V are bit-for-bit identical too. They can be reused without recomputation. The moment token j+1 differs, every token from j+1 onward has a different context, so different K,V. Nothing after the divergence is reusable.

The diagram below shows one request split into four segments. The three ✓ segments were in the previous request too; the ✗ segment is new. Follow the dashed arrows to see which price each segment pays:

flowchart LR
    subgraph HIT["Request 2 — shares a prefix"]
        A1["tools ✓"] --> A2["system ✓"] --> A3["history ✓"] --> A4["new turn ✗"]
    end
    A1 -.->|K,V reused| R1[cache read · 0.1x]
    A2 -.->|K,V reused| R1
    A3 -.->|K,V reused| R1
    A4 -.->|must prefill| R2[full price]

    style R1 fill:#2d6a4f,color:#fff
    style R2 fill:#bc6c25,color:#fff

The three ✓ segments bill as cache reads at 0.1× because their K,V are byte-identical to last time’s. The ✗ segment must be prefilled at full price — and nothing downstream of it can be reused, no matter how much of that text repeats.

Why one byte ruins everything downstream. A timestamp in the system prompt changes a token near position 30. Tokens 30..∞ now have different preceding context, hence different K,V, hence nothing after position 30 can be reused — even though 99% of the text is unchanged. That’s why datetime.now() in a system prompt is catastrophic rather than merely wasteful.

Why render order is tools → system → messages. The prefix is whatever comes first. Stable content must be positionally first to be cacheable, so the API puts the most-stable category first by construction. Your job is to keep volatile content after the last breakpoint.

What this looks like on the wire

A breakpoint is a marker you place on a content block saying “cache everything from position 0 through the end of this block.” You get at most four per request, and only their positions matter — the API caches the prefix, not the block.

Here is turn 1 of an agent loop. Look at the usage block the API returns: the three fields it reports are how you tell caching worked.

tools:    [ ...schemas... ]
system:   [ {"type": "text", "text": SYSTEM_PROMPT,
             "cache_control": {"type": "ephemeral"}} ]   <- breakpoint: tools+system
messages: [ {"role": "user", "content": "Reset my password"} ]

usage -> {"cache_creation_input_tokens": 6042,   # written, billed at 1.25x
          "cache_read_input_tokens": 0,          # nothing to reuse yet
          "input_tokens": 11}                    # the user turn, full price

Turn 2 appends the assistant reply and a tool result. Nothing before the breakpoint moved, so:

usage -> {"cache_creation_input_tokens": 0,
          "cache_read_input_tokens": 6042,       # the whole prefix, at 0.1x
          "input_tokens": 380}                   # only what's new

cache_read_input_tokens is the field to check: if it stays 0 across requests you know should share a prefix, something in that prefix is changing. cache_creation_input_tokens: 0 on the first request means the opposite problem — the prefix was too short to be worth caching (see below).

Why reads cost ~10% and writes cost 1.25×. A read skips prefill FLOPs entirely but still pays to move the cached K,V into GPU memory — not free, just much cheaper. A write pays normal prefill plus the cost of persisting the cache.

Those multipliers are relative to the normal input price, so 1.0 means “what this prefix would have cost uncached.” Send the same prefix twice and caching bills 1.25 + 0.1 = 1.35 against 1.0 + 1.0 = 2.0 uncached. Two requests is already a win.

Why there are two TTLs (time-to-live windows). A TTL is how long a cache entry survives before the server drops it, and you choose between two. The default is 5 minutes at a 1.25× write; the alternative is 1 hour at a 2× write. Longer retention costs more because the entry occupies serving memory the whole time.

The write multiplier moves the break-even point. Set “cost with caching over N requests” equal to “cost without” and solve for N — one write plus N−1 reads on the left, N full-price prefills on the right:

5-min:   1.25 + 0.10(N-1) = N  ->  N = 1.28  -> the 2nd request already wins
1-hour:  2.00 + 0.10(N-1) = N  ->  N = 2.11  -> N=2 loses (2.10 vs 2.00); N=3 wins

So the long TTL is not a free upgrade — it costs you one extra request before it pays. Take it only when you know the prefix gets re-hit across a gap longer than five minutes: a nightly batch over a shared corpus, not a live chat where turns arrive seconds apart.

Why caches are model-scoped. Different weights produce different K,V for identical tokens. There is no way to share a cache across models — which is why routing to a cheaper model mid-conversation costs you the whole prefix and often loses money overall.

Why there’s a minimum cacheable length. Below a few hundred tokens the bookkeeping outweighs the saved prefill, so short prefixes silently don’t cache — no error, just cache_creation_input_tokens: 0. The threshold is model-specific and not even monotonic across generations (512 tokens on claude-opus-5, 1,024 on claude-sonnet-5, 4,096 on claude-haiku-4-5; see chapter 09), which is one more thing a mid-conversation model switch can quietly break.

Interview move: when asked about caching, don’t recite “it’s a prefix match.” Say “caching stores the KV cache, and because attention is causal, a token’s K/V depend only on itself and everything before it — so a shared prefix is reusable and any change invalidates everything after it, and only after it.” Same rule, derived rather than memorized, and every follow-up becomes answerable.

6. Why quality degrades in long contexts

Caching fixes what a long context costs. It does nothing about what a long context does to quality.

Two distinct effects get lumped together as “context rot,” and separating them is the whole point. The first is attention dilution; the second is positional bias.

Attention dilution

The popular version of this effect says that with n tokens competing, each one gets about 1/n of the attention, so long contexts drown every individual fact.

That story is wrong — but so is the flat denial that says a sharp head is simply sharp at any length. The arithmetic of Attention and why context costs what it does settles it exactly, and the true answer is more useful than either.

Softmax weights always sum to 1, but nothing forces them to be spread evenly. What a head actually spends on one token depends on the gap between that token’s score and everyone else’s.

Set up the simplest version of that. Give the winner a score g while n competitors all score 0. Its softmax weight is then exp(g) over the total, and the total is exp(g) plus n copies of exp(0) = 1:

weight of winner  =  exp(g) / (exp(g) + n)

Now ask what gap the head needs in order to hold nine tenths of its attention on that one token. Set the weight to 0.9 and solve for g:

exp(g) / (exp(g) + n)  =  0.9
exp(g)                 =  0.9·exp(g) + 0.9n
0.1·exp(g)             =  0.9n
exp(g)                 =  9n
g                      =  ln(9n)

Substitute two window sizes into g = ln(9n):

n =      4 competitors  ->  ln(36)      =   3.58
n = 50,000 competitors  ->  ln(450,000) =  13.02

Dilution is logarithmic in n, not linear. That is the whole correction, and it is why the popular story is wrong: multiplying the competition by 12,500 raises the required gap by a factor of only 3.6, not by 12,500. Attention does not get spread thin in proportion to the window.

But logarithmic is not free, which is where the flat denial fails.

Hold the gap fixed at the 3.58 that bought 0.9 against four competitors — a head’s learned separation doesn’t grow just because your prompt did — and watch what that same head is left holding as the window grows. Each row is exp(3.58) / (exp(3.58) + n), so the first row is 35.9 / (35.9 + 4) = 0.90:

n =      4   ->  weight 0.90
n =     50   ->  weight 0.42
n =  5,000   ->  weight 0.0071
n = 50,000   ->  weight 0.0007

So sharp heads mostly survive long contexts, and mostly is the operative word.

A head needs a gap of about 13 to command a 50,000-token window the way a gap of 3.6 commands a five-token one, and nothing guarantees training handed it one. The demand grows slowly — but it does grow, and a head that falls short doesn’t fail loudly. It just stops dominating.

What that shortfall actually costs you is reliability, not signal.

For the right token to win, its score has to come out on top of a pool of candidates that grows with n. Every extra token in the window is another chance that some irrelevant passage happens to score higher than the one you need.

One head getting that comparison right is likely. Every head in every layer getting it right, on every decode step, becomes steadily less likely as the pool grows.

So the signal does not vanish, and a head with enough separation still finds its target. What you lose is the consistency with which the correct target is the one that gets focused on. And a mechanism that works 99% of the time per layer is not a 99% mechanism overall — compound it across eighty layers and 0.99^80 = 0.45, so it fails more often than it succeeds.

Positional bias from training

The second effect is about where in the window a fact sits, not how many facts are competing.

Retrieval accuracy across position is empirically U-shaped: strong at the beginning and the end, weakest in the middle. This is the “lost in the middle” result, from Lost in the Middle: How Language Models Use Long Contexts, Liu et al., 2023, and it has been reproduced across model families since.

It is a learned artifact of training data, where important content clusters at document starts and ends.

The sketch below plots recall (vertical) against where in the document the fact was planted (horizontal). Look at the shape only — two high shoulders and a flat trough between them:

 recall            SCHEMATIC — shape only, not plotted data
  100% |  ●                                             ●
       |     ●                                       ●
   80% |        ●                                 ●
       |            ●                         ●
   60% |                ●   ●   ●   ●   ●
       +----------------------------------------------------
        start        25%       50%       75%        end
        high recall        the trough           high recall

That sketch is a shape, not a measurement. The axis values are drawn to make the U legible; no run produced these exact points, and the real curve’s depth and symmetry vary by model and document length. What is reproducible is the shape — high at both ends, lowest in the middle — and the magnitude band quoted below.

Magnitudes matter here, and “degraded” is not a rounding error.

The measurement behind the curve is a needle-in-a-haystack test. Plant one sentence carrying a fact at a known position in a long document. Ask a question that only that sentence answers. Score whether the model finds it. Repeat with the sentence at every depth from 0% to 100%.

Across the published runs of that experiment (Liu et al. above, plus the vendor and community reproductions that followed), recall at the start and end sits near the model’s ceiling, while the middle trough gives up on the order of 20–40 accuracy points.

The exact figure depends on the model, the document length and how distinctive the planted fact is, so treat 20–40 as the observed band rather than a constant. Even at the bottom of that band, “attention is worse in the middle” understates it. It is the difference between a fact being found and a fact being invisible.

Design consequences, all derivable from that curve:

• Restate the task near the end of the prompt, not only at the top. The end is a high-recall position.
• A 40-turn agent buries its instructions in the middle — the worst position. This is why agents drift late in long sessions, and why compaction — periodically replacing a long stretch of old turns with a short summary of them — improves quality rather than merely saving money.
• Offloading beats stuffing. A pointer to a file at the end of the window outperforms 60k tokens of file content in the middle.

One session, with positions

The claim “a 40-turn agent buries its instructions in the middle” is worth instantiating, because it is not the system prompt that gets buried.

Take a support agent with 800 tokens of system prompt plus tool schemas, adding ~3,000 tokens per turn.

(These are not Deriving the numbers’s constants — the cost section runs a heavier reference agent at 6,000 tokens of prefix and 1,200 per turn — so don’t compare the two sets line by line. The shape of the conclusion does not depend on either choice.)

Laying the conversation out as one long array of token positions:

pos       0 –     800   system prompt + tool schemas        (start: high recall)
pos     800 –   3,800   turn 1: user's request
        ...
pos  33,800 –  36,800   turn 12: "only refund orders under $50"  <- the constraint
        ...
pos 117,800 – 120,800   turn 40: current user message         (end: high recall)

(Turn t starts at 800 + (t−1) × 3,000, so turn 12 opens at 33,800 and closes at 36,800 — where turn 13 begins.)

That constraint arrived mid-conversation as ordinary user text. By turn 40 it sits at 33,800 / 120,800 ≈ 28% depth — squarely in the trough — while the system prompt is still safely at position 0 and the latest message is safely at the end. This is why late-session drift looks so specific: the agent still remembers who it is and what was just said, and forgets the thing it was told in the middle. It is not “losing the plot” generically; it is losing a particular band of positions.

Compaction fixes this by construction, not by shrinking the bill. Replace turns 1–35 with an 800-token summary that restates the surviving constraints, and the array becomes:

pos      0 –    800   system prompt + tool schemas
pos    800 –  1,600   summary — including "refunds capped at $50"
pos  1,600 – 16,600   turns 36–40 verbatim

The constraint now sits around position 1,200 of 16,600 — it moved from 28% depth in a 120k window to ~7% depth in a 17k one. That is the actual mechanism by which compaction improves quality — it relocates facts out of the trough — and the cost saving is a side effect.

7. Embeddings, and why dense search misses ERR_4021

Why quality degrades in long contexts ended with “offloading beats stuffing”: keep less in the window, retrieve what you need. That advice is only as good as the retrieval — and retrieval is built on embeddings.

An embedding model is a separate, much smaller model that maps a piece of text to a fixed-length vector of floats. It is trained so that texts meaning similar things land close together, and texts meaning different things land far apart. Retrieval systems embed every passage in a corpus once, embed the incoming query, and return the passages whose vectors are closest.

Three words to pin down first, because chapter 05 leans on all of them.

• A dimension is one of the vector’s slots — one learned axis of variation, not a human-readable feature. Nobody assigned slot 412 the meaning “about databases”; the training objective landed on whatever coordinate system best separates the corpus. A 1,024-dimensional embedding is 1,024 such axes.
• Pooling is the step that gets you from The forward pass’s per-token vectors to one vector for a passage. The encoder produces one vector per token; pooling collapses them into a single fixed-length vector, typically by averaging. A 200-token passage becomes 1,024 floats — the same 1,024 floats no matter how long the passage was. Averaging is destructive by design, and it is precisely why one rare token cannot survive: it is one term in a mean of hundreds.
• Similarity is measured by angle, not distance, because training normalizes for magnitude. A vector’s length tracks incidental things (passage length, token frequency); its direction is what the objective shapes. Two passages saying the same thing at different lengths point the same way with different norms — so cosine, which divides both norms out, is the measure that tracks meaning.

Cosine similarity is the dot product of two vectors divided by both of their lengths. ‖a‖ means the length of a, computed as the square root of the sum of its squared slots. Dividing by both lengths is what removes magnitude and leaves only direction. The result runs from −1 to 1, where 1 means “pointing the same way”:

similarity(a, b) = (a · b) / (‖a‖ ‖b‖)

Four cosine similarities, worked by hand

Real embeddings have 1,024 dimensions or more. Four is small enough that every multiplication below fits on one line and you can check the whole thing with a calculator. The vectors themselves are invented for this example — no embedding model produced them — but the arithmetic is the arithmetic every vector database runs, and the effect they are chosen to show is real.

You will compare the query against four passages. Watch only one thing across the four blocks: the final cos line.

The user’s query is "why am I getting ERR_4021", embedded as q = [0.60, 0.50, 0.30, 0.10]. The first candidate, dA, is a passage about error handling in general that never mentions the code:

dA      = [0.62, 0.48, 0.28, 0.12]

q · dA  = 0.372 + 0.240 + 0.084 + 0.012      = 0.708
‖q‖²    = 0.36 + 0.25 + 0.09 + 0.01          = 0.71     ->  ‖q‖  = 0.8426
‖dA‖²   = 0.3844 + 0.2304 + 0.0784 + 0.0144  = 0.7076   ->  ‖dA‖ = 0.8412
cos     = 0.708 / (0.8426 × 0.8412)          = 0.9989

The second candidate, dB, is the passage you actually want: the one page in the corpus that documents ERR_4021 by name. Because the rare literal contributes four low-information tokens to a mean over hundreds, it barely nudges the vector away from “a passage about errors”:

dB      = [0.58, 0.52, 0.31, 0.09]

q · dB  = 0.348 + 0.260 + 0.093 + 0.009      = 0.710
‖dB‖²   = 0.3364 + 0.2704 + 0.0961 + 0.0081  = 0.7110   ->  ‖dB‖ = 0.8432
cos     = 0.710 / (0.8426 × 0.8432)          = 0.9993

The right passage wins — by 0.0004. Read that carefully, because the naive conclusion (“dense search ranks the wrong passage first”) is not what happened and is not the problem. Dense retrieval put the correct document on top. The problem is the size of the margin it did so by.

To see why 0.0004 is not a margin at all, keep the same passage and reword one sentence of it — same facts, same error code, a human editor’s ordinary rewrite. Pooling averages over different words now, so the vector shifts slightly, to dB′ = [0.56, 0.54, 0.31, 0.07]:

q · dB′ = 0.336 + 0.270 + 0.093 + 0.007      = 0.706
‖dB′‖²  = 0.3136 + 0.2916 + 0.0961 + 0.0049  = 0.7062   ->  ‖dB′‖ = 0.8404
cos     = 0.706 / (0.8426 × 0.8404)          = 0.9970

A cosmetic rewrite moved the score by 0.0023 — 5.3 times the 0.0004 margin the correct passage won by — and that is enough to drop it below dA, which still scores 0.9989.

(The 5.3 comes from the unrounded numbers: the true margin is 0.000424 and the true shift is 0.002259, so 0.002259 / 0.000424 = 5.33. Dividing the four-decimal values printed above gives 5.75, which is the same conclusion with rounding error in it.)

The identifier is still in the text. Nothing about the passage’s relevance changed. The ordering flipped anyway, because the ordering was never being decided by the identifier. It was being decided by wording noise several times larger than the identifier’s entire contribution.

For scale, take a genuinely unrelated passage — a billing FAQ, say, dU = [0.05, 0.10, 0.45, 0.15]:

q · dU  = 0.030 + 0.050 + 0.135 + 0.015      = 0.230
‖dU‖²   = 0.0025 + 0.0100 + 0.2025 + 0.0225  = 0.2375   ->  ‖dU‖ = 0.4873
cos     = 0.230 / (0.8426 × 0.4873)          = 0.5601

That contrast is the whole finding. Put the four scores side by side: dA 0.9989, dB 0.9993, dB′ 0.9970, dU 0.5601.

Dense embeddings separate relevant from irrelevant by an enormous distance — 0.99-something against 0.5601, a gap no amount of noise will close.

Within the relevant band they separate almost nothing. dA, dB and dB′ all land between 0.9970 and 0.9993, and their ordering is set by phrasing rather than by content.

Now run that against a real index. Your retriever fetches 50 candidates, a dozen of which are error-handling pages sitting in the same 0.99 band, each carrying its own arbitrary wording noise of about ±0.002. The one passage containing ERR_4021 has an intrinsic advantage of 0.0004 in that field — five times smaller than the noise.

Whether it lands at rank 2 or rank 11 is effectively a coin flip, and a top-5 cutoff turns that coin flip into a retrieval you either get or don’t.

So the failure is not blindness. It is resolution. Dense retrieval can see that your query is about errors; it cannot resolve which error page you meant, because the thing distinguishing them contributes less signal than the paraphrasing does.

That is a property of the compression, not a bug in the index. It will not improve by adding more dimensions or switching to a better model, because the objective those models are trained on is the objective that produced it.

That is also the sense in which this section’s title is meant. Dense search “misses” ERR_4021 not because the passage is absent or scored badly, but because it is one of a dozen near-ties and you only ever look at the top few.

BM25: the complementary blind spot

The key property to carry out of the worked example: an embedding is a lossy compression optimized for meaning, not for identity. Pooling a passage into 1,024 floats necessarily discards detail, and the training objective decided which detail is worth keeping — semantic gist, not rare literals.

So for a query like "why am I getting ERR_4021":

• The tokens ERR, _, 40, 21 (Tokens) each carry little semantic weight.
• Pooling washes them out against the surrounding words.
• The top of the results is therefore a near-tied cluster of passages about errors generally, and the one passage containing that exact string is somewhere in that cluster with no reliable claim to the top of it.

BM25 is the complement. BM25 is a classic lexical ranking function: it scores documents by which of the query’s exact terms they contain, weighting each term by inverse document frequency (IDF) — a score that rises as a term appears in fewer documents.

So a rare term like ERR_4021 gets a high weight precisely because it’s rare. That is the exact opposite of what pooling did to it.

The weight is an explicit function of rarity. Let N be the number of documents in the corpus and df be the document frequency, the number of documents containing the term:

idf = ln( (N − df + 0.5) / (df + 0.5) + 1 )

Substituting N = 100,000 and two very different df values, with the inner division shown:

"ERR_4021"  df =      3  ->  (99,997.5 / 3.5) + 1     = 28,572  ->  ln = 10.26
"error"     df = 40,000  ->  (60,000.5 / 40,000.5) + 1 =    2.5  ->  ln =  0.92

That is an 11× weight advantage (10.26 / 0.92), generated by the same property that made the embedding lose the term. Pooling punishes rarity — one term diluted in a mean of hundreds. IDF rewards it — one term with almost no competition.

Fusing the two isn’t a hedge. It is pairing two scorers whose blind spots are complements by construction, which is what the pipeline below does:

flowchart LR
    Q(["Query"]) --> D["Dense<br/>good at paraphrase<br/>blind to rare literals"]
    Q --> B["BM25<br/>good at rare literals<br/>blind to paraphrase"]
    D --> F["Fuse (RRF)"]
    B --> F
    F --> R["Rerank<br/>cross-encoder"]

    style F fill:#40916c,color:#fff
    style R fill:#2d6a4f,color:#fff

What Fuse (RRF) means. Reciprocal Rank Fusion combines the two result lists using only ranks — position 1, position 2, position 3 — and never the underlying scores. Each retriever contributes 1/(k + rank), and a document’s total is the sum over retrievers. The two example documents below show why that favors consistency:

score(d) = Σ over retrievers i of  1 / (k + rank_i(d))       k ≈ 60

a doc ranked 1st by dense and 8th by BM25:  1/61 + 1/68 = 0.0311
a doc ranked 3rd by both:                   1/63 + 1/63 = 0.0317   <- wins

Rank-based fusion exists because a cosine of 0.9993 and a BM25 score of 34.2 are not on a common scale. They have different ranges, different distributions, and no principled conversion between them. Any weighted sum of the raw numbers is a guess about that conversion.

Ranks throw the scale away and keep the only thing both retrievers agree on: ordering.

The constant k ≈ 60 damps the top of each list. Because 60 is large compared to the ranks involved, 1/61 and 1/63 are nearly equal, so being #1 in one retriever doesn’t automatically beat being solidly good in both — which is exactly the behavior you want when one retriever is having a blind-spot day. Same formula as chapter 05.

Why the Rerank cross-encoder stage beats both retrievers. A bi-encoder — what you search with — embeds the query and the document separately. They never see each other, which is precisely what lets you embed the whole corpus in advance and store it in an index.

A cross-encoder feeds query and document through the model together, so full attention runs across both at once. Far more accurate, and far too slow to run over a whole corpus, because nothing can be precomputed.

Hence the shape of the pipeline: retrieve 50 cheaply with the bi-encoders, rerank those 50 expensively with the cross-encoder, keep 5.

That’s the whole architecture of chapter 05, derived from what embeddings can and can’t represent.

8. Sampling, and why temperature=0 isn’t deterministic

Retrieval decides what goes into the window; sampling decides what comes out of it. Every pass of the loop ends with a vector of logits that must collapse into exactly one token, and temperature and top_p are the classical knobs on that collapse.

A caveat before the mechanics, because it changes what those knobs are worth to you.

Current frontier models have removed temperature and top_p from the API entirely. You cannot set either one on claude-opus-5. Steering is done through prompting and the effort parameter instead — a coarse low…max dial on how much thinking and tool work the model spends before answering, documented in Effort and thinking.

Two reasons to keep reading anyway.

First, temperature and top_p remain the vocabulary every provider, paper and interviewer uses for how a distribution gets narrowed, so you need to be able to say what they do.

Second, the last part of this section still bites you in production. It explains why identical inputs do not guarantee identical outputs even with all randomness turned off, which is a fact about the hardware and survives every API change.

From logits to one token

The final layer emits the logits of The forward pass — one raw score per vocabulary entry. Softmax (Attention and why context costs what it does) turns them into probabilities, and one token is sampled from the result.

Temperature divides every logit by a constant T before softmax runs.

Softmax converts differences between scores into ratios between probabilities (Attention and why context costs what it does). Dividing by a T smaller than 1 makes every logit bigger, which spreads those differences apart, which sharpens the distribution toward whichever token already scored highest — the argmax, meaning the position of the largest number in the list.

Push T toward 0 and the top token’s probability goes to 1. Sampling stops being random and always returns the top choice.

top_p, also called nucleus sampling, works by deletion rather than reshaping. Sort the tokens by probability, walk down the list adding them up, and keep the smallest group whose running total first exceeds p. Everything below that line is discarded, and the survivors are rescaled to sum to 1 again.

The whole lesson is six numbers. Take three logits — 3.0, 1.0, 0.5 — and run softmax on them at each temperature. Read the table across each row and watch the first column climb while the third collapses:

Notice that halving T doubled the logits — 3, 1, 0.5 became 6, 2, 1 — before softmax ever ran. Doubling every logit doubles every gap between them, and gaps are the only thing softmax responds to. That is the entire mechanism.

The third token went from a 1-in-15 shot (1 / 0.0674 = 14.8) to a 1-in-150 one (1 / 0.0066 = 151) without any of its own numbers changing.

top_p cuts rather than reshapes. At T = 1.0 the running totals down the row are 0.8214, then 0.8214 + 0.1112 = 0.9326, then 1.0.

With p = 0.9, the smallest set whose total exceeds 0.9 is the first two tokens, since 0.8214 alone does not clear the bar but 0.9326 does. The third token is dropped, and the survivors renormalize by dividing by 0.9326: 0.8214 / 0.9326 = 0.8808 and 0.1112 / 0.9326 = 0.1192.

The third token’s probability didn’t shrink — it became impossible. That is a different kind of change, and it is why top_p is the safer knob for cutting off tail nonsense.

Why temperature=0 still isn’t reproducible

Even at temperature 0, output isn’t reproducible across runs.

Floating-point addition isn’t associative: (a + b) + c and a + (b + c) can differ in the last bit, because each intermediate result gets rounded to the nearest representable float.

GPU kernels sum in an order that depends on batch composition — on who else’s request happened to be batched with yours, which you do not control and cannot observe. Different order, different last-bit rounding, occasionally a different argmax. From there the two sequences diverge completely, because every later token is conditioned on the one that differed.

Practical: never build a system whose correctness depends on identical output across runs. Cache by input hash if you need stability, and if a test needs determinism, assert on a property of the output, not on the exact string. This is also why eval CI needs N=3 majority (chapter 08) rather than exact-match gating.

That last point is the one worth carrying out of this section: the knobs at the top of it are gone from the API, and the reason you still cannot promise a byte-identical answer twice has nothing to do with them.

9. Structured output is a guarantee, not a request

Sampling and why temperature0 isnt deterministic ended by warning that you can never promise the exact string a model will produce. Its shape is a different matter.

“Return JSON matching this schema” in a prompt is a request the model usually honors. output_config.format is different in kind.

At each decode step, before sampling, the runtime masks the logits of every token that would make the output invalid under the schema, setting them to −∞. Sampling then draws from what remains.

That is one extra box inserted into The forward pass’s loop, between the logits and the sample:

flowchart LR
    L["Logits over<br/>full vocab"] --> M["Mask: set invalid<br/>tokens to -inf"]
    M --> SM["Softmax over<br/>survivors"]
    SM --> T["Sample"]

    style M fill:#2d6a4f,color:#fff

Watch one decode step

Take a two-field schema, and note the field order — it is the whole recommendation of this section made concrete:

{"type": "object",
 "properties": {"reasoning": {"type": "string"},
                "verdict":   {"type": "string", "enum": ["pass", "fail"]}},
 "required": ["reasoning", "verdict"], "additionalProperties": false}

Say the model has emitted {" and is choosing the next token.

The runtime has compiled that schema into a grammar: a machine that tracks how much of the output has been produced so far and can answer, at every step, which continuations are still legal. After {", the schema says the first required property is reasoning, so exactly one string can follow — reasoning".

So the runtime takes the ~100,000 logits over the full vocab from The forward pass and sets every invalid token to −∞ — every token that isn’t the next piece of reasoning". Then it runs softmax over the survivors.

Recall from Attention and why context costs what it does that exp(−∞) = 0. So the invalid tokens carry probability exactly 0, and the sole survivor renormalizes to 1.0. Sampling then “chooses” the only thing left.

Later, mid-verdict, the grammar permits exactly two continuations — pass and fail — so the mask leaves two live tokens and the model’s preference between them is preserved. Masking removes the illegal; it doesn’t override the model’s judgment among the legal.

The call itself, with the field order that matters:

client.messages.create(
    model="claude-opus-5",
    output_config={"format": {"type": "json_schema", "schema": JUDGE_SCHEMA}},
    messages=[...],
)

(output_config.format is the API-level parameter; the SDK’s typed helper client.messages.parse(..., output_format=PydanticModel) is a convenience wrapper over the same thing, which is the name you’ll see in chapter 08.)

Invalid output isn’t unlikely — it has probability zero by construction. That’s why you should never write a JSON-repair retry loop around a structured-output call: there is nothing to repair.

Two costs worth knowing: a one-time schema compilation on first use (later calls hit a cache), and a mild quality effect if your schema forces an awkward generation order — put a reasoning field before the answer field so the model can think before committing, which is exactly why the LLM-judge schema in chapter 08 is ordered that way.

Tool calling is the same machinery. Tool schemas are serialized into the prompt, the model emits a structured call, the runtime parses it and sets stop_reason: "tool_use". There is no separate “function calling engine” — it’s tokens in, tokens out, with constrained decoding keeping the shape valid.

10. Deriving the numbers

That was the last mechanism; what remains is arithmetic. None of the headline cost numbers used across the rest of the series — the per-turn agent cost, the caching discount, the multi-agent multiplier, image cost — needs to be memorized, because each one is a few lines of algebra away from the mechanisms above.

Why an agent turn costs what it does

The API is stateless, as the anatomy of a request established at the top of this chapter, so turn t resends the entire history. Two symbols, used by every chapter that cites this one:

• P — the fixed prefix: system prompt + tool schemas. Paid on every request, never grows.
• a — the per-turn delta: assistant text + tool result for one turn. This is what accumulates.

Turn t sends the prefix once plus the deltas from all t−1 turns before it. Summing that over the whole run gives the n·P term (the prefix, resent every turn) plus a sum of integers, which is where the quadratic comes from:

input(turn t)            =  P + (t−1)·a
total input over n turns  =  n·P  +  a·(0 + 1 + ... + (n−1))
                          =  n·P  +  a·n(n−1)/2   ≈   n·P + a·n²/2

Substitute chapter 09’s reference agent — P = 6,000 (system + tool schemas), a = 1,200 (assistant text + tool result per turn).

The table below runs that formula at 10 turns and at 40 turns. The ratio row is 40-turn value over 10-turn value, column by column. n(n−1)/2 is 45 at n = 10 and 780 at n = 40:

Read the ratio row, not the total. 4× the turns cost 10.3× the tokens, and the two component columns tell you why: the fixed prefix grew only 4×, the quadratic term grew 17.3×, and the total lands between them, weighted by which term is bigger.

That bracket is worth stating explicitly, because both endpoints get quoted as if they were the answer. 16× — from 40²/10² — is the quadratic term alone, true only in the limit P → 0. 4× is the linear term alone, true only as a → 0.

The real multiplier for 10 → 40 turns lives between 4× and 16×, and slides toward 16× as a grows relative to P. Quote 10.3× for this agent, and quote the bracket for agents you haven’t measured.

Convention footnote: here turn 1 carries no delta, so the sum is a·n(n−1)/2 — the same convention chapter 09 uses. Counting the delta from turn 1 instead gives a·n(n+1)/2, larger by exactly a·n (48,000 tokens at n = 40, about 4%). Both round to a·n²/2, which is why chapters 01/02/04/06 can quote the approximation; just don’t mix conventions inside one calculation.

Now apply caching, with arithmetic. Put a breakpoint at the end of each turn (Prompt caching derived). Turn 1 writes the whole prefix at 1.25×. Every later turn reads what already existed at 0.1× and writes only its own new a tokens at 1.25×:

turn 1        1.25 × 6,000                                  =    7,500
turns 2..40   0.1 × (prefix carried in)                      =  112,320
              1.25 × 1,200 × 39   (each turn's new tokens)   =   58,500
                                                    total    =  178,320

The 112,320 is the only line there that isn’t one multiplication, so here is where it comes from. Arriving at turn t, everything except that turn’s own new tokens has been seen before and is therefore a cache read: the 6,000-token prefix, plus the deltas from the t − 2 turns before the newest one. Summed over the 39 turns from 2 to 40:

prefix carried in at turn t   =  6,000 + (t−2) × 1,200

Σ over t = 2..40   =  39 × 6,000  +  1,200 × (0 + 1 + ... + 38)
                   =  234,000     +  1,200 × 741
                   =  234,000     +  889,200
                   =  1,123,200 tokens read from cache

billed at 0.1×     =  0.1 × 1,123,200  =  112,320

1,176,000 → 178,320 billed-equivalent tokens: 1,176,000 / 178,320 = 6.6×, not 10×.

The gap between 6.6× and the 10× you might expect from a 0.1× read price is the point. The newest turn is always uncached and always pays the 1.25× write, so the discount applies to history but never to the present. How close you get to 10× depends on what fraction of each request is history.

That is the entire reason caching is the first optimization in chapter 09. It is also why caching does not make the quadratic go away: it multiplies the quadratic term by ~0.1, but it does not change the term’s shape.

Why multi-agent is “4–15×”

Not a folk number. A multi-agent setup is one orchestrator model that splits a task into pieces, w workers that each run their own independent agent loop of s steps on one piece, and then the orchestrator again to stitch the results together.

flowchart TD
    U(["Task"]) --> O1["Orchestrator<br/>decompose into pieces"]
    O1 --> W1["Worker 1<br/>own agent loop, s steps"]
    O1 --> W2["Worker 2<br/>own agent loop, s steps"]
    O1 --> W3["Worker w<br/>own agent loop, s steps"]
    W1 --> O2["Orchestrator<br/>synthesize results"]
    W2 --> O2
    W3 --> O2
    O2 --> R(["Answer"])

The three things being counted:

single agent:   1 context, grows to B tokens              <- the denominator
orchestrator:   1 decompose call + w summaries + 1 synthesis
each worker:    s steps over its own window, W tokens, quadratic in s

The ratio needs a denominator, so name it. B is what the single agent would have billed on the same question: one context, n·P + a·n²/2, stopped whenever it decided it had enough.

The numerator is w·W + O. W is one worker’s total. O is the orchestrator: it prefills P to decompose, reads w summaries of ~800 tokens each, and does the same again to synthesize — hence the 2 × (...) in the table.

Two bracket ends, low and high. Read the last column:

Where the four inputs come from, honestly. B and W are illustrative bracket ends, not measurements.

They are round numbers chosen to span the plausible range: a single agent that stops early (25,000) versus one that works the question harder (40,000), against workers on short windows (40,000) versus deep ones (60,000). You should substitute your own, which is the entire point of writing the ratio as a formula rather than quoting a multiplier.

W is just n·P + a·n²/2 evaluated for one worker. At the reference agent’s P = 6,000 and a = 1,200, eight steps give 8 × 6,000 + 1,200 × 28 = 48,000 + 33,600 = 81,600. A worker on a leaner prefix or a shorter leash lands in the 40k–60k band used here. The formula is the claim; the four numbers are a worked illustration of it.

The range is wide because both ends move. The high end has more workers reading deeper windows and is compared against a thriftier single agent.

Notice how small O is: 5.7–10.3% of the total (18,400 / 178,400 = 10.3% at the low end, 21,600 / 381,600 = 5.7% at the high one). The orchestrator is not the expense. w × W is the expense, which names the levers immediately — worker count (linear), worker step budget (quadratic within each worker), worker model tier (~2× between tiers).

It is a work-volume multiplier, not an efficiency penalty, and reading it the other way is the usual mistake. You are comparing one agent that stopped at 8 searches against 4 workers doing 8 each, which is 32 searches.

Per unit of work, fan-out is actually cheaper. One agent pays a·n²/2 on a single growing context. Split that same n steps across w workers and each one runs only n/w steps, paying a·(n/w)²/2 — and w of those sum to w · a·n²/(2w²) = a·n²/(2w).

The quadratic divides by the worker count, because splitting one long context into w short ones is exactly what kills quadratic growth. See chapter 06.

Why images dominate computer-use cost

Don’t memorize a token count for “an image” — memorize the formula and the tier. The model does not bill images by area. It cuts the image into 28×28-pixel patches and charges one token per patch, so the count is a product of two ceilings:

visual tokens  =  ceil(width / 28)  ×  ceil(height / 28)

Each tier then imposes two limits, and an image over either is downscaled — aspect ratio preserved — until it fits both. The last column applies all of this to the same 1080p screenshot, so you can compare the two tiers directly:

Check the formula against both rows.

On the high-resolution tier nothing is resized, because the long edge of 1,920 px is under the 2,576 cap and the resulting token count is under 4,784. So ceil(1920/28) = 69 and ceil(1080/28) = 39, giving 69 × 39 = 2,691.

On the standard tier the long edge is over its 1,568 px cap and the patch count would be over the 1,568-token cap, so the image is scaled down to 1456×819 — the largest 16:9 size fitting both — giving ceil(1456/28) × ceil(819/28) = 52 × 30 = 1,560.

Both limits matter, and the token one is the one people forget. Scaling 1080p to exactly the 1,568-pixel long edge gives 1568×882, which works out to 56 × 32 = 1,792 tokens. That is over the standard tier’s 1,568-token cap, so that size is not actually reachable there.

A figure that satisfies the pixel limit can still violate the token limit, which is why quoting a long-edge cap alone tells you nothing about cost.

w × h / 750 is a useful mental shortcut and not the billing rule. It approximates 784 pixels per patch (28 × 28) while dropping both ceilings, and it runs about 3% high: 2,073,600 / 750 = 2,765 against the true 2,691.

The reason is worth being precise about, because the two approximations pull in opposite directions.

Dropping the ceilings makes the estimate run low. 2,073,600 / 784 = 2,645, which is 1.7% under 2,691, because rounding each axis up to a whole patch is exactly what plain division throws away.

What pushes the number back over the line is the divisor itself. 750 is 4.3% smaller than 784, and dividing by a smaller number inflates the result — by 4.5%, since 784/750 = 1.045.

That +4.5% overshoot more than cancels the ceilings’ −1.7% undershoot, for a net +2.7%. Use the shortcut to estimate; use the patch formula when the number has to be right, and check it against the tier’s token cap before believing it.

Now compute the cost of a whole run. An image is resent on every subsequent turn until you trim it, so by turn t the request carries t images. Over a 20-step run at 2,691 tokens each on the high-resolution tier, using 1 + 2 + ... + 20 = 210:

Σ (t · 2,691) for t = 1..20  =  2,691 × 210  =  565,110 image tokens

Now keep only the last 3 images. Turns 1 and 2 hold just 1 and 2 images, and every turn from 3 on holds exactly 3, so the coefficient is Σ min(t,3) = 1 + 2 + (18 × 3) = 57, not 60:

57 × 2,691  =  153,387     ->  565,110 / 153,387  =  3.7× reduction

The ratio does not depend on the per-image count. Both sides are a coefficient times 2,691, so the 2,691 cancels and the ratio is just 210 / 57 = 3.7. Change the tier or the resolution and only the absolute totals move.

Two “trimming wins” appear in this series and they measure different things. This 3.7× is image tokens only, on a 20-step run, uncached. The 1.4× below is the whole bill including text and output, on the 7-turn task in case study 01. Neither is wrong; quoting one where the other belongs is.

And that shared word uncached is the whole lesson, because this is a case where two mechanisms in this chapter pull against each other.

Trimming is a Deriving the numbers argument: fewer tokens resent is less money.

Caching is a Prompt caching derived argument: a shared prefix is reusable, and a change invalidates everything after it.

Those collide, because a sliding window over images is a change to the prefix. Dropping the oldest image every turn moves the point where this request stops matching the last one, by one image, every turn. So the surviving images get re-prefilled at full rate, forever, instead of billing at 10%.

Which way the trade lands depends entirely on whether you were caching in the first place:

uncached:  trimming is a ~1.4x win     (you avoid resending tokens)
cached:    trimming is a ~1.9x loss    (you avoid tokens that cost 0.1x,
                                        and pay 1.25x to rewrite the prefix)

Lowering the resolution has no such conflict — it shrinks every image without moving the divergence point.

When two mechanisms give opposite advice, the one that touches the prefix usually wins, because prefix effects compound over every remaining turn. Case study 01 derives both numbers.

Every number in this chapter, in one table

If you can reproduce this column-by-column, you can rebuild the rest of the series from first principles. Each row is mechanism → formula → the values I substituted → what came out.

The mechanism → rule map

One last table. The left column is a mechanism from this chapter; the right column is every practical rule that falls out of it. Cover the right column and try to regenerate it:

If you can reconstruct the right column from the left, you can answer follow-up questions this series never anticipated — which is the actual goal.

Next: 01 — Agent Foundations.