09 — Production & Cost

An agent’s running cost reads like the model provider’s bill, but nearly all of it is set by your own design decisions — and it can usually be cut several-fold without making the agent worse.

An agent, here, is a program that calls a large language model (LLM) — a service you send text to and get generated text back from — in a loop, feeding each result back in so the model can decide the next step.

The loop is the whole reason cost matters. One agent task is not one model call; it is ten or twenty. So anything that makes a single call cheaper or faster multiplies across every step of every run.

This chapter derives the per-task bill from a description of one run, names the three factors it decomposes into, and gives a fixed order for reducing them.

Inference is the act of running a trained model on a prompt to produce output. It is the thing you pay the model provider’s application programming interface (API) for, and the only thing. Your inference cost is your agent cost.

Graphics processing unit (GPU) engineering, inference optimization and agent design are one stack, not three separate fields: a fact about how a GPU reads its own memory ends up dictating which line of your agent to change first.

What goes in, and what comes out

The input to every calculation in this chapter is a description of one agent run:

• how many model calls it makes, written n;
• how many tokens each call sends and receives;
• which model serves each call;
• whether the prompt is cached.

A token is the unit a model reads, writes and bills in — a common word or word-fragment, roughly three-quarters of an English word on average, so 1,000 tokens is about 750 words. Prices are quoted per million tokens, abbreviated MTok.

The output is two numbers and a decision:

• dollars per completed task;
• seconds of wall-clock latency per task;
• and which single change buys the most of either.

One worked example runs the length of the chapter. A 12-call agent starts at $0.876 per task, and by the end costs $0.137 — a 6.4× reduction with no quality change on the hard path, meaning the share of tasks that still run every step on the strongest model.

Model routing is where the traffic gets split into a hard path and a cheap one. Until then, every task is a hard-path task.

The price list you will be substituting into

Every dollar figure in this chapter comes from the three rows below. Read the table as: what it costs to send a million tokens to this model, what it costs to get a million back, and the largest prompt it will accept.

Input tokens are everything you send: the system prompt, the tool definitions, the whole conversation so far, and the new turn. Output tokens are what the model generates in reply. Context is the largest prompt the model will accept at all, in tokens — 1M means one million.

Three multipliers that bend those prices

The list price is never quite what you pay. Three multipliers sit on top of it, and each one is a lever a later section pulls:

• Cache read: ~10% of the input price. A token the API has already seen, in a prompt it has already processed, bills at a tenth.
• Cache write: 1.25× the input price, with a five-minute time-to-live (TTL — how long the cached copy survives before it is discarded). Or 2× for a one-hour TTL. Putting something into the cache costs a premium; reading it back is what repays you.
• Batch API: 50% of everything. The Batch API is a queue for work you do not need answered immediately.

Prompt caching the highest leverage lever derives what the cache multipliers buy you; Batch and semantic caching covers the batch one.

The one pattern to notice in the table

The output/input ratio is exactly 5× on every tier. $25/$5 on Opus, $15/$3 on Sonnet, $5/$1 on Haiku.

That is not three independent pricing decisions that happened to agree. It is one hardware asymmetry showing up three times.

Prefill vs decode the fact underneath every cost rule derives that asymmetry, and is also honest about its limit: the hardware tells you output must cost more, and tells you the multiplier should be the same across a vendor’s tiers, but it does not hand you the integer 5. That last step is a pricing decision.

1. The cost identity

One equation governs this whole chapter; every later section moves one of its terms. Two of the three terms also turn out to be coupled rather than independent, and that coupling drives most of the advice that follows.

The three factors

The cost of one task splits into exactly three multiplied factors, each owned by a design decision. No factor is owned by the model provider; all three are yours.

flowchart TD
    T["cost per task<br/>= calls x tokens/call x price/token"] --> A["calls per task<br/>controlled by: PATTERN"]
    T --> B["tokens per call<br/>controlled by: CONTEXT"]
    T --> C["price per token<br/>controlled by: MODEL + CACHE"]

    A --> A1["ch 02 pattern choice<br/>step caps, fewer round trips<br/>parallel over serial tools"]
    B --> B1["ch 04 offload, compact, truncate<br/>downsample images"]
    C --> C1["route to cheaper tiers<br/>prompt caching, batch API"]

    style A fill:#40916c,color:#fff
    style B fill:#40916c,color:#fff
    style C fill:#2d6a4f,color:#fff

In words: cost per task = calls × tokens/call × price/token. One factor at a time:

• Calls per task, controlled by PATTERN. The shape of the agent loop, chosen when you pick the pattern in ch 02. You move it by capping the number of steps, cutting round trips, and running tools in parallel instead of one after another.
• Tokens per call, controlled by CONTEXT. How much text you put in each prompt. You shrink it by offloading large tool outputs to files, compacting history, truncating, and downsampling images before sending them (ch 04).
• Price per token, controlled by MODEL + CACHE. Routing work to cheaper model tiers, prompt caching, and the Batch API.

The identity, written out

Two lines: the exact sum, then the approximation you actually reason with.

cost(task) = SUM over calls c of  [ in_tokens(c) x price_in  +  out_tokens(c) x price_out ]
           ~ n x t_bar x p_bar          (mean-field form)

The first line is the honest sum: for every call the agent makes, add up what its input tokens cost and what its output tokens cost.

The second line is the same thing with averages substituted — n calls, t_bar tokens per call on average, p_bar dollars per token on average (a mean-field form: every individual value replaced by the population average). The shape of the answer survives the substitution, and the shape is what you reason with.

Three factors, three different owners. n is an architecture decision, t_bar is a context-engineering decision, p_bar is a procurement decision. That is why “make it cheaper” is never one conversation — you have to say which factor you are moving.

The factors are not independent, and the coupling is what surprises people

The API is stateless: it remembers nothing between calls. Every turn resends the entire conversation so far (Deriving the numbers derives the token arithmetic of that).

That single fact means t_bar is not a free parameter. It is a function of n.

Set up two symbols. P is the stable prefix — the system prompt and tool definitions, which never change. a is the number of tokens each turn appends to the conversation. Then the input to call number t is:

in_tokens(call t) = P + (t-1)a

Call 1 sends P + 0a. Call 2 sends P + 1a, because one turn happened before it. Call 3 sends P + 2a. And so on.

Now add those up over all n calls. The P appears n times. The deltas are 0a + 1a + 2a + ... + (n-1)a, which is a times the sum of the integers from 0 to n-1 — and that sum is n(n-1)/2:

total input over n calls = n.P + a.n(n-1)/2      <- quadratic in n

The first term grows with n. The second grows with n². That second term is the one people miss.

Cutting calls pays superlinearly

“Superlinearly” means the saving is larger than the fraction of calls you removed. Cut a third of the calls and you save more than a third of the tokens.

Here is the arithmetic on a concrete agent: a 12-call ReAct agent. ReAct stands for Reason + Act, the pattern where the model alternates between writing a thought and calling a tool (React reason act). Its numbers:

• P = 6,000 tokens of system prompt plus tool schemas
• a = 1,200 tokens per turn of assistant text plus one tool result

Substitute into n.P + a.n(n-1)/2 for n = 12 and again for n = 8. The turn-delta counts come from n(n-1)/2: 12 × 11 / 2 = 66 and 8 × 7 / 2 = 28.

n = 12:  12 x 6,000 + 1,200 x 66   = 72,000 + 79,200 = 151,200 input tokens
n =  8:   8 x 6,000 + 1,200 x 28   = 48,000 + 33,600 =  81,600 input tokens

         (66 = 11x12/2 turn-deltas;  28 = 7x8/2)

Now compare the two cuts side by side:

• Calls: 12 → 8, a cut of 4 / 12 = 33%.
• Input tokens: 151,200 → 81,600, a cut of 69,600 / 151,200 = 46%.

A 33% cut in calls produced a 46% cut in input tokens. The extra 13 points came from the quadratic term — the deltas you removed were being resent by every call that came after them.

This is the single best argument for fixing the pattern before fixing the prompt, and it is the reason for the ordering in Optimization order.

The reference task, uncached

Now put dollars on the 151,200 input tokens. Two more facts about the same 12-call agent: it generates 400 output tokens per call, and every call goes to claude-opus-5 at $5 per million input and $25 per million output.

Output tokens total 12 × 400 = 4,800. Multiply each side by its price:

input   151,200 x $5/1M   = $0.7560
output  12 x 400 = 4,800 x $25/1M = $0.1200
                                    -------
                                    $0.8760 per task

Hold that number. Every optimization below is measured against $0.876.

Note the split already: input is $0.756 of it, or 86%. That is not because input tokens are expensive — they are the cheap half, at a fifth the price — but because there are 31.5 times more of them (151,200 / 4,800).

What interviewers probe: “Where does the money actually go in an agent?” Weak: “the model is expensive.” Strong: name the three factors, say which design decision owns each, then point out that factor 1 and factor 2 are coupled quadratically — so the outer factor is worth more than its size suggests.

2. Prefill vs decode — the fact underneath every cost rule

Why are output tokens the expensive half? Why can caching never speed up generation? Why does trimming a long prompt help latency even after caching has already fixed the price? One hardware asymmetry answers all three.

Learn it once and most of the advice in this chapter stops needing to be memorized — though it is worth being exact about how far the asymmetry gets you toward the 5× and where the derivation stops.

The two phases

Serving one request happens in two phases with completely different performance characteristics: prefill runs once, straight through; decode is a loop that runs once per output token.

flowchart TD
    subgraph PF["PREFILL - the whole prompt, one shot"]
        P1["all input tokens in parallel,<br/>one large batched matmul<br/>each weight read ONCE for the whole prompt"]
        P3["bottleneck: GPU FLOPs<br/>thousands tok/s"]
        P1 --> P3
    end
    subgraph DC["DECODE - one token at a time"]
        D1["compute Q for ONE new token"]
        D2["re-read every weight<br/>+ the ENTIRE KV cache"]
        D3["emit 1 token, append its K,V<br/>bottleneck: memory bandwidth, tens tok/s"]
        D1 --> D2 --> D3 --> D1
    end
    PF --> DC

    style PF fill:#2d6a4f,color:#fff
    style DC fill:#bc6c25,color:#fff

Phase 1: prefill

The first phase is PREFILL — the whole prompt, one shot. The model reads every input token you sent, all in parallel, as one large batched matmul: a matrix multiplication, the arithmetic operation a GPU is built to do in bulk.

Because the whole prompt goes through together, each weight in the model is read once for the whole prompt rather than once per token. The cost of loading the model is amortized — spread across every token in the prompt instead of paid again for each one.

The limit is therefore raw arithmetic throughput. The bottleneck is GPU FLOPs, floating-point operations per second, and prefill runs at thousands of tokens per second.

Phase 2: decode

The second phase is DECODE — one token at a time — and it is a loop. Each pass through it does three things: compute Q for one new token, re-read every weight in the model plus the entire KV cache, then emit 1 token, append its K and V, and go round again.

Two definitions make that loop readable.

Q is the query vector: the model’s representation of “what is this new token looking for.”

The KV cache (key/value cache) is the model’s stored summary of every token seen so far. For each earlier token it holds a key vector K saying what that token offers, and a value vector V carrying what it contributes.

Attention works by matching the new token’s Q against every stored K. So producing even one token requires streaming the entire cache from memory. That makes the loop limited by memory bandwidth — how fast bytes move from memory into the compute units — not by arithmetic. It runs at tens of tokens per second.

The asymmetry, in one sentence

Prefill amortizes one pass over the weights across the whole prompt. Decode pays a full pass over the weights and the entire KV cache to produce a single token.

That asymmetry is what output pricing is tracking.

Deriving the decode ceiling

To see why decode is bandwidth-bound rather than compute-bound, size the cache and divide by how fast bytes move. Two definitions first.

Heads. Attention does not run once per layer as a single wide operation. Each layer splits it into several independent heads, and each head attends over the same tokens in its own narrow slice of the representation, d numbers wide. So a layer does not store one K and one V per token — it stores one pair per head. H × d is the width of what a single layer keeps for a single token.

bf16. Brain float 16, a 16-bit floating-point format. 2 bytes per number.

So for a model with L layers, H key/value heads and head dimension d, here is the cache cost per token. The values substituted are for a 70B-class dense transformer using grouped-query attention (GQA, where many query heads share one key/value head — this is what keeps H at 8 rather than 64): L = 80, H = 8, d = 128.

bytes/token = 2 (K and V) x L x H x d x 2 bytes
            = 2 x 80 x 8 x 128 x 2  =  327,680 B  ~  320 KB per token

The leading 2 is because both a K and a V vector are stored. The trailing 2 is the bytes per number.

Now hold a 50,000-token conversation in that cache and ask how fast one token can possibly come out. The hardware is an H100-class data-center accelerator, whose high-bandwidth memory (HBM) moves roughly 3 terabytes per second:

KV resident      = 50,000 x 320 KB          =  16 GB
HBM bandwidth    ~ 3 TB/s
time per token   = 16 GB / 3 TB/s            =  5.3 ms
decode ceiling   = 1 / 0.0053                ~  190 tok/s

That is a ceiling, not an estimate. Even with infinite arithmetic capacity, the bytes have to move.

The 190 is rounded, and here is the unrounded version

Carried through without rounding: 327,680 B is exactly 320 KiB, so the 50,000-token cache is 16.384 GB. At 3 TB/s that is 5.46 ms per token and 183 tok/s — 4% below the rounded figure.

The roundings stay because the claim is about the order of magnitude: decode is in the hundreds of tokens per second, not the thousands and not the tens. If you are checking the arithmetic rather than the argument, check it against 183.

Reconciling the four throughput numbers in this chapter

Four different tokens-per-second figures appear across these pages. They are not in conflict; they are measured at different points.

Decode’s ~50 sits several times below its 190 ceiling because the accelerator is serving other people’s requests too.

Prefill does not follow that pattern, and the reason is worth stating. This chapter never derives a prefill ceiling — only the decode one. So both prefill numbers, the “thousands” above and §6’s ~20,000, are loaded observations with no ceiling to sit under.

That is also why the loaded prefill figure is the larger of the two rather than a fraction of something. Prefill is the phase batching genuinely helps: concurrent requests share a single pass over the weights, and what remains is arithmetic a GPU can saturate.

Decode cannot be helped that way. Every request owns its KV cache, and every cache must be streamed again for every token. That is why loading an endpoint drives decode down and prefill up.

Derive with ceilings where you have one; budget with loaded numbers.

Batching helps the weight read, not the KV read. Twenty concurrent requests share one pass over the weights, so the cost of loading the model is split twenty ways. But every request has its own KV cache, and every request’s cache must be streamed every step — no sharing there. That residual, unamortizable cost is what output tokens are priced on.

How far this gets you toward the 5× — and where it stops

Be precise about what that derivation proved, because this is the point at which a careless answer becomes a wrong one.

What the hardware does establish, and you can defend all of it:

1. Output must cost more than input. Prefill reads each weight once for the entire prompt; decode reads every weight and the whole KV cache once per token. The per-token work is not comparable, and it is worse for decode by a large factor.
2. The multiplier should be roughly constant across a vendor’s tiers. All three models run the same two-phase algorithm on the same class of accelerator. The asymmetry between the phases is a property of the algorithm and the memory hierarchy, not of the model’s size — so it does not have a reason to be 5 on Opus and 2 on Haiku. This is why the identical ratio across the table is evidence and not coincidence.

What it does not establish is the number 5. Nothing in this section computes it.

In fact the raw hardware ratio is nowhere near 5. Prefill runs in the thousands of tokens per second and decode at a couple of hundred, so the per-request throughput gap is one to two orders of magnitude — call it 10× to 100×, not 5×.

Pricing lands far below that for two reasons:

1. Batching amortizes the weight read across concurrent requests, which shrinks the gap.
2. List prices carry margin, which is a commercial choice, not a physical one.

A third fact sits underneath both without being a reason of its own: a served token’s cost depends on batch size, sequence length and accelerator utilization. That is what makes the ratio impossible to quote from hardware alone — not what makes it small. None of those quantities are stated in this chapter, and two of the three inputs above are commercial rather than physical.

So: 5× is a pricing decision informed by the hardware asymmetry, not a quantity derivable from it. Anyone who tells you they derived the 5 from bandwidth figures skipped the batching term.

That distinction is worth holding onto, because the usable conclusion survives it intact. Every rule in this chapter that depends on the ratio — shorten output before input, edit over write, terser tool arguments, lower effort — needs only “output is several times more expensive and sets the latency floor.” It never needs the 5 to be exactly 5.

The three consequences you will actually use

Everything above collapses into three working rules. Read the middle column as the one-line justification you would say out loud in an interview.

The first row’s example is worth unpacking, because it is the same principle in a different costume: a coding agent that rewrites a whole file (write) must generate every line as output tokens, while one that emits a small patch (edit) generates only the changed lines. Same result, a fraction of the decode.

What interviewers probe: “Why is output more expensive than input?” The weak answer is “generation is harder.” The strong answer is one sentence: prefill is a parallel, compute-bound matmul over the whole prompt; decode is sequential and memory-bandwidth-bound because every single token re-reads the full weight set and the entire KV cache. Then note that the ratio is the same 5× on every tier, which is what tells you it is tracking the algorithm rather than a per-model margin call. Do not claim you can compute the 5 from bandwidth numbers — you cannot without the batch size, and an interviewer who knows this will ask.

3. Prompt caching — the highest-leverage lever

Exactly one optimization costs nothing in quality, and the hardware argument above says why it can exist: prompt caching, which attacks prefill and only prefill. It starts paying almost immediately, an agent conversation is the ideal thing to feed it, and a short list of mistakes will silently switch it off.

The mechanism

Caching stores the KV cache for a prefix — the leading stretch of the prompt, counted from its very first byte — so that prefill can be skipped on the next request that starts with those same bytes.

It works because attention is causal: a token may only attend to itself and the tokens before it, never to tokens that come later. So a token’s K and V depend only on the text up to that point.

Two consequences follow, and they are the whole of prompt caching:

1. A shared prefix is reusable. If two requests start with the same bytes, the K and V for those bytes are identical, so the second request can load them instead of recomputing them.
2. A change invalidates everything after it, and only after it. Nothing before the changed byte is affected; nothing after it survives (Prompt caching derived).

The cache key is the literal bytes of the prompt. “The same prefix” means byte-identical, not merely equivalent in meaning.

Prompt ordering: cacheable vs never-caches

Two prompt layouts, distinguished by where the volatile part sits: at the back in the cacheable one, at the very front in the one that never caches.

flowchart LR
    subgraph GOOD["Cacheable"]
        A1["tools (stable)"] --> A2["system (stable)"] --> AC{{breakpoint}} --> A3["history"] --> A4["new turn"]
    end
    subgraph BAD["Never caches"]
        B1["system + timestamp"] --> B2["tools"] --> B3["history"]
    end

    style AC fill:#2d6a4f,color:#fff
    style B1 fill:#9d0208,color:#fff

The upper row is the cacheable ordering. Tools (stable) first, then system (stable), then a breakpoint — an explicit marker telling the API “cache everything up to here” — and only after it the history and the new turn, the two parts that grow every request.

The lower row is the ordering that never caches. Put a system + timestamp block at the very front and every request differs a few dozen tokens in, so nothing behind it can ever be reused.

“Position 30” is used below as a stand-in for that offset. The exact number depends on your prompt; the argument only needs it to be early.

In code, the breakpoint is a cache_control marker on the last block you want cached. Everything from the start of the prompt up to and including that block gets stored:

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=8192,
    system=[{"type": "text", "text": BIG_STABLE_PROMPT,
             "cache_control": {"type": "ephemeral"}}],   # tools + system cached
    tools=TOOLS,                                          # sorted, deterministic
    messages=messages,
)
print(resp.usage.cache_read_input_tokens)   # if 0 across repeats -> invalidator

Break-even, derived

Caching is not free. The first request pays a premium to write the cache, and later requests pay a much smaller amount to read it. So the question is: how many requests does it take before the write premium is repaid?

Take N requests that share a prefix of P tokens. To keep the algebra clean, measure everything in units of one uncached prefill of P. So an uncached request costs 1.0 by definition, a cache write costs 1.25, and a cache read costs 0.10 — those are just the multipliers from the price list.

uncached:  N x 1.0
cached:    1.25 (the write)  +  (N-1) x 0.10 (the reads)

break even when   1.25 + 0.10(N-1) = N
                  1.15 = 0.90 N
                  N = 1.28

The middle line is worth expanding: 1.25 + 0.10N - 0.10 = N, so 1.15 = 0.90N, so N = 1.28. Requests come in whole numbers, so anything from 2 up wins.

Check N = 2 directly: cached is 1.25 + 0.10 = 1.35 against uncached 2.00. That is a 32% saving across the pair.

Check N = 10: cached is 1.25 + 9 × 0.10 = 2.15 against uncached 10.00. That is 4.7×.

The one-hour TTL needs one more request

The long TTL costs 2× to write instead of 1.25×, so the same algebra shifts:

2.00 + 0.10(N-1) = N   ->   1.90 = 0.90 N   ->   N = 2.11

At N = 2 the long TTL loses: 2.00 + 0.10 = 2.10 against 2.00 uncached. N = 3 is its first win: 2.00 + 0.20 = 2.20 against 3.00.

That single extra request is the whole difference between the two TTLs. It is why you pick the long one only for prefixes you know get re-hit across a gap longer than five minutes — a nightly batch job over a shared 80,000-token corpus, not a chat session.

The minimum-prefix floor

There is also a floor below which caching silently does nothing. The minimum cacheable prefix is model-dependent, and the values are not ordered by generation:

Below that it does not cache and does not tell you: no error, and cache_creation_input_tokens: 0 in the response.

Why the agent conversation prefix is the ideal caching target

A prefix is worth caching only if it has two properties, and it needs both:

1. It grows monotonically — it only ever appends, never rewrites. Turn t’s message array is turn t-1’s array plus an append. Nothing in the middle is rewritten, so the previous request’s prefix is a genuine prefix of this one — never a diverging branch.
2. It stays byte-stable. Tool schemas and the system prompt do not move; history is immutable once written.

Now compare that with a retrieval-augmented generation (RAG) prompt — one that searches a document store on each turn and pastes the retrieved passages into the prompt.

A RAG prompt has neither property. Fresh documents arrive every turn, and they land near the front, ahead of the question. So the prefix is rewritten, not appended to.

That is why the agent loop is where caching pays several-fold and a retrieval prompt is where it pays almost nothing.

Two cautions on the numbers you will hear quoted for this:

• The “5× vs 1.2×” shorthand is a rule of thumb, not a derived result.
• That 5× is a different quantity from the output/input 5× in Prefill vs decode the fact underneath every cost rule. They are unrelated and merely happen to be the same integer.

The derived caching win on this chapter’s own reference task is 2.9×, computed just below. The mechanism is the reliable part: retrieved documents belong after the breakpoint, not before it.

The practical rule follows directly: put a breakpoint on the last block of each new turn, and every subsequent request reads the entire prior conversation at 10% of list price.

The reference task, cached

Apply that to the running example. The same 12-call agent now uses a rolling breakpoint: on every call, the breakpoint moves to the end of what has been written so far.

Walk the first three calls to see the pattern:

• Call 1 sends 6,000 tokens (the prefix). Nothing is cached yet, so all 6,000 are a write.
• Call 2 sends 6,000 + 1,200 = 7,200. The first 6,000 are already cached, so they are a read; the new 1,200 are a write.
• Call 3 sends 6,000 + 2,400 = 8,400. The first 7,200 are cached (call 2 wrote its delta), so they are a read; the new 1,200 are a write.

Generalizing: call t (for t ≥ 2) reads whatever existed at call t-1, which is 6,000 + (t-2) × 1,200 tokens, and writes 1,200 fresh ones.

Sum the reads over calls 2 through 12. That is 11 terms. Each carries the 6,000 prefix, and the deltas run 0 × 1,200 up to 10 × 1,200 — so the multiplier is 0 + 1 + ... + 10 = 10 × 11 / 2 = 55:

reads  = SUM(t=2..12) [6,000 + (t-2) x 1,200]
       = 11 x 6,000 + 1,200 x 55  = 66,000 + 66,000 = 132,000 tokens @ 0.10x
writes = 6,000 (first call) + 11 x 1,200            =  19,200 tokens @ 1.25x
check:   132,000 + 19,200 = 151,200  = total input  OK

reads   132,000 x $5/1M x 0.10  = $0.0660
writes   19,200 x $5/1M x 1.25  = $0.1200
output    4,800 x $25/1M        = $0.1200
                                  -------
                                  $0.3060 per task

The check line matters and you should always write one. Reads plus writes must equal the 151,200 input tokens computed in The cost identity, because caching changes the price of each token, never the number of them.

$0.876 → $0.306. That is a 2.9× cut on the task, for zero quality change.

On the input half alone it is 4.1×: the $0.756 of uncached input became $0.066 + $0.120 = $0.186.

Read the new bill, don’t celebrate it

Take the $0.306 apart:

Output is 39% of the bill and cache writes are another 39% — 78% that caching cannot touch. That is your signal that the next lever is shorter outputs and fewer turns, not more caching.

Re-derive this split after every optimization. It tells you what to do next.

Silent invalidators — the audit list

Every entry below changes a byte somewhere in the prefix. By the causal-attention argument above, everything after that byte stops being reusable. Read the right column as: how early is the changed byte, and therefore how much is destroyed.

The system-prompt row has a fix worth knowing: append a {"role": "system", ...} message to messages[] instead of editing the top-level system field. It sits after the cached prefix, so the history survives, and it still carries operator authority rather than reading as user text.

Diagnose in one line, then diff. If cache_read_input_tokens == 0 across repeated requests that should share a prefix, an invalidator is at work. Dump the rendered request bytes for two consecutive calls and diff them. The first differing byte is the bug.

That is a five-minute fix. The size of the win depends on your own N and your own prefix share, and the break-even arithmetic above already gives you the three reference points:

That spread is where the “3-10×” rule of thumb comes from. Quote your own number, not the range.

4. Model routing

Caching cut the price of the tokens you resend. The next lever cuts the price of the model serving them, because not every step needs the smartest model — extraction and classification are not planning. Send each step to the cheapest model that can do it, with one caveat that makes the naive version backfire.

The shape is one cheap classifier at the front and three destinations behind it. The router itself is a model call — it has a cost, and the first thing to check is whether that cost outweighs the saving.

flowchart TD
    T([Task]) --> C{"Router - Haiku<br/>~400 in, ~20 out"}
    C -->|"extraction, classification,<br/>formatting"| H["Haiku 4.5<br/>$1 / $5"]
    C -->|"standard reasoning,<br/>most tool calls"| S["Sonnet 5<br/>$3 / $15"]
    C -->|"planning, synthesis,<br/>hard debugging"| O["Opus 5<br/>$5 / $25"]

    style C fill:#bc6c25,color:#fff
    style H fill:#2d6a4f,color:#fff
    style S fill:#40916c,color:#fff
    style O fill:#95d5b2,color:#000

The router’s classification call reads about 400 tokens describing the task and emits about 20 naming a destination.

The three destinations:

• Haiku takes extraction, classification and formatting, because the answer is already present in the input and only has to be lifted out.
• Sonnet takes standard reasoning and most tool calls.
• Opus takes planning, synthesis and hard debugging — the work where the model has to hold a lot in mind at once.

What routing is worth, derived

Split the traffic into paths. A share s_i of tasks goes down a path that costs c_i. Add the router’s own cost, which every task pays:

cost = c_router + SUM s_i . c_i

Step 1: price the router

Do this first, because if the router is expensive the whole idea collapses. It is a ~400-token classification with ~20 tokens out, on Haiku at $1 in / $5 out per million:

400 x $1/1M  +  20 x $5/1M  =  $0.00040 + $0.00010 = $0.0005

That is five hundredths of a cent, or 0.16% of the $0.306 cached task from Prompt caching the highest leverage lever. The router is effectively free. Only the shares and the branch costs matter.

Step 2: price the cheap path

Take the reference agent and suppose 60% of tasks turn out to be simple lookups. Those resolve in 3 calls on Haiku, with the same prefix P = 6,000 and per-turn delta a = 1,200, and 800 output tokens each.

Same formulas as Prompt caching the highest leverage lever, just with n = 3 and Haiku prices. Total input is n.P + a.n(n-1)/2 with 3 × 2 / 2 = 3 turn-deltas. Call 1 writes the 6,000 prefix; calls 2 and 3 each write their 1,200 delta. Call 2 reads 6,000; call 3 reads 6,000 + 1,200 = 7,200. Output is 3 × 800 = 2,400:

simple path, cached (3 calls):
  total input = 3 x 6,000 + 1,200 x 3          = 21,600 tokens
  writes      = 6,000 (call 1) + 2 x 1,200     =  8,400 tokens @ 1.25x
  reads       = 6,000 (call 2) + 7,200 (call 3)= 13,200 tokens @ 0.10x
  check         8,400 + 13,200 = 21,600  OK

  writes   8,400 x $1/1M x 1.25 = $0.01050
  reads   13,200 x $1/1M x 0.10 = $0.00132
  output   2,400 x $5/1M        = $0.01200
                                  --------
                                  $0.0238

Step 3: blend

Now substitute into cost = c_router + SUM s_i . c_i. The shares are 60% cheap at $0.0238 and 40% hard at the $0.306 from Prompt caching the highest leverage lever:

blended = $0.0005 + 0.60 x $0.0238 + 0.40 x $0.3060
        = $0.0005 + $0.0143 + $0.1224
        = $0.1372 per task

$0.306 → $0.137, a further 2.2×. End to end: $0.876 → $0.137, a 6.4× reduction with no change to the hard path.

For a second data point: case study 06 runs the same arithmetic on a support workload and attributes 2.7× to routing alone on the model bill. That is smaller than you would guess, because routing’s real leverage there is that it enables the other two levers rather than that it beats them. Routing, model tiering and caching together are 16.3× on that workload.

The caveat interviewers reward: caches are model-scoped

The naive version of routing — “use the cheap model whenever the turn looks easy” — loses money. The reason is that a cached token and a fresh token are priced completely differently, so comparing models on list price compares the wrong thing.

Re-price every model by its effective input cost per million tokens — list price times the relevant cache multiplier — and the surprise is in the first two rows.

A warm Opus prefix is half the price of a cold Haiku prefix — $0.50 against $1.00 per million tokens, from $5 × 0.10 and $1 × 1.00.

Two terms, since everything below turns on them. Warm means the prefix is already in the cache and reads at 10%. Cold means nothing is cached and the whole prompt must be prefilled at full price.

Caches are model-scoped: the K and V vectors Opus computed are not valid for Haiku, because they came from different weights. So switching models mid-conversation throws the warm prefix away and re-prefills the entire history at the new model’s cold rate.

The arithmetic at turn 20

Work it at turn 20 of the reference agent. The accumulated context is 6,000 + 19 × 1,200 = 28,800 tokens.

The warm case splits that 28,800 in two. Only the 27,600 that existed at turn 19 are already in the cache and readable at 0.10×. The 1,200 tokens this turn appends have never been seen, so they are a fresh write at 1.25×.

The cold cases have no such split. Nothing is cached, so all 28,800 bill at full rate.

stay on Opus (warm):
  read  27,600 x $5/1M x 0.10 = $0.01380     <- 28,800 - 1,200 new
  write  1,200 x $5/1M x 1.25 = $0.00750     <- this turn's delta
  out      400 x $25/1M       = $0.01000   ->  $0.0313

switch to Sonnet (cold):
  in    28,800 x $3/1M        = $0.08640
  out      400 x $15/1M       = $0.00600   ->  $0.0924   <- 3.0x WORSE

switch to Haiku (cold):
  in    28,800 x $1/1M        = $0.02880
  out      400 x $5/1M        = $0.00200   ->  $0.0308   <- a wash, and you gave up capability

Downgrading to Sonnet mid-conversation costs 3× more than staying on Opus. The bigger model is cheaper, because it kept its cache.

Downgrading all the way to Haiku is a wash — $0.0308 against $0.0313, 1.6% cheaper — while losing two tiers of intelligence. Nobody takes that trade deliberately. It only looks attractive if you priced the cheap model at list and forgot the warm prefix you were throwing away.

And it gets sharper as the history grows, because the discarded prefix gets larger every turn.

The rule, and the splits that respect it

Route at task boundaries, not per turn. These four splits all obey that:

Every one of these gives the cheap model a fresh window — its own conversation, starting empty — which is why they work and “use Haiku for the easy turns” does not.

What interviewers probe: “Why not just downgrade the model when the turn is easy?” If you answer only “quality,” you missed the mechanism. Say: caches are model-scoped, a warm Opus read is $0.50/MTok against a cold Haiku prefill at $1.00/MTok, so on a long history the downgrade is a net loss before you even discuss accuracy — route at task boundaries or give the cheap model its own window.

5. Effort and thinking

Routing decides which model answers; a second dial decides how much the model reasons before answering. Turning it up can — counterintuitively — make a task cheaper.

That dial is effort, an API request parameter. It appears as output_config: {"effort": ...} in the call below, alongside model and max_tokens.

It is the intelligence/latency/cost dial, and it is independent of which model you chose — you set both.

The mechanism: effort moves the number of thinking tokens the model generates. Thinking tokens are internal reasoning the model produces before its visible answer, and they are billed as output tokens — the expensive half established in Prefill vs decode the fact underneath every cost rule. So raising effort raises the expensive half of your bill.

There are five levels:

Setting it looks like this, alongside adaptive thinking (which lets the model decide per request how much reasoning a question deserves):

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=16000,
    thinking={"type": "adaptive"},
    output_config={"effort": "medium"},
    messages=messages,
)

Two non-obvious points follow, both derivable from the identity in The cost identity.

Higher effort can be cheaper end-to-end

This is the counterintuitive one. Effort raises t_bar, because thinking tokens are extra tokens per call. But it can lower n, because a model that plans better takes fewer steps and retries less — and n is the factor with the quadratic coupling.

The mistake to avoid is comparing a percentage against a price. Put both sides in dollars.

What you save by going from 12 calls to 8 on the reference agent. That removes 151,200 - 81,600 = 69,600 input tokens:

69,600 x $5/1M  =  $0.348 saved

What you spend to get it: say 300 extra thinking tokens on each of the 8 surviving calls, billed at the output rate:

8 x 300 x $25/1M  =  2,400 x $25/1M  =  $0.06 spent

$0.348 / $0.06 is a ~6:1 win for the more expensive setting.

Do that subtraction before assuming low effort is cheap. Measure cost per completed task, never per call.

Sweep it

Effort defaults carried over from another model are almost never right. Run low, medium and high across your evaluation set — the fixed collection of tasks with known-good answers that you score changes against — and pick a level per route.

6. Latency

A cheaper agent is not automatically a faster one: latency and cost respond to different levers. Finding the levers that move the clock takes one decomposition — a request’s wall-clock time splits into two measurable parts — and one observation: the outer loop dominates both.

The three stages of a request

flowchart LR
    U([User]) --> Q["Queue<br/>rate limits, admission<br/>sets TTFT"]
    Q --> P["Prefill<br/>~ input tokens<br/>sets TTFT"]
    P --> D["Decode<br/>~ output tokens<br/>sets TPOT"]
    D --> R([Response])

    style P fill:#2d6a4f,color:#fff
    style D fill:#bc6c25,color:#fff

A request passes through three stages before the user sees a Response:

1. It waits in a queue governed by rate limits and admission control.
2. It is prefilled, which takes time proportional to the input tokens.
3. It is decoded, which takes time proportional to the output tokens.

Two metrics divide that up. TTFT is time to first token: how long the user stares at nothing. TPOT is time per output token: how fast text appears once it starts.

The queue sets TTFT, and so does prefill. Those two stages are additive — TTFT is queue time plus prefill time, not one or the other. Decode sets TPOT.

The two metrics respond to different levers

TTFT (time to first token)  = queue + prefill(input_tokens)      <- prefill-dominated
TPOT (time per output token)= one decode step                    <- decode-dominated
total  =  TTFT  +  (output_tokens - 1) x TPOT

The - 1 in the last line is not a typo. The first output token arrives at TTFT by definition, so only the remaining output_tokens - 1 cost a TPOT each.

Work it on a concrete call: 40,000 input tokens, 800 output tokens, prefill at ~20,000 tokens per second, decode at ~50 tokens per second (the loaded-endpoint figures from Prefill vs decode the fact underneath every cost rule).

One simplification to flag before the arithmetic: the first line prices TTFT as prefill alone, which quietly assumes the request never waited in a queue. The last row of the table below is the only place that assumption matters.

TTFT  = 40,000 / 20,000            = 2.0 s
TPOT  = 1 / 50                     = 20 ms
total = 2.0 + 799 x 0.020          ~ 18.0 s     (decode is 89% of it)

Decode’s share is 15.98 / 17.98 = 89%. Two seconds of prefill, sixteen seconds of decode.

Which levers actually move the 18 seconds

Apply one lever at a time. The right column is what matters: the two levers that touch decode move the clock, and the two that touch input barely register.

Why the last row looks strange

The oddity is in the budget, not the lever. The 18.0 s above computes TTFT as 40,000 / 20,000 — pure prefill — which quietly sets the queue to zero, even though this section has just defined TTFT as queue plus prefill.

So there is no queue term in that total for a faster network to shrink. The row instead prices a deployment that does carry a queue, at 0.1 s: it is the one lever in the table that adds a term to the budget and then removes it again, rather than shrinking a term already there. Total goes 18.0 → 18.1 → 18.0.

Its 0.1 s is an assumption about your network, not a figure derived on this page.

Why the cache row only buys 9%

The first row is the one that disappoints people, so read it slowly.

Caching 36,000 of the 40,000 input tokens does not make prefill free. The cached keys and values still have to be streamed from memory. They just move far faster than recomputing them would.

Loading a cached K/V pair runs roughly 10× faster than computing it — the same order as the 10% you are billed for it. So price the two halves of the prompt at their two different rates: 4,000 fresh tokens at 20,000 tok/s, and 36,000 cached ones at 200,000 tok/s.

fresh prefill    4,000 / 20,000 tok/s   = 0.20 s
cached K/V load 36,000 / 200,000 tok/s  = 0.18 s
                                          ------
new TTFT                                  0.38 s  ~  0.4 s

So TTFT falls from 2.0 s to 0.4 s rather than to zero. That is a 5× cut on a term that was only 11% of the wall clock to begin with — hence the 9% total.

The 200,000 tok/s cache-load rate is the one assumed input here rather than a derived one. Treat it as an order-of-magnitude figure. Note that even setting it to infinity only takes TTFT to 0.20 s and the total to 16.2 s, which does not change the conclusion.

Caching is the cost lever; output length is the latency lever. They are not the same lever, and conflating them is why people report “we added caching and it didn’t feel faster.” It didn’t.

Put both effects of caching on this one call side by side:

• Wall clock: -9% (18.0 s → 16.4 s)
• Bill: -74% ($0.22 → $0.058, working below)

uncached  input   40,000 x $5/1M         = $0.200
          output     800 x $25/1M        = $0.020   ->  $0.220

cached    read    36,000 x $5/1M x 0.10  = $0.018
          fresh    4,000 x $5/1M         = $0.020
          output     800 x $25/1M        = $0.020   ->  $0.058

A very good trade, and a very bad demo.

Where agent time actually goes

Zoom out from one call to a whole run and the picture changes again.

One entry in the table needs a name explained first: programmatic tool calling is where the model writes a short script that calls several tools itself, so the whole sequence runs in one round trip instead of one round trip per tool.

Ranked by contribution to an agent run’s wall clock, the top entry is not any single call — it is the number of calls.

Put the reference agent’s 12 calls at 18 seconds each and you get 12 × 18 = 216 seconds — three and a half minutes.

The outer loop dominates everything inside it. That is why “reduce round trips” outranks every per-call optimization on latency, exactly as it does on cost.

Three practical wins

1. Stream. Streaming sends tokens to the client as they are generated instead of waiting for the complete response.

   It does not reduce total time. It reduces perceived time, because the user sees output after TTFT (2 s) instead of after TTFT + 16 s.

   It also avoids HTTP timeouts on large max_tokens values, and is required above roughly 16,000 tokens of max_tokens. A non-streamed request holds the connection open for the entire decode, and 16,000 / 50 tok/s = 320 seconds — over five minutes of silence on the wire.

2. Emit all parallel tool calls in one assistant turn, and return all results in ONE user message. This is the one people get wrong, and the failure is silent.

   The two snippets differ in one place: where the for loop sits. In the wrong version it wraps the messages.append, producing one message per tool. In the right version it sits inside the content list, producing one message holding every result.

   # WRONG - three user messages, one result each
   for tu in tool_uses:
       messages.append({"role": "user", "content": [
           {"type": "tool_result", "tool_use_id": tu.id, "content": run(tu)}]})
   
   # RIGHT - one user message, all results
   messages.append({"role": "user", "content": [
       {"type": "tool_result", "tool_use_id": tu.id, "content": run(tu)}
       for tu in tool_uses]})

   Why it matters mechanically. The model is a next-token predictor conditioned on the transcript (The forward pass), and the transcript is its strongest available evidence about what shape a turn should take.

   So look at what the wrong version writes into that transcript: every assistant turn holds one tool_use, and every user turn after it holds one tool_result. That pattern is a demonstration that turns contain one call.

   Few-shot conditioning — the model copying the pattern it sees in its own context — does the rest, and the model stops batching.

   Nothing errors. Your p95 latency (the time the slowest 5% of requests take) doubles and the trace looks normal. It also costs money: one extra round trip per tool means one extra full-history prefill each.

3. Prefetch the obvious. If 90% of sessions start with the same lookup, fire it before the model asks. That converts a serial TTFT + tool into a parallel one.

7. Streaming with everything on

In production the levers do not get pulled one at a time. Below is a call with streaming, caching, thinking and usage reporting all switched on at once — the shape to copy.

Four things to spot in the code: messages.stream instead of messages.create (that is the streaming switch), the cache_control marker on the system block from Prompt caching the highest leverage lever, the effort setting from Effort and thinking, and the usage read at the bottom that feeds the accounting in Cost accounting.

with client.messages.stream(
    model="claude-opus-5",
    max_tokens=64000,
    system=[{"type": "text", "text": SYSTEM,
             "cache_control": {"type": "ephemeral"}}],
    thinking={"type": "adaptive", "display": "summarized"},
    output_config={"effort": "high"},
    tools=TOOLS,
    messages=messages,
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()

u = final.usage
print(f"\nin={u.input_tokens} out={u.output_tokens} "
      f"cache_read={u.cache_read_input_tokens} stop={final.stop_reason}")

One line there is a user-experience decision rather than a cost one: display: "summarized", which controls whether the model’s internal reasoning is surfaced.

The default omits thinking text. On a high-effort call that means the user gets TTFT, then a long silent gap while thinking tokens decode, then output — and streaming does not help, because there is nothing to stream during the gap. Summarized thinking fills it with something true.

8. Batch and semantic caching

Two cost levers live outside the request path entirely. One is a pure win if you can tolerate delay; the other is far more dangerous than it looks.

The Batch API

The Batch API takes work you submit now and returns it within 24 hours, at 50% off.

It is right for anything not user-facing: overnight evaluation runs, bulk classification, backfills, dataset generation. It is wrong for anything interactive.

In the terms of The cost identity, it halves p_bar, the price factor. Every token bills at half rate, so the task costs half as much. On the reference task that is $0.306 → $0.153.

Quality cost is zero, which is rare. The entire cost is paid in latency — which makes it a product decision rather than an engineering one.

Semantic caching

A semantic cache is a different animal from the prompt cache in Prompt caching the highest leverage lever. Prompt caching reuses computation for byte-identical text and can never change an answer. A semantic cache reuses an answer for a question that merely resembles an old one — and that can absolutely change the answer.

“Close enough” is measured two ways at once. Embeddings are vectors of numbers positioned so that texts with similar meanings sit near each other; a similarity threshold is the cutoff score you pick, above which you call two questions the same.

The implementation is four lines. Everything that follows is an argument about one number in it — the threshold default:

from __future__ import annotations   # so `str | None` works on Python 3.9

def cached_answer(q: str, threshold: float = 0.97) -> str | None:
    hit = vector_cache.search(embed(q), k=1)
    if hit and hit.score >= threshold and not hit.stale:
        return hit.answer
    return None

Why the threshold must be set by the cost of being wrong

Lowering the threshold catches more questions and saves more calls, so the temptation is to tune it by hit rate. That is the wrong objective, because the two ways of being wrong are not symmetric — and the asymmetry is enormous.

Write the expected loss as a function of the threshold t:

expected loss(t) = P_miss(t) x c_call  +  P_false_hit(t) x c_wrong

A miss is a question the cache should have answered and didn’t. A false hit is a question the cache answered with a stored answer that was not actually right for it.

Now price the two costs.

c_call: what a miss costs you

A miss costs you the LLM call you were trying to avoid. Not the 12-call reference agent from The cost identity — a semantic cache displaces a single answer, so the thing you save is one call.

Price that one call: a cached FAQ answer on Haiku at $1 in / $5 out per MTok, with a 2,000-token cached prefix, 500 fresh tokens of question and retrieved passage, and 450 tokens out.

cached prefix  2,000 x $1/1M x 0.10 = $0.00020
fresh input      500 x $1/1M        = $0.00050
output           450 x $5/1M        = $0.00225
                                      --------
                                      $0.00295  ~  $0.003

c_wrong: what a false hit costs you

A false hit ships a confidently wrong answer to a user. In support, an answer that deflects the ticket — closes it without a human ever seeing it — but is wrong costs you a re-contact, an escalation, and some trust.

Call it $8. That $8 is an assumption about your business, not a derived figure. The argument below only needs it to be three orders of magnitude above c_call, which almost any support economics will give you.

The ratio, and what it does to the tuning

c_wrong / c_call  =  $8 / $0.003  =  ~2,700 : 1

One false hit undoes 2,700 saved calls. Run both threshold settings over 10,000 queries and that ratio does the rest.

The four rates below — hit rates of 40% and 18%, false-hit rates of 3.0% and 0.1% — are illustrative shapes, not measured values. They are an assumption in the same way the $8 is. Everything downstream reproduces exactly from them, but yours will differ.

threshold 0.92:  hit rate 40%  ->  4,000 hits, 3.0% of them false = 120 wrong
    saved   4,000 x $0.003 = $12.00
    damage    120 x $8     = $960.00        net  -$948

threshold 0.97:  hit rate 18%  ->  1,800 hits, 0.1% of them false = 1.8 wrong
    saved   1,800 x $0.003 = $5.40
    damage    1.8 x $8     = $14.40         net  -$9

Loosening the threshold by 5 points turned a $9 rounding error into a $948 loss — a 105× swing ($948 / $9), off a change that looks like tuning.

Note also that even the tight setting does not break even. It is $9 in the red on cost alone, before you count the trust.

The property that actually carries the argument

The argument does not need those four numbers. It needs one property of them.

As the threshold drops, the false-hit rate has to rise faster than the hit rate. Here the hit rate roughly doubles (18% → 40%) while the false-hit rate goes up thirtyfold (0.1% → 3.0%).

That is the shape you should expect, and there is a reason for it: loosening a similarity threshold admits questions in order of decreasing similarity, so the marginal admission is always the least similar one yet — the one most likely to be a false hit.

Any pair of curves with that shape, against a c_wrong / c_call ratio anywhere near 2,700:1, produces the same conclusion. Measure your own two rates before quoting a swing; measure them again before quoting 105×.

Two conclusions

Both are worth saying out loud in an interview.

1. Tune the threshold against c_wrong, not against hit rate. A dashboard showing “cache hit rate up to 40%” is reporting the wrong number.

2. Semantic caching is a latency feature that happens to save a little money — not a cost lever.

   Compare the two caches directly on this chapter’s own numbers. Prompt caching took the reference task from $0.876 to $0.306: a derived $0.570 saved per task at zero risk. The tight semantic-cache setting above nets -$9 across 10,000 queries and carries a tail of wrong answers.

   If someone proposes semantic caching as a cost fix, prompt caching in Prompt caching the highest leverage lever is the larger win by any accounting, and the only one of the two with no failure mode.

In practice: start at 0.95 or above, exclude anything user-specific or time-sensitive, give entries a TTL, and log every hit with its similarity score so you can audit the tail.

9. Rate limits and resilience

Sooner or later the API says no. The engineering question is how to retry without turning one spike into a self-sustaining one.

Every design decision in the retry path lives on the yes branch out of the 429 diamond.

flowchart TD
    R([Request]) --> A{429?}
    A -->|no| OK([Response])
    A -->|yes| B["Read retry-after"]
    B --> C["Backoff + jitter"]
    C --> D{"Attempts < N?"}
    D -->|yes| R
    D -->|no| F{Fallback?}
    F -->|yes| G["Smaller model or<br/>degraded mode"]
    F -->|no| E([Fail with context])

    style G fill:#bc6c25,color:#fff

Walk it. A request either succeeds or comes back 429, the HTTP status code meaning “too many requests.”

On a 429 you read retry-after — the header in which the server tells you how long to wait — then apply backoff and jitter before trying again:

• Backoff means waiting longer after each successive failure. Typically each wait doubles.
• Jitter means adding a random offset to that wait.

If attempts are still under your limit N, retry. Once they are exhausted, you take one of two exits: a fallback path (a smaller model or a degraded mode), or fail with context — surface an error that says what was being attempted, rather than a bare exception.

The SDK — the official client library — already retries status codes 408, 409, 429 and 5xx with backoff, two attempts by default (max_retries=2). Add three things on top of it:

• Jitter. Without it, N workers that all hit a 429 at the same instant back off by the same deterministic amount, wake in the same instant, and reproduce the spike exactly. Jitter is what breaks the synchronization.
• A graceful degradation path — a smaller model, a cached answer, or an honest “busy, try again.” The Model routing caveat applies here too: a fallback model starts cold, so budget the full cold prefill, not the warm read you are used to.
• Awareness that timeouts multiply. Worst-case wall clock per call is timeout × (max_retries + 1). With a 60-second timeout and 2 retries that is 60 × 3 = 180 seconds — for one call. The reference agent makes 12 of them, so the worst case for the task is 36 minutes.

One last thing to check before you move traffic: different model tiers draw on separate rate-limit pools, so shifting traffic between them does not inherit headroom. Check the target tier’s limits before you migrate volume.

10. Cost accounting

None of the derivations above survive contact with production unless you can measure them: four billing fields, one correct way to combine them, and two ratios that tell you what to fix next.

Instrument from day one. You cannot retrofit this after the bill arrives.

The code below is two functions. cost_usd prices one call from its four billing fields; task_report rolls a whole task up and computes two ratios. The comments in it are the load-bearing part — read them, not just the arithmetic.

PRICE = {
    "claude-opus-5":   {"in": 5e-6,  "out": 25e-6},
    "claude-sonnet-5": {"in": 3e-6,  "out": 15e-6},
    "claude-haiku-4-5":{"in": 1e-6,  "out": 5e-6},
}

def cost_usd(model: str, u) -> float:
    """Four line items. Omitting the cache terms misprices every cached call:
    on this chapter's reference task, dropping them reports $0.12 against a
    true $0.306. How far off you are depends entirely on your read/write mix,
    so compute it, don't quote a multiplier."""
    p = PRICE[model]
    return (u.input_tokens * p["in"]
            + u.output_tokens * p["out"]
            + (u.cache_creation_input_tokens or 0) * p["in"] * 1.25
            + (u.cache_read_input_tokens or 0) * p["in"] * 0.10)

def task_report(calls: list) -> dict:
    """Roll up per task, and keep the ratios that say what to fix next."""
    total = sum(cost_usd(c.model, c.usage) for c in calls)
    reads  = sum(c.usage.cache_read_input_tokens or 0 for c in calls)
    writes = sum(c.usage.cache_creation_input_tokens or 0 for c in calls)
    fresh  = sum(c.usage.input_tokens for c in calls)
    out_cost = sum(c.usage.output_tokens * PRICE[c.model]["out"] for c in calls)
    return {
        "cost_per_task": total,
        "calls": len(calls),
        # Writes belong in the denominator: a written token is a token that
        # did NOT hit the cache, and it bills at 1.25x -- the most expensive
        # input there is. Omit them and this chapter's own reference task
        # reports 100% (132,000 / 132,000) while 19,200 tokens were written
        # at a premium. With writes: 132,000 / 151,200 = 87%.
        "cache_hit_rate": reads / max(reads + writes + fresh, 1),  # low -> section 3
        "output_share": out_cost / max(total, 1e-12),              # > 0.35 -> be terser
    }

The subtlety: the four usage fields are disjoint

input_tokens excludes the cached tokens. The four fields — input_tokens, output_tokens, cache_creation_input_tokens, cache_read_input_tokens — do not overlap, which is why cost_usd simply adds all four rather than adjusting one by another.

Getting this wrong is the most common cost-dashboard bug, and it fails in both directions:

• Double-count (treat input_tokens as the total and add the cache fields on top) and your spend looks worse than it is.
• Drop the cache terms and it looks better. On this chapter’s reference task, where every input token is either a read or a write, input_tokens is 0 — so a dashboard that ignores the cache fields reports $0.12 against a true $0.306.

What to report

Report cost per completed task, not cost per call. An agent making 20 cheap calls to fail is worse than one making 5 expensive calls to succeed, and per-call dashboards hide that completely.

Track these four alongside it. The last row is the one nobody has:

11. Optimization order

Every lever is now on the table, so the remaining question is sequence — and the sequence is forced rather than a matter of taste.

The labels on the arrows are the argument. Each one is the reason its step must finish before the next begins.

flowchart TD
    M["1. Measure"] -->|"no denominator, no ratio"| C["2. Fix caching"]
    C -->|"repriced every model"| N["3. Cut calls per task"]
    N -->|"outer factor, quadratic"| T["4. Cut tokens per call"]
    T -->|"pattern is now final"| R["5. Route models"]
    R -->|"first quality trade"| E["6. Tune effort"]
    E -->|"last: costs latency"| B["7. Batch what isn't interactive"]

    style C fill:#2d6a4f,color:#fff
    style N fill:#2d6a4f,color:#fff
    style R fill:#bc6c25,color:#fff

In words:

• Measure first, because with no denominator, no ratio is computable.
• Fix caching next, because it reprices every model — so any routing decision made before it is made on prices that are wrong.
• Cut calls before tokens, because calls are the outer factor and quadratic in their effect.
• Cut tokens only once the pattern is final.
• Route models fifth, because it is the first quality trade in the list.
• Batch last, because it costs latency — a product property rather than an engineering one.

Steps 2 and 3 are the free wins: caching costs no quality at all, and cutting calls changes the pattern rather than the answer. Step 5 is where the character of the list changes — it is the first step that trades quality for cost.

Do these in order. Stop when it is cheap enough. The order is not preference — each step changes the inputs to the next one.

The table below is the same list with the full justification per step. The right column is the one to be able to say out loud.

Do not start at step 5

Downgrading the model on an uncached, chatty agent costs you accuracy, and on a long history it can cost you money too — Model routing worked a mid-conversation Sonnet downgrade out at 3× worse.

Fixing caching first costs nothing and, on this chapter’s reference task, is a derived 2.9× (Prompt caching the highest leverage lever).

One number to be careful with: the 10× you will hear quoted for caching is the asymptotic ceiling on the cached prefix alone, not what a whole task moves by. Output tokens and cache writes are untouchable, and on the reference task they are 78% of the post-caching bill.

Lead with the 2.9×. It is the one this chapter derives, and an interviewer can check it.

What interviewers probe: “Your agent costs 10× the estimate. What do you do?” Weak: “switch to a cheaper model.” Strong: “First I’d look at cache_read_input_tokens — if it’s zero there’s a silent invalidator and that’s a 5-10× fix for free. Then calls per task, because history is resent so calls are coupled quadratically to tokens. Model routing is fifth, because it’s the first lever that trades quality, and because switching models throws away a warm prefix.”

Cheat sheet

Each row pairs a symptom you will actually observe with the mechanism that produces it and the first thing to check.

Next: 10 — Design Interview Playbook — the method for the 45-minute round.