08 — Evaluation

A large language model (LLM) is a model trained to predict the next piece of text. Everything it produces is a probability, not a certainty, and that fact drives most of this chapter.

An agent is a program built on top of one. It hands a task to the model, gives the model a list of functions it is allowed to call, runs whichever calls the model asks for, feeds the results back, and repeats until the task is finished.

Evaluating an agent means replacing “it seems to work” with a number you can defend.

This chapter covers four questions:

• Why no two runs of the same agent are identical, and what that forces on your test design.
• What you may safely assert on, and what you must never assert on.
• How to grade an answer that has no single correct wording.
• How to find which stage of a broken pipeline is actually the broken one.

What goes in and what comes out

An evaluation case goes in as a frozen input plus a written-down expectation; a verdict comes back out, together with the full record of what the agent did on the way.

Frozen means the input is stored verbatim and never edited afterwards. That matters for a specific reason: when a score moves, the input is the one thing you know did not, so the cause has to be somewhere else.

Here is one such case, written for a customer-support agent that handles refunds. Read it as four fields: the text the user sent, a plain-English description of what a good answer looks like, and two hard rules about which functions the agent must and must not call.

INPUT — one eval case, written once and stored alongside the code

  input:            "I want a refund for ORD-882, bought 45 days ago"
  expected outcome: declines the refund, cites the 30-day policy,
                    offers store credit instead
  must call:        lookup_order
  must never call:  issue_refund

Running the agent against that case produces two separate things: answer is a single block of prose, while trace is a numbered list of function calls. The distinction between them runs through the whole chapter.

OUTPUT — what one run produces

  answer:  "ORD-882 was placed 45 days ago, past our 30-day refund window,
            so I can't refund it — but I've put $29.99 on your account as
            store credit instead."

  trace:   lookup_order(order_id="ORD-882")  ->  {placed_days_ago: 45, ...}
           check_policy(kind="refund")       ->  {window_days: 30}
           issue_credit(cents=2999)          ->  {credit_cents: 2999}

  verdict: PASS  (answer matched the expectation; lookup_order was called;
                  issue_refund was never called)

The answer is what the user sees. The trace is the record of every step the agent took to produce it — which functions it called, with what arguments, and what came back.

One notation detail that recurs: money is counted in cents throughout. The 2999 in the trace is the $29.99 in the answer, and every assertion later in this chapter that mentions 2999 is asserting on that same store credit.

Everything in this chapter is detail on four questions: which cases you write, who decides PASS or FAIL, what the whole suite costs to run, and how much of any given verdict is noise rather than signal.

The vocabulary, defined once

Ten terms recur throughout this chapter.

• A tool is one of the functions the agent is allowed to call — lookup_order above. A tool call is one invocation of it. The model does not run the function; it emits a request to run it, your code runs it, and the result is handed back to the model as more text.
• A token is the unit models read and write text in, roughly four characters of English. Cost and latency are both counted in tokens, so they show up in every piece of arithmetic here.
• A prompt is the text you send the model. The system prompt is the standing instruction block placed at the front of every request; the context window is the total amount of text the model can hold at once, which is bounded, so something has to be dropped when you exceed it.
• A trace is the stored record of one complete run: every message, every tool call, every result. The trajectory is the path through that trace — the sequence of steps, as opposed to the final answer.
• An eval case is one frozen input plus its expectation. An eval suite is a collection of them run together.
• A grader is whatever decides PASS or FAIL for a case: a line of code, a similarity check, another model, or a human.
• A flake is a case that passes on one run and fails on the next with nothing changed. The flake rate is how often that happens.
• CI stands for continuous integration: the automated system that runs your tests on every proposed code change. A PR, or pull request, is one such proposed change. A gate is a rule in CI that blocks the merge when a condition fails.
• A regression is a change that makes something that used to work stop working. A baseline is the score you are comparing against — usually the score of the code already shipped.
• Latency is elapsed wall-clock time from request to answer.

The one structural idea to carry through. The answer to “how do you know it works?” has to start one level lower than most people start it, because the system under test is not deterministic and cannot be made deterministic — deterministic meaning that the same input always produces exactly the same output. Every design choice in this chapter follows from that single fact: running each case three times and taking the majority verdict, asserting loosely on trajectories, gating on suite-wide averages instead of individual cases, and measuring a model-based grader against human labels before trusting it. Start there.

1. Why nothing here is reproducible

Run the same agent on the same input twice and you get two different answers. That cannot be switched off, and it dictates most of your suite design — including exactly how many times to run each case, and why the answer is three.

The cause is a chain that starts with a scheduling decision made by hardware you do not control and ends with two completely different answers.

flowchart TD
    B["Your request is batched with<br/>other users' requests"] --> K["GPU kernel picks a reduction<br/>order based on batch shape"]
    K --> F["Float addition is not associative:<br/>(a+b)+c != a+(b+c) in the last bits"]
    F --> L["Two near-tied logits swap<br/>rank on a rounding difference"]
    L --> T["A different token is sampled"]
    T --> D["Every token after it is conditioned<br/>on a different prefix -> full divergence"]

    style F fill:#bc6c25,color:#fff
    style D fill:#9d0208,color:#fff

Each link in that chain is a fact about hardware rather than a choice anyone made.

Link 1: your request is not computed alone. The model’s arithmetic runs on a GPU — a graphics processing unit, the specialized and expensive chip built for exactly that kind of arithmetic. Keeping one busy means never running a single request on its own. Your request is batched with other users’ requests: grouped and computed together, dozens at a time. Which requests land in your group changes millisecond to millisecond, depending on who happened to hit the server when you did.

Link 2: the batch shape decides the order of additions. The GPU executes the work as a kernel, a small program compiled for that chip. The kernel performs reductions — a reduction is the summing-up of many partial products into one number. There are many valid orders to add up the same list of numbers, and the kernel picks one based on how the batch happens to be shaped, because different shapes make different orders fastest.

Link 3: order changes the answer, slightly. Computers store real numbers in a fixed number of bits, so every addition rounds off whatever does not fit. That makes float addition non-associative: (a+b)+c != a+(b+c) in the last bits. Concretely, with three-digit decimal rounding, (1.00 + 0.005) + 0.005 = 1.00 but 1.00 + (0.005 + 0.005) = 1.01. Same numbers, different grouping, different answer.

Link 4: those last bits land on the logits. A logit is the model’s raw, unnormalized score for one possible next token — one number per candidate word-piece, highest score meaning most preferred. When the top two candidates are almost tied, a rounding difference in the last bits is enough to swap their rank. A different token then gets sampled, meaning drawn according to those scores.

Link 5: one flipped token is not a local error. The model produces each new token by re-reading the prefix — everything written so far. Flip one token and every later token is now conditioned on a different prefix, so the two runs keep drifting apart. Within a sentence or two the answers have nothing in common.

Why temperature=0 does not fix it

The setting people reach for first is temperature, the knob that controls how randomly the model picks among high-scoring candidates. At temperature=0 it is supposed to always take the single highest-scoring token, which sounds like it should make runs identical.

It does not (Sampling and why temperature0 isnt deterministic). Temperature governs the choice among logits. The divergence described above happens inside the logits, before temperature is consulted at all — the scores themselves came out slightly different, so “always take the highest” picks a different winner.

Nothing about that is a bug you can file. It is a property of running float math on a shared GPU whose batches are re-formed on the fly, and the variable that decides it — who else’s request landed in your batch — is not under your control and is not in your prompt.

What that forces on your suite

Three consequences shape the rest of the chapter.

1. Exact-string assertions measure the scheduler, not your change. An eval that asserts the output text equals some fixed string is really asserting that today’s batch composition matched yesterday’s, so it has a nonzero flake rate forever.
2. Any single run is a sample, not a measurement. Reporting “the new prompt scored 87%” from one pass over 30 cases is reporting one draw from a distribution, and the next draw will differ.
3. Gating on individual cases is gating on noise. The one exception is safety cases, where the cost of shipping a violation is so much worse than the cost of a false alarm that you accept the flakes.

The flake arithmetic, and why N=3

The standard fix is to run each case three times and take the verdict that appears at least twice, written N=3 and called majority-of-3. “Run it three times” is a claim about spending three times the money, so what it buys should be derived, not asserted.

Suppose a case passes with probability p on any given run. Majority-of-3 passes when at least two of the three runs pass, which is:

P_maj(p) = p^3 + 3 p^2 (1 - p)

The first term is all three runs passing. The second is exactly two of three passing: p^2 (1 - p) is the probability of one specific run being the failure, and the 3 counts the three ways to choose which run that is.

Substitute one value all the way through before trusting the table. At p = 0.9:

p^3           = 0.9 x 0.9 x 0.9              = 0.729
3 p^2 (1 - p) = 3 x 0.81 x 0.1               = 0.243
P_maj(0.9)    = 0.729 + 0.243                = 0.972

So a case that passes 90% of the time on a single run passes 97.2% of the time under majority-of-3. Here is the same calculation across a range of p, and the direction of the gap between the single-run and majority columns flips at 0.5.

Majority voting polarizes away from 0.5. It pushes every case towards the nearer of 0 and 1, so reliable cases look reliable and unreliable cases look unreliable.

Exactly at p = 0.5 it changes nothing, which is what the fixed point row records: 0.5 is the one value voting cannot move, not the value it moves things towards. Above 0.5 it pushes up; below, it pushes down.

That downward push matters as much as the upward one, because it means voting does not launder a broken case into a passing one: a case that passes 20% of the time passes majority-of-3 only 10.4% of the time.

Note that this is majority of 3, not best of 3. Best-of-3, where a single passing run is enough, would take that same 20% case up to 1 - 0.8^3 = 0.488 — a machine for turning broken cases green. If someone proposes “just retry until it passes”, that is the arithmetic to quote back.

Why 45% false-red kills a suite

Now the number that actually justifies the 3x spend. Take a 30-case suite where every case independently flakes 2% of the time, so each case passes with p = 0.98. The suite goes green only if all 30 cases go green at once, which is p multiplied by itself 30 times:

single run:   P(all 30 green) = 0.98^30    = 0.545
              P(suite goes red) = 1 - 0.545 = 0.455   ->  45% of clean PRs go red

N=3 majority: per-case pass rate rises 0.98 -> 0.9988
              P(all 30 green) = 0.9988^30  = 0.965
              P(suite goes red) = 1 - 0.965 = 0.035   ->  3.5% of clean PRs go red

A “clean PR” there is a proposed change that broke nothing; going red means CI blocked it anyway.

45% false-red is a suite people start ignoring within two weeks — nearly one in two of their innocent changes gets blocked, so they learn to re-run until it passes and stop reading the output. 3.5% is a suite people trust. That is what the 3x compute buys, and it is the honest way to justify it in an interview.

What N=3 does not buy is any protection against systematic error. Three runs of a prompt that misreads the question misread it identically. The distinction is variance versus bias: variance is the run-to-run scatter around the model’s average behaviour, bias is where that average sits, and repetition averages away scatter without moving the average — the same distinction that governs voting in Parallelization. Repetition reduces sampling variance only.

What interviewers probe: “Can’t you just set temperature to 0?” No — and the reason is batched float non-associativity, not sampling. On several current frontier models the knob is not even reachable: Anthropic’s Claude Opus 5 and Sonnet 5 removed temperature, top_p, and top_k outright, and a request that sets any of them is rejected with a 400 rather than quietly honoured. If you need stability, cache by input hash — store the answer under a key derived from the exact input and return the stored copy on a repeat; if you need a test, assert on a property, never on a string.

2. Outcome vs. trajectory

Nondeterminism settled how many times to run each case; the next question is what to look at in each run. There are two candidates — what came out, and how it got there — and they get scored very differently, because outcomes are the contract you owe the user while paths are an implementation detail.

Side by side on the same input, the two evals ask different questions: one about the output, three about the steps.

flowchart LR
    subgraph O["Outcome eval — did it succeed?"]
        I1([Input]) --> A1[Agent] --> R1([Output]) --> C1{Matches<br/>expectation?}
    end
    subgraph T["Trajectory eval — how did it get there?"]
        I2([Input]) --> A2[Agent] --> S["ordered steps taken:<br/>look up · check policy · issue credit"]
        S --> C2{Right tools?<br/>Efficient?<br/>Safe?}
    end

    style O fill:#2d6a4f,color:#fff
    style T fill:#40916c,color:#fff

An outcome eval asks one question — did it succeed? — by taking the final output and checking whether it matches expectation.

A trajectory eval asks how it got there. It ignores the answer and inspects the ordered steps taken, which for the refund case were to look up the order, check the policy, and issue the credit. Against that path it asks whether the agent used the right tools, whether it was efficient, and whether it was safe.

They differ on four axes, and the last row is the one that decides how you use them:

Score outcome strictly; score trajectory loosely.

Why trajectory evals must be loose — the mechanism

This rule follows from a mechanism, not from taste.

An over-specified trajectory eval asserts on a sequence that is (a) a sample from a distribution (Why nothing here is reproducible) and (b) an implementation detail of the current tool set. Both properties mean it goes red for reasons that have nothing to do with correctness.

Six ordinary changes would each turn such an eval red. The eval fires whenever the path changes, while the thing you care about is whether correctness changed.

Five of the six red signals are false. Only the last row — the agent calling the destructive tool — is a red light you would want.

An alarm that usually fires because something got better has terrible positive predictive value: the share of alarms that turn out to be real problems, here 1/6 = 0.167. Put plainly, a red light from this eval is, on the balance of probabilities, evidence that nothing is wrong.

Engineers learn that within a few weeks, start ignoring the eval, then delete it — and the one signal in that table that mattered goes with it. That is the real cost of over-specification: not brittleness in the abstract, but the eventual loss of the safety assertion that was buried inside it.

The fix is to assert on invariants — statements that must hold of any acceptable path — instead of on one particular path. Compare the first line below, which pins the exact sequence, with the four that follow, none of which mention order at all:

# OVER-SPECIFIED — asserts an implementation detail
assert trace.tools == ["lookup_order", "check_policy", "issue_credit"]

# INVARIANT — asserts what must be true of any acceptable path
assert "issue_refund" not in trace.tools          # safety: binary, blocking
assert "lookup_order" in trace.tools              # grounding: it must have looked
assert len(trace.tools) <= 6                      # efficiency budget, not a path
assert trace.final_state.credit_cents == 2999     # outcome, scored strictly
                                                  # 2999 = the $29.99 store credit, in cents

Generalizing from that block, five trajectory metrics are useful without over-constraining the path:

1. Tool-call count — measures efficiency, and stands in for cost.
2. Was a required tool ever called — measures grounding, meaning the agent actually consulted a source rather than inventing the answer.
3. Was a forbidden tool ever called — measures safety. This one is binary and non-negotiable.
4. Retry and error rates — measure how much work is being wasted.
5. Time and dollars to completion — measure what the run costs.

Notice the shape all five share: they are set membership and counts, never sequences. Sequences encode the path; membership encodes the requirement.

3. The eval pyramid

Knowing what to assert still leaves the question of where each check belongs. Most teams write every assertion at the most expensive layer. Pushing checks downward is one of the cheapest structural improvements available, in both dollars and debugging time.

Four layers stack from cheapest at the top to most expensive at the bottom, and the case count falls as the cost per case rises.

flowchart TD
    A["Unit — tools & parsers<br/>deterministic · ms · every commit"] --> B
    B["Component — one prompt, one skill<br/>~50 cases · minutes · every commit"] --> C
    C["Integration — full agent, fixed env<br/>~30 cases · ~10 min · every PR"] --> D
    D["Production — live traffic<br/>continuous · sampled"]

    style A fill:#2d6a4f,color:#fff
    style B fill:#40916c,color:#fff
    style C fill:#95d5b2,color:#000
    style D fill:#7209b7,color:#fff

The four layers differ in what they wrap around.

• Unit evals test the ordinary code that surrounds the model — the tools themselves and the parsers that read the model’s output. No model is involved, so they are deterministic, take milliseconds, and run on every commit (one saved change to the codebase).
• Component evals exercise a single prompt or a single skill in isolation: around 50 cases, a couple of minutes, still on every commit.
• Integration evals run the full agent against a fixed environment. Fixed means the tools and their data are pinned to the same starting state every time, so the model is the only thing that varies. About 30 cases, roughly ten minutes, once per PR.
• Production evaluation runs continuously against live traffic on a sampled basis, and is the subject of Online evaluation.

The dollar argument

Push checks down. The argument is usually made on taste; here it is on arithmetic.

Start with two per-case figures. An integration case costs on the order of $2 in model calls and takes on the order of 90 seconds. Both are order-of-magnitude figures that depend on your model, prompt length, and tool latency — measure your own before quoting them in an interview.

Running each case three times for the majority-vote gate from Why nothing here is reproducible triples both. The block below carries that $2 through to a weekly bill, then re-runs it with 80% of the assertions moved down a layer:

integration case:  ~$2, ~90s, and N=3 gating triples it       ->  ~$6 / case
30-case suite per PR:   30 x $6                                =  $180
40 PRs/week:            40 x $180                              =  $7,200 / week

move 80% of the assertions to unit/component:
   6 cases still need the full agent:  6 x $6 x 40             =  $1,440 / week

That is $7,200 a week down to $1,440 — a saving of $5,760 a week, from moving assertions rather than changing the agent.

Why the ten-minute figure needs concurrency

Money adds up; wall-clock time does not, and the difference is worth being explicit about, because the “~10 min” on the pyramid above is not the serial time.

Thirty cases at N=3 is 90 runs. Ninety runs of 90 seconds each is 8,100 seconds — 135 minutes if you run them one after another, which no one will wait for. The ten-minute figure assumes you run cases concurrently, in waves of however many the worker pool allows:

serial:      30 cases x 3 runs x 90s          = 8,100s = 135 min
16 at once:  ceil(90 runs / 16) = 6 waves     6 x 90s  =   9 min

Eval cases are independent by construction, so the suite is embarrassingly parallel — the only ceiling is your provider’s rate limit, which is the number you should size the worker pool against. A suite that takes 135 minutes because nobody parallelized it will be moved to a nightly job, and a gate that runs nightly is not a gate on the PR.

The dollar saving is real but it isn’t the main one. The diagnostic saving is bigger. A unit failure says parse_date("next tuesday") returned None, which names the defect and the fix in one line. An integration failure says case 17 failed and costs a human twenty minutes of trace reading to reach the same sentence — and by the arithmetic above it says it against a 45% background rate of false reds.

Only test at the integration layer what only exists at the integration layer: multi-step planning, tool selection under ambiguity, recovery after an error, and the safety invariants.

Start with 20 hand-written cases before building any harness — a harness being the framework that loads cases, runs them, and collects results. Twenty real examples beat an elaborate framework with five. Grow the set from production failures, so that every incident becomes a permanent regression case.

4. Building the dataset

The cases themselves have to come from somewhere, and the three available sources are not interchangeable. Source them well and give every case the right fields, and a red suite becomes diagnosable in a minute rather than an afternoon.

flowchart LR
    P[Production traces] --> F[Failure mining]
    H[Hand-written<br/>edge cases] --> D[(Eval set)]
    F --> D
    S[Synthetic<br/>variations] --> D
    D --> R[Regression suite]
    D --> B[Benchmark set]

    style F fill:#2d6a4f,color:#fff

Three streams feed one eval set:

• Hand-written edge cases come from you: the unusual inputs you can think of, plus adversarial ones written deliberately to try to make the agent misbehave.
• Production traces feed failure mining — the practice of reading the runs that went wrong in the real system and turning each one into a case.
• Synthetic variations are cases generated by a model from an existing case, to get volume cheaply.

The single eval set that results is then used in two different modes, and it is worth keeping the names apart. As a regression suite it runs on every change, to check that nothing which used to work has stopped working. As a benchmark set it runs once against several candidate designs, to pick between them.

Each source has a characteristic blind spot, and the third row’s is the one people miss:

The third row deserves its mechanism spelled out, because it is the one people get wrong.

A model generating eval cases samples from the same distribution it answers from. Cases it finds natural to write are cases it finds natural to solve, so a synthetic set systematically under-samples exactly the inputs your agent fails on. The cases you most need are the ones the generator is least likely to think of.

Use synthetic generation for volume on categories you already know are hard. Never use it as the source of the categories.

Every case needs four fields

Those four are input, expected outcome, grading method, and a tag.

The tag is the field people skip and then wish they had. Tags let you say “retrieval regressed” rather than “score went down”, and per-tag deltas — the change in score broken out by tag — are what make a red suite diagnosable quickly.

Here is the refund case from the top of the chapter, written out in that shape:

CASE = {
    "id": "refund-outside-window",
    "input": "I want a refund for ORD-882, bought 45 days ago",
    "expect": {
        "outcome": "declines refund, cites 30-day policy, offers store credit",
        "must_call": ["lookup_order"],
        "must_not_call": ["issue_refund"],
        "max_tool_calls": 4,
    },
    "grader": "llm_judge",
    "tags": ["policy", "refusal", "billing"],
}

must_not_call is the highest-value field in that object. It is binary, free to check, needs no judge, has no flake rate beyond the model’s own, and it catches the failures that cost money. It is also the only field in there that should block a merge on a single case — everything else gates on aggregate (Regression testing in ci).

5. Graders

Every case in Building the dataset carried a grader field, and filling it in is where the real decisions live, because most answers have no single correct wording. One question sorts the four kinds of grader. The hardest kind — using a model to grade another model — then needs a schema detail that decides whether its scores mean anything, and some arithmetic before you trust it at all.

The choice is a decision tree with one question at the top, and the four leaves get more expensive and less trustworthy as you move right.

flowchart TD
    G{Can you check it<br/>in code?} -->|Yes| CODE["Code assertion<br/>free · deterministic"]
    G -->|No, but there's<br/>a reference| SIM["Similarity / rubric match"]
    G -->|No| JUDGE["LLM judge<br/>calibrate first"]
    G -->|Highest stakes| HUM["Human<br/>ground truth"]

    style CODE fill:#2d6a4f,color:#fff
    style JUDGE fill:#bc6c25,color:#fff

The top question is always the same: can you check it in code?

• Yes → write a code assertion and stop.
• No, but there is a reference — a known-good answer someone wrote by hand — → score the output against it, either by similarity or by a rubric match. A rubric is a written scoring guide with named criteria.
• No reference either → fall back to an LLM judge: a second model prompted to grade the first one’s output. Calibrate it before trusting it.
• Highest stakes → pay for human ground truth. Ground truth means a label a person assigned that you treat as correct by definition.

Always prefer a code assertion. Tests pass, JSON (JavaScript Object Notation, the standard text format for structured data) validates, the SQL (Structured Query Language, the language databases are queried in) statement returns 3 rows, the forbidden tool was never called — these are free, instant, and never drift.

A judge is a second probabilistic system layered on the first. It adds its own variance (Why nothing here is reproducible), its own biases, and its own cost. Reach for it only for genuinely subjective dimensions such as tone, helpfulness, and faithfulness — whether every claim in the answer is actually supported by the source material it was given.

LLM-as-judge, done properly

Here is a judge written the way it should be written. It uses the Anthropic SDK (software development kit, the official client library) and pydantic, a Python library that turns a class definition into a JSON Schema — a machine-readable description of the shape the output must have.

Three details carry the design: the order of the fields in Judgment (the reasoning field is first, and that is load-bearing), the RUBRIC string with a written meaning for every score level, and the cache_control marker in the request — which, it turns out, does nothing here.

import anthropic
from pydantic import BaseModel, Field

client = anthropic.Anthropic()

class Judgment(BaseModel):
    reasoning: str = Field(description="Cite specific evidence before scoring.")
    correct: bool      # one field per criterion, not a dict:
    grounded: bool     # structured outputs require additionalProperties: false
    complete: bool     # on every object, and dict[str, bool] compiles to
    safe: bool         # additionalProperties: {"type": "boolean"}
    score: int = Field(ge=1, le=5)
    passed: bool

RUBRIC = """Score 1-5 on these criteria. A criterion is met only if you can
quote evidence for it.

5 — Fully correct, grounded in retrieved passages, no unsupported claims.
4 — Correct with a minor omission.
3 — Partially correct, or one unsupported claim.
2 — Substantially wrong or mostly ungrounded.
1 — Wrong, or fabricated a source.

Set correct, grounded, complete and safe independently.
Pass requires score >= 4 AND safe == true."""

def judge(question: str, answer: str, reference: str) -> Judgment:
    r = client.messages.parse(
        model="claude-opus-5",
        max_tokens=2048,                               # thinking + output, together
        system=[{"type": "text", "text": RUBRIC,       # identical every call
                 "cache_control": {"type": "ephemeral"}}],   # no-op here — see below
        messages=[{"role": "user", "content":
                   f"<question>{question}</question>\n"
                   f"<reference>{reference}</reference>\n"
                   f"<answer>{answer}</answer>"}],
        output_format=Judgment,
    )
    return r.parsed_output

Two details in that code are doing real work, and two more are traps that this exact code walks into.

The two that work: the rubric is a discrete 1-5 scale with a written anchor for every level, so “4” means something specific rather than a vibe. And the reasoning field is declared first in the class, which is not a style choice at all — it is the subject of the next subsection.

Trap 1: the schema guarantees less than you think

What the schema actually guarantees is narrower than “the output is forced to satisfy it.”

Constrained decoding enforces the structure: which field names appear, in which order, and of which JSON type. The model cannot emit passed: "yes" or omit safe, because those tokens are never legal at that position.

It does not enforce value constraints. The ge=1, le=5 on score compiles to JSON Schema minimum/maximum, which structured outputs do not support. The SDK strips those keywords from the schema it sends and re-checks them on the client afterwards.

So a judge that returns score: 7 is not impossible. It is a ValidationError raised inside your own process, after you have already paid for the call. Split the two tiers in your head:

• Enforced by the decoder: field names, field order, JSON types, required-ness.
• Enforced only by pydantic, after the fact: numeric bounds, string lengths, any other value-level rule.

Catch ValidationError around judge() and count how often it fires. A judge that trips it regularly is a judge that is not reading its own rubric.

Trap 2: the cache marker silently does nothing

The cache_control marker on that rubric never fires, and this is the failure mode Why cache hit rate is the highest signal derived metric is about.

Providers only cache a prefix above a minimum length. On claude-opus-5 that minimum is 512 tokens. The rubric above is about 106 — roughly a fifth of the way there.

Nothing tells you. The marker is accepted, no error is raised, cache_creation_input_tokens comes back as 0, and every call pays full price for the rubric forever.

Caching becomes real here only once the stable prefix crosses the minimum, which it will as soon as the rubric carries the few-shot examples a production judge needs. The way to know which side of the line you are on is to read the usage object on the response, not to trust the presence of the marker. That is the whole argument for cache hit rate as a metric, arriving one section early.

Trap 3: max_tokens is one budget for two things

max_tokens=2048 bounds the reasoning and the visible output together.

On current models reasoning is on by default — claude-opus-5 runs adaptive thinking unless you explicitly disable it — and max_tokens caps the thinking tokens and the answer as a single pool.

A reasoning-first judge is therefore the shape most likely to truncate. Spend 2,000 tokens thinking and the Judgment object gets 48. That does not come back as a low score. It comes back as an unterminated JSON string, a stop_reason of max_tokens, and a parse failure.

Size max_tokens for the reasoning you are asking for, not for the size of the object you want back, and treat parse failures as a signal to raise it rather than as a flaky judge.

Why reasoning is the first field — the mechanism

Field order in a judge schema changes the judge’s answers, and the reason falls straight out of how structured output is produced.

The two orderings are two different processes, and the whole difference is what is in the context at the moment the score gets written.

flowchart LR
    subgraph BAD["score first"]
        B1["Emit: score"] --> B2["Context at that moment:<br/>zero reasoning tokens"]
        B2 --> B3["Emit: reasoning<br/>conditioned on the score<br/>already in context"]
        B3 --> B4["Post-hoc justification"]
    end
    subgraph GOOD["reasoning first"]
        G1["Emit: reasoning<br/>N forward passes of<br/>evidence gathering"] --> G2["Emit: score<br/>conditioned on that evidence"]
        G2 --> G3["Score reflects the analysis"]
    end

    style B4 fill:#9d0208,color:#fff
    style G3 fill:#2d6a4f,color:#fff

Constrained decoding is the technique that guarantees structured output by restricting, at every step, which tokens are legally allowed next. It emits schema fields in declaration order.

The mechanism is worth spelling out rather than linking. The decoder walks the schema top to bottom. At each position it masks out every token that could not legally continue the JSON for the field it is currently on. So immediately after the opening brace, the only key the model is permitted to write is the first field in the class — and it cannot reach the second field without having emitted a complete value for the first (Structured output is a guarantee not a request).

The order is not cosmetic. It is the order in which the model is forced to commit.

Two distinct effects follow, and you want both in an interview answer.

1. Conditioning direction. Putting score first means the model must emit the score before anything else, at a moment when the context contains no analysis of the answer at all. Whatever comes out is a snap judgment.

The score then sits in the context, and the model goes on to emit reasoning conditioned on that number already being there. The reasoning comes out fluent and self-consistent — and it is a post-hoc justification of a verdict already committed to.

Declaring reasoning first inverts the arrow. The evidence is written first, and the score is emitted conditioned on it, so the score reflects the analysis instead of the other way round.

2. Serial compute. One forward pass — one complete trip of the input through the network — produces exactly one token (The forward pass).

Two hundred reasoning tokens are therefore two hundred additional forward passes applied to this specific judgment, each one able to attend over the evidence written so far. Score-first buys zero of them. Reasoning tokens are not a formality; they are how you spend more compute on a single decision.

They are no longer the only way to do it. Current models expose a thinking budget of their own — Anthropic’s is the effort setting, which controls how deeply the model reasons before it answers — and that is a second, independent dial on the same quantity.

But the two are not interchangeable. The thinking budget buys forward passes whose text you never see. A reasoning field buys forward passes whose text lands in the trace, gets read by a human debugging a bad verdict, and can be quoted back in a calibration review. For a judge you want both.

Put the two resulting outputs side by side and the difference is visible in one read. Both judged the same answer; only the second one caught that the citation was invented:

score-first schema:
  {"score": 5, "reasoning": "The answer correctly identifies the 30-day
   policy and cites the source..."}          <- the citation was fabricated;
                                                the judge never checked, it
                                                justified

reasoning-first schema:
  {"reasoning": "The answer cites [policy-v2 sec 4]. The reference contains
   no section 4; the retrieved passages are policy-v2 sec 1-3. The claim
   about store credit is unsupported.", "score": 2, ...}

Five rules

Five rules cover everything a judge needs to be worth its cost.

1. Put the reasoning field first in the schema, for the two reasons derived immediately above.
2. Use discrete scales with defined anchors — 1-5, each level described. Never “rate 0-100”: the model has no calibrated notion of 73 versus 76, and the extra resolution is pure noise that you will then average and report to three decimal places.
3. Judge one dimension at a time. A blended “quality” score hides which thing regressed, whereas a separate true-or-false verdict per dimension is what lets a red suite be diagnosed at all.
4. Calibrate the judge against human labels — roughly 50 to find out whether it is obviously broken, several hundred before you let it gate anything — and report the agreement with its confidence interval, using the arithmetic below.
5. Know the biases, and know the mechanism behind each one, because the fix follows from the mechanism.

The three biases, with mechanisms

Three biases show up in every judge, and each one has a known cause — which matters, because the fix is readable off the cause rather than guessed at.

Seven terms in that table need unpacking:

• A pairwise prompt shows the judge two candidate answers at once and asks which is better.
• Recall is U-shaped means the model reliably uses material at the very start and the very end of its context window, and is measurably worse at using material in the middle.
• The flip rate is the fraction of pairs whose winner changes when you swap the two candidates’ positions.
• Preference-tuned describes the training stage in which a model is adjusted toward outputs that human raters preferred — raters who, on average, rewarded length.
• Recall and precision describe any grader. Recall is the share of things that were really there which it found. Precision is the share of the things it flagged that were really there. A long answer gives the judge more text in which to find a match for each criterion, so it finds more of the real ones (recall rises) while also matching on more text that does not really support the claim (precision falls).
• A model’s output distribution is the spread of text it is likely to produce, and its stylistic prior is the leaning toward certain phrasings that it brings before reading anything at all.

Note that all three fixes are structural rather than prompt-level, which is the same ordering principle as advisory-versus-enforcement in Advisory vs enforcement mechanically: an instruction in a prompt is a request the model may ignore, while a change in how you run the comparison is a constraint it cannot. “Please ignore length” in the rubric is advisory and decays; comparing at matched length does not.

Calibration: what an agreement rate actually means

A judge is itself a model, so before you let it gate anything you have to measure it against people. Label ~50 cases by hand, run the judge on the same 50, and build the confusion matrix — the two-by-two count of how the two graders’ verdicts lined up.

Rows are what the judge said; columns are what the human said. The top-left 39 is “both said PASS”, and the top-right 5 is “the judge said PASS but the human said FAIL” — the judge let five real failures through.

                 human: PASS   human: FAIL   judge total
judge: PASS           39            5            44
judge: FAIL            1            5             6
human total           40           10            50

The two diagonal cells (39 and 5) are the cases they agreed on, so raw agreement is their sum over the total:

raw agreement  p_o = (39 + 5) / 50 = 0.88

88% sounds good and is nearly meaningless on its own. Here is why: a judge that returns PASS unconditionally, having read nothing at all, would agree with the human on all 40 of the human’s PASS cases and none of the 10 FAILs, scoring 40/50 = 0.80. The set is 80% pass, so 80% agreement is what you get for free.

So you have to subtract the agreement two graders would reach by luck alone. The way to compute that: if the two graders were independent, the chance they both say PASS is just the product of how often each says PASS on its own — the human said PASS 40 times out of 50, the judge 44 out of 50. Same for both saying FAIL. Add the two products:

p_e = P(both say PASS by chance) + P(both say FAIL by chance)
    = (40/50) x (44/50)          + (10/50) x (6/50)
    = 0.80   x  0.88             +  0.20   x  0.12
    = 0.704                      +  0.024
    = 0.728

Then subtract that floor from the observed agreement, and divide by how much room there was above the floor:

kappa = (p_o - p_e) / (1 - p_e)
      = (0.88 - 0.728) / (1 - 0.728)
      = 0.152 / 0.272
      = 0.56

That statistic is Cohen’s kappa: agreement above chance, divided by the agreement that was available above chance. 1.0 is perfect, and 0.0 is no better than guessing at the observed rates.

The scale does not stop at 0. Kappa runs down to -1, and a negative value means the two graders agreed less often than their own pass rates predict. That is not a slightly worse judge; it is systematic anti-correlation, and in practice it almost always means a bug rather than a weak model — an inverted boolean, a swapped column, a passed field being read as failed. Check the wiring before you tune the prompt.

A kappa of 0.56 is conventionally called moderate. Good enough for triage, meaning sorting cases so a human looks at the suspicious ones first. Not good enough to gate a release.

The number raw agreement hid

Go back to the top-right cell of the matrix. Of the 10 cases a human failed, the judge failed only 5:

judge recall on the FAIL class = 5 / 10 = 0.50

Recall on a class is the share of that class the grader actually caught. So: the judge misses half the failures, and failures are the entire reason the eval exists.

Report per-class recall alongside kappa, always. A judge with 0.95 agreement and 0.40 fail-recall is a machine for producing green dashboards.

Both statistics come out of one small function. Note that the p_e line is the same product-of-marginals calculation done above, and fails filters to the human-FAIL rows before measuring what the judge caught:

def calibration(pairs: list[tuple[bool, bool]]) -> dict:
    """pairs: (human_passed, judge_passed)"""
    n = len(pairs)
    agree = sum(h == j for h, j in pairs)
    p_o = agree / n

    hp = sum(h for h, _ in pairs) / n            # human pass rate
    jp = sum(j for _, j in pairs) / n            # judge pass rate
    p_e = hp * jp + (1 - hp) * (1 - jp)          # chance agreement
    kappa = (p_o - p_e) / (1 - p_e) if p_e < 1 else 1.0

    fails = [(h, j) for h, j in pairs if not h]
    fail_recall = sum(not j for _, j in fails) / len(fails) if fails else None
    return {"agreement": p_o, "kappa": kappa, "fail_recall": fail_recall}

50 labels can reject a judge; they cannot accept one

Everything above is two point estimates from a sample of 50, and Why nothing here is reproducible opened this chapter by insisting that a single sample is not a measurement. That argument does not stop applying because the thing being sampled is now a grader. Put error bars on both numbers before either one is allowed near a gate.

An error bar here is a confidence interval: the range of true values that are consistent with what you observed. A 95% interval means that if you repeated this whole labelling exercise many times, about 95% of the intervals you computed would contain the true value.

Kappa first. Its large-sample standard error — the typical distance between an estimate and the truth — has a standard formula you can look up rather than derive. Substituting p_o = 0.88, p_e = 0.728, and n = 50:

SE(kappa) ~ sqrt( p_o (1 - p_o) / ( n (1 - p_e)^2 ) )
          = sqrt( 0.88 x 0.12 / ( 50 x 0.272^2 ) )
          = sqrt( 0.1056     / ( 50 x 0.073984 ) )
          = sqrt( 0.1056     /   3.6992 )
          = sqrt( 0.02855 )
          = 0.169

A 95% interval is the estimate plus or minus 1.96 standard errors:

0.5588 +/- 1.96 x 0.169  =  0.5588 +/- 0.331  =  [0.23, 0.89]

0.5588 is the same kappa rounded to 0.56 above, carried to four places so the arithmetic reproduces.

Fail-recall is worse off, because it rests on the 10 human FAIL cases and nothing else. Its 95% interval is computed with the Wilson interval — the small-sample-safe interval for a proportion, which unlike the textbook p +/- 1.96 sqrt(p(1-p)/n) cannot run past 0 or 1. For 5 out of 10 that comes out as [0.24, 0.76].

Now lay both intervals next to the thresholds you would gate on, and the two numbers behave completely differently:

kappa        = 0.5588   95% CI ~ [0.23, 0.89]   gate 0.6  -> INSIDE  the interval
fail-recall  = 5/10     95% CI   [0.24, 0.76]   gate 0.8  -> OUTSIDE the interval

The kappa result is not evidence about the gate at all. The interval spans fair to almost perfect, and 0.6 sits in the middle of it: this judge might be well past the bar or well under it, and 50 labels cannot tell you which. Reporting “kappa = 0.56, below our 0.6 threshold” as though it were a finding is exactly the mistake Why nothing here is reproducible warns about, one level up.

The fail-recall result is evidence. The whole interval lies below 0.8, so the data rules the judge out at 95% confidence, and it does so despite resting on only ten events — because the estimate is far enough from the threshold that even a wide interval does not reach it.

That asymmetry is the lesson, and it generalizes: 50 labels is enough to reject a judge and not enough to accept one. A cheap sample can only ever be conclusive when the answer is bad enough to clear the noise band by itself. Confirming that something is good means resolving a small difference near the threshold, and that is precisely what a small sample cannot do.

So price the acceptance decision honestly. The standard sample-size formula for a proportion is n = z^2 x p(1-p) / margin^2, where z = 1.96 for 95% confidence, p is the value you expect, and margin is how tight you want the interval to be. To pin fail-recall to within about 8 points at a true value near 0.8:

n = 1.96^2 x 0.8 x 0.2 / 0.08^2
  = 3.8416 x 0.16      / 0.0064
  = 0.6147             / 0.0064
  = 96 examples in the FAIL class alone

That 96 is failures, not cases. At a 20% fail rate — the rate on the set above — collecting 96 failures means labelling 96 / 0.2 = 480 cases.

480 labels is the real price of promoting a judge from triage to gate, an order of magnitude more than the “~50 human labels” everyone quotes, this chapter included.

The 50-label pass is not wasted. Run it first, because it is cheap and it kills bad judges outright. Just do not read a survival as a pass.

Rule of thumb, with that in mind: kappa >= 0.6 and fail-recall >= 0.8, measured on enough labels that the whole confidence interval clears the threshold, before a judge is allowed to gate anything. A point estimate above the bar with the bar inside its interval means “not measured yet”, not “passed”. Until then the judge is a triage tool that routes cases to a human, not a gate.

6. Diagnosing a RAG agent in the right order

A calibrated grader can tell you a case failed; it cannot tell you which stage of a multi-stage pipeline failed it. Retrieval makes the worked example here, because the order of investigation is forced by the mechanism rather than chosen by preference — and getting it backwards is the most expensive common mistake in this chapter.

RAG stands for retrieval-augmented generation: looking a question up in your own documents and pasting the passages you find into the model’s prompt, so the answer is built from text the model can actually read rather than from whatever it absorbed during training. The piece that finds passages is the retriever; the piece that writes the answer from them is the generator.

The diagnosis is a fixed sequence of yes/no questions, starting from the symptom. Each “No” names the stage to fix and stops you there: you never look at a later stage until the earlier one is ruled out.

flowchart TD
    S["Success rate is low"] --> R{"Recall@k:<br/>is the gold passage<br/>in the retrieved set?"}
    R -->|No| FIX1["Fix retrieval first.<br/>chunking · hybrid search · rerank"]
    R -->|Yes| P{"Is it in the<br/>final prompt<br/>after truncation?"}
    P -->|No| FIX2["Fix context assembly.<br/>k, ordering, budget"]
    P -->|Yes| G{"Is the answer<br/>grounded in it?"}
    G -->|No| FIX3["Fix generation.<br/>prompt · citations · model"]
    G -->|Yes| FIX4["The case is fine.<br/>Check the grader."]

    style FIX1 fill:#2d6a4f,color:#fff
    style FIX2 fill:#40916c,color:#fff
    style FIX3 fill:#95d5b2,color:#000

Walk the tree from the top. You observe that success rate is low. The first question is recall@k: is the gold passage in the retrieved set? — the gold passage being the specific passage that contains the answer, and recall@k the fraction of cases where it appears among the top k results the retriever returned. If it is not there, fix retrieval first, which means revisiting chunking (how documents are cut into retrievable pieces), hybrid search (combining meaning-based matching with literal keyword matching), and rerank (a second, slower scoring pass over the first pass’s shortlist).

If the passage was retrieved, ask whether it is in the final prompt after truncation — models accept only a bounded amount of text, so assembly code drops whatever does not fit. If it was dropped, fix context assembly: the value of k, the ordering of passages, and the token budget. If it survived to the prompt, ask: is the answer grounded in it? Grounded means every claim in the answer is supported by text that was actually present. If not, fix generation — the prompt, the citation requirements, or the model. And if the answer was grounded and the case still scored as a failure, the case is fine. Check the grader.

This ordering is a dependency, not a preference. The generator can only ground a claim in text that is physically present in its context. If the gold passage was never retrieved, no prompt change, no model upgrade, and no judge rubric can produce a correct grounded answer — the information does not exist in the forward pass.

Turning the complaint into an allocation decision

The reason that ordering is worth enforcing is that it lets you price the two possible fixes. Decompose end-to-end success into the two stages:

success  =  recall@k  x  P(correct | gold passage retrieved)

The vertical bar in that second term reads “given”: it is the probability the answer is correct given that the gold passage was retrieved. In words: you succeed when the retriever finds the passage and the generator uses it correctly.

Both terms are measurable separately, so measure them. You observe end-to-end success directly and you measure recall@10 directly; rearranging the formula gives you the generator’s term without measuring it:

observed success           = 0.61
recall@10                  = 0.72
=> P(correct | retrieved)  = 0.61 / 0.72 = 0.85     the generator is fine

That division hides an assumption, and you should state it out loud before you spend a sprint on it. The complete decomposition has two terms, not one:

success = recall@k x P(correct | retrieved)  +  (1 - recall@k) x P(correct | NOT retrieved)

Dividing observed success by recall assumes the second term is zero — that when the gold passage is missing, the agent always fails.

That is exactly true for a question the model cannot possibly answer from training data: last quarter’s revenue, this customer’s order. It is false for anything the model already knows, where it will answer correctly from parametric memory — what the model absorbed during training — with no retrieval at all.

When the assumption breaks, the decomposition credits retrieval for wins retrieval had no part in, and P(correct | retrieved) comes out too high. Suppose the model gets 20% of the non-retrieved cases right anyway. The non-retrieved share is 1 - 0.72 = 0.28, so:

0.61 = 0.72 x P(correct | retrieved) + 0.28 x 0.20
0.61 = 0.72 x P(correct | retrieved) + 0.056
=> P(correct | retrieved) = (0.61 - 0.056) / 0.72
                          = 0.554 / 0.72
                          = 0.77     not 0.85

Eight points of “the generator is fine” were on loan from the model’s training data.

Measure the term rather than assuming it: run the same suite with retrieval disabled and read off the success rate. If it is near zero, use the one-term shortcut with a clear conscience. If it is not, your eval set is partly testing the model’s memory rather than your pipeline — which is worth knowing on its own.

With both numbers in hand, price the two possible fixes by asking what each would be worth if it fully succeeded. Each line multiplies the two stage terms and compares against the current 0.61:

perfect generator, current retrieval:   1.00 x 0.72 = 0.72   (+0.11)
current generator, recall -> 0.95:      0.85 x 0.95 = 0.81   (+0.20)

Fixing retrieval is worth nearly twice as much as a perfect generator here — and it is also the cheaper fix, because it is chunking, hybrid search, and reranking (Evaluating retrieval) rather than model spend. Measuring faithfulness while recall@10 sits at 0.72 attributes a retrieval bug to the generator, and you spend a week on prompt wording for a ceiling of +0.11.

The most common cause of low recall@k

Worth knowing before you start. Most retrievers match on embeddings — fixed-length lists of numbers that stand in for a passage’s meaning, produced so that related texts come out numerically close.

Those are called dense embeddings because every slot in the list carries a value. They are lossy compression optimized for meaning: squeezing a passage into a few hundred numbers throws information away, and what survives is the gist.

So exact literals get washed out — error codes, order IDs, and SKUs (stock-keeping units, a retailer’s product codes) (Embeddings and why dense search misses err_4021). If a user searches for ERR_4021 and the retriever returns passages about errors in general, that is the failure. Check whether your misses are literal-shaped before touching the generator at all.

The whole diagnosis is four counts

The function below computes them, and it splits the second term of the two-stage formula in half: P(correct | retrieved) is really “survived assembly” times “solved once it was actually in the prompt”, so the tree above and the arithmetic here are the same decomposition at two different resolutions — and the three conditional rates still multiply back to end-to-end success.

def diagnose(cases) -> dict:
    retrieved_ok = sum(c.gold_id in c.retrieved_ids for c in cases)
    in_prompt    = sum(c.gold_id in c.prompt_ids for c in cases)
    solved       = sum(c.passed for c in cases)
    n = len(cases)
    return {
        "recall_at_k":         retrieved_ok / n,
        "survived_assembly":   in_prompt / max(retrieved_ok, 1),
        # pass rate given the passage was in the prompt — NOT groundedness
        "solved_given_prompt": solved / max(in_prompt, 1),
        # recall_at_k x survived_assembly x solved_given_prompt == end_to_end
        "end_to_end":          solved / n,
    }

survived_assembly is the step people forget: the passage was retrieved, and then dropped by a context budget or pushed into the low-recall middle of the window. It looks like a generation failure and it is a plumbing failure.

And be precise about what the third rate measures, because the tempting name for it — generation quality — claims more than the count supports. solved_given_prompt counts cases the grader passed, and a case can pass while being ungrounded: the model had the passage in front of it, ignored it, answered from memory, and happened to be right. That is a pass and a latent failure at the same time. Groundedness is a different measurement that needs a different instrument — the faithfulness judge from Graders, checking each claim in the answer against the passage that was actually present. Report both: the pass rate tells you whether users are getting right answers, the grounded rate tells you whether they will keep getting them when the model changes.

7. Metrics that matter

Every step of that diagnosis assumed the numbers were already on a dashboard — so which numbers belong there, and which single derived metric detects breakage soonest? Reporting accuracy on its own is the most common junior tell in an agent interview, and the table below is the fix.

Ten metrics earn a place, each for telling you something none of the others would.

Five entries in that table need a gloss:

• p50 and p95 are percentiles. p50 is the median; p95 is the value that 95% of requests come in under, so it describes the slow tail rather than the typical case.
• The four usage fields are the token counts the provider returns with every response: fresh input tokens, output tokens, tokens read from cache, and tokens written to cache. Each is billed at a different rate, which is why cost per task has to sum all four.
• A hallucination is a claim the model states confidently that its sources do not support — which is why the grounding row says it is the same quantity read the other way up.
• Escalation is handing a run over to a human.
• An error taxonomy is your written classification of the ways a run can fail, which is what makes “recovered from an error” a countable event rather than an impression.

Report success rate together with cost and p95 latency. Quoting accuracy alone is the single most common junior tell.

Why cache hit rate is the highest-signal derived metric

One entry in that table deserves its own argument, because it is the one that belongs on the first dashboard you build and the one people leave off.

First, some background. Prompt caching, also called prefix caching, stores the model’s processed form of a prompt prefix so that a later request beginning with exactly the same tokens can skip recomputing it and be billed at a fraction of the normal input rate. The cache hit rate is the share of your input tokens that were served that way. It has three properties nothing else on that list has.

1. It is computable from one request. Cost per task needs an aggregation window wide enough to beat traffic variance. Hit rate comes straight out of a single usage object, so a regression is visible on the first affected request rather than on tomorrow’s bill.

2. It is a step function, not a drift. A step function jumps from one value to another with nothing in between, which is the opposite of a metric that slides. Prefix caching is exact-match on tokens (Prompt caching derived) — one token different at the front and nothing behind it can be reused. So adding a datetime.now() timestamp to the system prompt does not degrade the hit rate gradually. It takes it from 0.92 to 0.00 on every request, instantly, with no error and no exception raised anywhere. There is no gradual version of this failure.

3. Its magnitude is large and derivable. Price one agent turn twice — once with the cache working and once without — and the gap is the size of the failure.

The turn: a 12,000-token stable prefix (the system prompt and tool definitions, identical every request), 1,000 new input tokens, and 300 output tokens. On claude-opus-5, cache reads bill at $0.50 per million tokens (written $0.50/MTok), fresh input at $5/MTok, and output at $25/MTok. A price per million times a token count gives dollars once you divide by a million, so 12,000 tokens at $0.50/MTok is 12,000 x 0.50 / 1,000,000 = $0.0060.

cached:    12,000 x $0.50/MTok  +  1,000 x $5/MTok  +  300 x $25/MTok
         =      $0.0060         +     $0.0050       +    $0.0075   = $0.0185

uncached:  13,000 x $5/MTok                         +  300 x $25/MTok
         =      $0.0650                             +    $0.0075   = $0.0725

$0.0725 / $0.0185 = 3.9x

Note what changed between the two: in the uncached case the 12,000 prefix tokens join the 1,000 new ones as 13,000 tokens all billed at the full $5/MTok rate, instead of being read back at a tenth of that.

That 3.9x is the steady-state ratio, and the first request of any new prefix does not get it. Writing the cache costs more than plain input — about 1.25x, so $6.25/MTok — which makes the request that populates the cache the most expensive one you will send:

first request:  12,000 x $6.25/MTok  +  1,000 x $5/MTok  +  300 x $25/MTok
              =      $0.0750         +     $0.0050       +    $0.0075   = $0.0875

                $0.0875 / $0.0725 = 1.21x  ->  the first call costs MORE than not caching

The premium is paid back on the second call. Over two requests, caching costs $0.0875 + $0.0185 = $0.106 against 2 x $0.0725 = $0.145 uncached, so you are ahead from the second request onward — caching is still obviously right for an agent that re-sends the same prefix all day.

But it means a prefix that changes every request is worse than useless. You pay the 1.25x write, never read it back, and the marker gives you no error to notice by. Quote the 3.9x as what a warm cache buys, not as what appears on the first invoice.

So a silent cache break is a ~4x per-request cost event. On a daily invoice that also moves with traffic volume, prompt length and model mix, a 4x on one code path can hide inside normal variance for a day or two. The hit rate shows it in one request; the bill shows it in one week. That is the whole argument, and it is why this metric goes in the first dashboard you build (Prompt caching the highest leverage lever).

One gotcha the formula encodes: input_tokens excludes cached tokens, so cache_read / input is not the hit rate — all three fields have to go in the denominator. The three are disjoint, meaning no token is counted twice, so adding them gives the true total prompt size:

def cache_hit_rate(usage) -> float:
    read = usage.cache_read_input_tokens or 0
    write = usage.cache_creation_input_tokens or 0
    total = read + write + usage.input_tokens      # the three are disjoint
    return read / total if total else 0.0

Interpretation: a hit rate of 0.0 on a multi-turn agent — one that goes several exchanges back and forth, and therefore re-sends a long identical prefix every time — is always a structural bug. Either the cache breakpoint is missing (the marker saying “everything up to here is stable, store it”), or a value that changes every request was placed before that marker, or the prefix is shorter than the minimum length the provider will cache — which is the trap the judge in Llm as judge done properly falls into, where a 106-token rubric sits under a 512-token minimum and the cache_control marker silently does nothing. It is never a traffic pattern.

8. Regression testing in CI

A suite pays for itself once it runs automatically on every proposed change — blocking real regressions without crying wolf. Three fixes and one versioning discipline cover it.

flowchart LR
    PR([Pull request]) --> U[Unit evals<br/>seconds]
    U --> C[Component evals<br/>~2 min]
    C --> I["Integration, 30 cases, N=3<br/>~10 min"]
    I --> G{Gate}
    G -->|"success >= baseline - noise<br/>AND safety == 0"| M([Merge])
    G -->|regression| B([Block + per-tag diff])

    style M fill:#2d6a4f,color:#fff
    style B fill:#9d0208,color:#fff

The pipeline runs cheapest-first. A pull request triggers the unit evals, which finish in seconds; then the component evals, about two minutes; then the integration cases — 30 of them at N=3, about ten minutes of wall clock because the 90 runs go out concurrently (The eval pyramid), not the 135 minutes they would take in sequence. Only then does the gate fire, and its condition is success >= baseline - noise AND safety == 0: the suite’s success rate must be no worse than the baseline once the measured noise band is subtracted, and safety violations must be exactly zero. Anything else blocks the merge and reports a per-tag diff, so the author sees which category moved rather than just a red cross.

Non-determinism makes naive gating flaky, and Why nothing here is reproducible gives the exact size of the problem: 45% false-red on a 30-case suite at a 2% per-case flake rate. Three fixes attack three different terms in that arithmetic.

1. Run N=3 and take the majority for pass/fail cases. This attacks the per-case flake rate directly, moving it from 0.98 to 0.9988.

2. Gate on aggregate, not per-case — “success >= 85%”, not “case 17 must pass”. This attacks the 30-way multiplication, because one case flipping moves the aggregate by 3.3 points instead of turning the whole suite red. The exception is safety cases, which stay individually blocking, because the cost asymmetry between a false red and a shipped violation is not close.

3. Track a moving baseline and alert only on a drop beyond noise. The noise band has to be measured rather than guessed: run the unchanged suite five times and take the spread of the resulting scores — here, the gap between the highest and the lowest of the five, which is the crudest possible estimator and the one people actually compute. If the run-to-run spread is 4 points, then a 3-point “regression” is not one.

   Five is a floor, not a principled number, and it is worth saying why rather than quoting it as a rule. Five runs of a ten-minute suite is under an hour of machine time, which is the largest number most teams will actually pay before shipping, and it is enough to tell a 1-point band from a 5-point one — which is the decision the gate needs. What five runs cannot do is estimate the tail: the max-minus-min of five draws systematically understates the true range, because five draws are unlikely to include the extremes, so a band measured this way is an optimistic one. Treat it as a lower bound, re-measure it whenever the model or the suite changes, and if a gate is going to block merges all day, spend the twenty runs. This is the same small-sample caveat as the judge in Labels can reject a judge they cannot accept one, applied to your own suite: a cheap sample can rule a regression in, but it cannot certify that a quiet suite is quiet.

Version the four invisible artifacts

That third fix only works if you know what changed. Version and diff the four things that change behaviour invisibly, recording a hash of each — a short fingerprint computed from the contents, so any edit at all produces a different fingerprint.

Each row is a change that produces a score movement with no accompanying code diff to blame it on.

Two of those rows need a word of explanation. An alias repoint is when a name you pin, such as latest, is quietly moved by the provider to point at a different underlying model, so your code is unchanged and the model is not. And the last row is the one that bites hardest, because the delay hides the cause: a silent index rebuild produces a mysterious eval drop three days later, whereas a version stamp turns that into a one-minute diagnosis.

What interviewers probe: “Your eval suite is red — walk me through it.” The answer is an ordered diagnosis, not a guess: (1) did anything version-stamped change? (2) is the drop outside the measured noise band, or did you just observe a sample? (3) which tag moved — if it is the retrieval tag, Diagnosing a rag agent in the right order says stop looking at the generator. Anyone can say “I’d look at the traces.”

9. Observability

Every diagnosis above assumes the evidence exists — that the right things were recorded while the agent ran.

Observability is the practice of instrumenting a system so that you can answer questions about a past run without reproducing it. That matters more here than anywhere else, since Why nothing here is reproducible says you cannot reproduce it. You cannot evaluate what you cannot see. Instrument before you optimize.

The record has the shape of a tree: one run at the root, and below it every piece of work that happened during that run, with the attributes to record for each kind.

flowchart TD
    R([Run: trace_id]) --> S1[Span: model call]
    R --> S2[Span: tool call]
    R --> S3[Span: retrieval]
    S1 --> A1["model, tokens in/out,<br/>cache read/write,<br/>stop_reason, latency, cost"]
    S2 --> A2["tool, args, result size,<br/>error, duration"]
    S3 --> A3["query, ids returned,<br/>scores, index version"]

    style R fill:#1d3557,color:#fff

At the root sits one run, tagged Run: trace_id — a single identifier stamped on everything that run touches. Hanging off it are spans, one per unit of work, each with a start time, an end time, and a bag of attributes.

Three span kinds cover an agent:

• Model call — records model, tokens in/out, cache read/write, stop_reason, latency, cost.
• Tool call — records tool, args, result size, error, duration.
• Retrieval — records query, ids returned, scores, index version.

The minimum viable trace, as a checklist

Six things, each one mapping to a diagnosis you cannot make without it:

1. A trace_id threading every span, including subagents. Without one, a run split across several cooperating agents cannot be reconstructed at all (Communication).
2. The full message history at each turn. If the messages are too large to keep inline, store a hash plus a pointer into blob storage — a cheap bulk file store you can fetch the full text back from later.
3. Every tool call with its name, arguments, result, duration, and error.
4. The usage object per call — input_tokens, output_tokens, cache_read_input_tokens, and cache_creation_input_tokens. All four, because the running cost ledger in Budget enforcement needs all four to price a turn.
5. stop_reason per call — the field saying why generation ended.
6. The version stamps from Regression testing in ci — prompt hash, tool-set hash, model ID, index version.

Item 5 is the one people skip and the one that explains the most confusing incidents. max_tokens (ran out of room) and refusal (declined to answer) are silent killers: neither raises an exception. A run that “answered badly” had actually been truncated at max_tokens, and the trace looked fine because the text field was populated.

10. Online evaluation

Measurement does not stop at launch. The offline suite the previous nine sections build is a fixed sample of a distribution that moves, so it goes stale: production is the real distribution, and your eval set is not.

Six signals are available in production, and they fall into four kinds:

• Explicit — signals the user deliberately gives you.
• Implicit — signals inferred from what the user does rather than says.
• Automated — scores produced by a judge on live traffic.
• Controlled — comparisons you set up on purpose.

The implicit rows are the ones worth dwelling on: they cost nothing to collect and nobody has to be asked.

Two abbreviations in that table: HITL is human in the loop, an existing path for handing a run to a person, which means the escalation signal is free if you already have one. An A/B test splits live traffic between two versions at random and compares the results, and randomizing is what makes the comparison honest — without it, any difference could be a difference in who got which version.

Shadow mode is the safest launch. Run the new agent alongside the old one on real traffic, log both, and ship nothing the user sees. You get a real-distribution comparison at zero user risk, and it is the only way to catch the failure class that offline evals structurally cannot: inputs you never imagined, which is by definition exactly the set your hand-written cases exclude.

Cheat sheet

Ten questions, each with the mechanism that forces its answer. Read it middle column first: if you can reproduce the mechanism, the right column follows without memorizing it.

Next: 09 — Production & Cost — making it fast and affordable enough to ship.