02a — Building One: Evaluator–Optimizer, Three Tiers

This chapter builds one loop three times, at three levels of fidelity:

1. Pseudocode.
2. A graph framework — the same loop expressed as nodes and edges.
3. The vendor’s SDK — a production version. SDK stands for software development kit: the official client library for calling the model.

All three are the same loop. All three make the same number of model calls — six, at a cap of three rounds. An implementation that makes seven has a bug, for a reason Tier 1 traces call by call.

The two design decisions an interviewer will probe: when the loop stops, and what happens when the judge is wrong.

What the pattern is

An Evaluator–Optimizer is a draft-and-critique loop. It uses three moves:

• One large language model (LLM) call generates an answer — the draft.
• A second call judges that draft against criteria you wrote down in advance — the verdict.
• On a failing verdict, the generator revises, using the judge’s specific complaints.

Repeat until the judge passes the draft, or until a round cap — a hard limit on iterations that you set — stops the loop.

flowchart TD
    A[Generate draft] --> B[Evaluate draft against criteria]
    B --> C{Passed?}
    C -->|Yes| D[Return draft, True]
    C -->|No| E{Round cap reached?}
    E -->|Yes| F[Return draft, False]
    E -->|No| G[Revise using issues]
    G --> B

Why the judge is a separate call

The key detail is that the judge runs as a separate call with a fresh context.

“Fresh context” means the judge’s request contains only the task and the draft. It does not contain the conversation that produced the draft — no earlier attempts, no reasoning, no “here is my answer” framing.

That matters because the model reads the draft as input to judge rather than as something I just wrote. A model asked to critique its own visible output in the same conversation tends to defend it. The same model handed the same text cold will find the missing citation.

The full treatment — why the fresh context matters, when to use the pattern, how it oscillates, and what it costs — is in Evaluatoroptimizer. This file covers the code.

Input and output

The loop takes two things in:

• A task in ordinary text — “write the API integration guide for the payments endpoint”.
• A written list of criteria — the standards the judge grades against.

It returns two things:

• The last draft it reached.
• A boolean saying whether that draft actually passed.

Both halves of the return value matter. A loop that hands back a draft without saying whether it passed is indistinguishable from one that succeeded. This is the most dangerous bug in the pattern: the caller ships a failing draft believing it was approved.

And the draft is the last one, not the best one. Nothing in the loop ranks two drafts against each other. Nothing keeps the highest-scoring one. The judge returns a pass/fail verdict, not a score you could sort on — so there is nothing to sort by even if you wanted to.

In an interview, state those two things in order: “the last draft, and a flag” first, then “the loop keeps no ranking, so it cannot promise the best one” — because that is the follow-up question.

The other pattern asked for in code this often is ReAct, built at the same three tiers in chapter 01. Together the two files cover the loops you are most likely asked to write in an interview.

Tier 1 — Pseudocode

The pseudocode has three return statements; two of the three report that the draft did not pass.

draft = generate(task)
for round in 1..MAX:
    verdict = evaluate(task, draft)          # fresh context, structured output
    if verdict.passed: return draft, True
    if round == MAX:   return draft, False   # judged, and it failed: report the miss
    draft = generate(task, draft, verdict.issues)

There is no fourth way out of the loop. Every draft that can be returned has passed through evaluate first, which is what lets the final return report False honestly instead of handing back a draft no evaluator saw.

Counting the calls

The call count is a common interview question. Trace it rather than memorize it.

One generate runs before the loop. Then each round runs one evaluate, and each round except the last runs one generate to revise. Trace it at MAX = 3:

        generate  ->  d0                      (before the loop)
round 1: evaluate d0 -> fail  ->  generate -> d1
round 2: evaluate d1 -> fail  ->  generate -> d2
round 3: evaluate d2 -> fail  ->  return (d2, False)

Count the columns: 3 generates (d0, d1, d2) and 3 evaluates. So N generates plus N evaluates — 2N calls at a cap of N rounds, which is six at N = 3.

Why chapter 02 says 2N+1

Evaluatoroptimizer prices the same pattern at 2N+1, seven at N = 3. Both numbers are right, about different loops.

2N+1 counts the version that revises on its way out the door — one that runs generate after the round-3 verdict and returns that new draft. Trace that variant and the last two steps become:

round 3: evaluate d2 -> fail  ->  generate -> d3  ->  return (d3, False)

Seven calls, and the extra call is exactly the draft nobody judged. d3 was produced after the last evaluation, so no evaluator ever saw it; the caller receives an unreviewed artifact.

That gives a rule worth stating: generates and evaluates pair up when you judge everything you might return, so an odd call count means one draft went out unread.

The verdict schema, which both tiers below need

Before either real implementation, define what a verdict is. Both tiers depend on it, and it is written down first because it is the only part of this design that is neither framework nor plumbing — it is the contract.

pydantic is a library that turns an ordinary Python class into a machine-checkable description of the data you want back. You declare the fields and their types; pydantic then does two jobs with that one declaration. It produces the schema you send to the model, and it validates the model’s reply into a real Python object.

The class below plays that dual role: it is the schema handed to the judge, and the type the judge’s answer arrives as. It has four fields and two helper functions, each explained below.

from pydantic import BaseModel, Field


class Verdict(BaseModel):
    reasoning: str = Field(description="Cite specific evidence before judging.")
    criteria_met: dict[str, bool]
    passed: bool
    issues: list[str]


# The five names the judge is asked about. Tier 3's CRITERIA block spells the
# same five out in prose; add a line there and add its key here, or `accepted`
# starts approving a criterion nobody checked.
CRITERION_KEYS = ("api_cited", "code_runs", "errors_named",
                  "sentence_len", "no_invention")


def accepted(v: Verdict) -> bool:
    """`passed` is the model's claim; `criteria_met` is its evidence.

    Requiring both — and requiring the evidence to cover every criterion you
    asked about — is what stops a verdict of `{"criteria_met": {}, "passed":
    true}` from ending the loop with a draft nobody checked.
    """
    return (v.passed
            and all(k in v.criteria_met for k in CRITERION_KEYS)
            and all(v.criteria_met.values()))


def as_feedback(issues: list[str]) -> str:
    """`issues` is a list; a prompt is text. Render it once, here."""
    return "\n".join(f"- {i}" for i in issues)

The four fields, and why each is there

Field order matters here.

reasoning comes first because of how constrained decoding works. Constrained decoding is the mechanism that forces the model’s output to match the declared shape, by blocking any token that would break it (Structured output is a guarantee not a request). It generates fields in the order they are declared. Put passed first and you force the model to commit to a verdict before it has written down a single piece of evidence for one. Put reasoning first and the evidence exists before the verdict does.

criteria_met is a dict from criterion name to pass/fail. It reports each named criterion separately, so a failure says which criterion failed rather than just that something did.

passed is the single boolean the loop branches on.

issues is the list fed back to the generator. It exists because a generator told only “rejected” has nothing to change — it will resample a fresh draft rather than revise the one it has.

The two helpers, and why they ship with the schema

Both helpers are places the contract leaks if you skip them.

Gate on accepted, not on v.passed. Nothing in the schema ties the boolean to the dict — they are two independently generated fields. So this is perfectly valid output:

{"reasoning": "", "criteria_met": {}, "passed": true, "issues": []}

Schema-valid, and it ends the loop on round 1 against an empty criteria dict. accepted is the one line that makes criteria_met load-bearing rather than decorative: it demands the claim (passed), demands the evidence covers all five keys, and demands every one of those five is True.

Render with as_feedback, not the raw list. issues is a list[str], and what goes into the next prompt is text. Interpolate the list directly into an f-string and the generator is shown this:

['add caveats', 'cite section']

That is a stringified Python list — brackets, quotes, and commas included — where you meant two bullet points. It is one line to fix, and both tiers below would otherwise get it wrong in the same way.

Tier 2 — LangGraph

LangGraph is a library that expresses an agent as a graph. Nodes are ordinary functions. Edges are the allowed transitions between them. The framework runs the graph, carrying a shared state dictionary from node to node: each node reads the state, returns a partial update, and the framework merges that update in.

The state’s keys and their types are declared up front with a TypedDict — a Python class that describes the shape of a dictionary (which keys it has, and what type each value is) without changing the fact that it is a plain dict at runtime. The S class in the listing below is exactly that: the loop’s state, spelled out.

Verdict, accepted, and as_feedback are the ones from the section above, reused rather than repeated. The code below runs that section’s copies of the helpers.

The stand-in that makes the listing runnable

Neither langgraph nor langchain-anthropic is a dependency of this repo. The block below registers a stand-in — a fake module that satisfies the imports — under both names, unconditionally, so every machine runs the same thing rather than whatever happens to be installed.

The stand-in models exactly four graph semantics, and nothing else:

1. A node is a function from state to a partial state dict.
2. The runner merges each returned dict into the state.
3. The router returns the name of the next node.
4. A key the schema does not declare never becomes part of the state.

Checkpointing, resume, and streaming — the reasons point 1 below gives for using a graph framework — are absent. This bounds what the next section can claim: what runs here is this chapter’s node and routing logic, not the framework.

Skip this block on a first read. It is scaffolding; the pattern is the listing after it.

# A stand-in for the two imports in the listing below. It is a model of four
# graph semantics, NOT LangGraph, and it never calls a model.
import sys
import types

START, END = "__start__", "__end__"

CALLS: list = []      # "generate" / "evaluate", in the order the graph makes them
PROMPTS: list = []    # the prompt text each of those calls was handed
INITIAL: list = []    # the state dict every invoke() was seeded with


def canned(passed: bool, issues=("add caveats", "cite section"), met=None) -> Verdict:
    """One scripted verdict. `met=None` means 'evidence agrees with the claim'."""
    return Verdict(reasoning="cited section 4.2", passed=passed, issues=list(issues),
                   criteria_met=dict.fromkeys(CRITERION_KEYS, passed) if met is None
                                else met)


# Verdicts the judge returns, oldest first. Four of them, and the demo run at
# the end of the listing below should need exactly one: a loop that runs on
# past a passing verdict has to be caught by a named assertion in the next
# section, not by this stub running out of replies.
SCRIPT = [canned(True, issues=[])] * 4


class MiniChat:
    """Records the prompt, returns the next scripted reply. No network."""

    def __init__(self, **kw):
        self.schema = None                       # set by with_structured_output

    def with_structured_output(self, schema):
        self.schema = schema
        return self

    def invoke(self, prompt):
        PROMPTS.append(prompt)
        if self.schema is None:                  # the generator: replies are text
            CALLS.append("generate")
            return types.SimpleNamespace(content=f"d{CALLS.count('generate') - 1}")
        CALLS.append("evaluate")                 # the judge: replies are Verdicts
        if not SCRIPT:
            raise AssertionError("the judge ran more rounds than the script allows")
        return SCRIPT.pop(0)


class MiniStateGraph:
    """Nodes, one conditional branch, and a merge. That is the whole model."""

    def __init__(self, schema):
        self.schema, self.nodes, self.edges, self.branch = schema, {}, {}, {}

    def add_node(self, name, fn):
        self.nodes[name] = fn

    def add_edge(self, src, dst):
        self.edges[src] = dst

    def add_conditional_edges(self, src, router, allowed):
        self.branch[src] = (router, list(allowed))

    def compile(self):
        return self

    def invoke(self, state, max_steps=64):
        state = dict(state)          # NOTHING is defaulted: a missing key is a KeyError
        INITIAL.append(dict(state))  # recorded so the next section can read the seed
        node = self.edges[START]
        for _ in range(max_steps):   # a graph that never reaches END is an error here,
            if node == END:          # not a hang
                return state
            fn = self.nodes[node]
            for k, v in (fn(state) or {}).items():
                if k in self.schema.__annotations__:   # a key the schema does not
                    state[k] = v                       # declare is dropped, not merged
            if node in self.branch:
                router, allowed = self.branch[node]
                node = router(state)
                assert node in allowed, node
            else:
                node = self.edges[node]
        raise AssertionError("the graph never reached END")


sys.modules["langgraph"] = types.ModuleType("langgraph")
sys.modules["langgraph.graph"] = types.ModuleType("langgraph.graph")
sys.modules["langgraph.graph"].StateGraph = MiniStateGraph
sys.modules["langgraph.graph"].START = START
sys.modules["langgraph.graph"].END = END
sys.modules["langchain_anthropic"] = types.ModuleType("langchain_anthropic")
sys.modules["langchain_anthropic"].ChatAnthropic = MiniChat

The listing

The pattern, in four parts: the two model clients, the state class S, the three functions (generate, evaluate, route), and the graph wiring that connects them.

from typing import TypedDict
from langgraph.graph import StateGraph, START, END
from langchain_anthropic import ChatAnthropic

# max_tokens is a truncation ceiling, not a budget: a reply that reaches it
# comes back cut off mid-sentence with `stop_reason: "max_tokens"`. 8192 for a
# draft, 2048 for a verdict that is four short fields.
llm = ChatAnthropic(model="claude-opus-5", max_tokens=8192)
# with_structured_output is what makes v.issues / v.passed exist — without it,
# judge.invoke returns text and you are back to parsing prose.
judge = ChatAnthropic(model="claude-opus-5", max_tokens=2048).with_structured_output(Verdict)

class S(TypedDict):
    task: str
    draft: str
    feedback: str         # text, not v.issues — see as_feedback above
    rounds: int
    passed: bool          # written by evaluate, read by route — must be declared

def generate(s: S) -> S:
    prompt = s["task"] if not s["draft"] else (
        f"{s['task']}\n\nPrevious:\n{s['draft']}\n\nFix:\n{s['feedback']}")
    return {"draft": llm.invoke(prompt).content, "rounds": s["rounds"] + 1}

def evaluate(s: S) -> S:
    v = judge.invoke(f"Task: {s['task']}\nDraft: {s['draft']}")
    return {"feedback": as_feedback(v.issues), "passed": accepted(v)}

def route(s: S) -> str:
    return END if s.get("passed") or s["rounds"] >= 3 else "generate"

g = StateGraph(S)
g.add_node("generate", generate)
g.add_node("evaluate", evaluate)
g.add_edge(START, "generate")
g.add_edge("generate", "evaluate")
g.add_conditional_edges("evaluate", route, ["generate", END])
app = g.compile()

# Seed EVERY key in S. LangGraph does not default a missing one, so
# `app.invoke({"task": "..."})` raises KeyError: 'draft' inside the first node,
# and adding only `draft` then raises KeyError: 'rounds'.
final = app.invoke({"task": "write the payments integration guide",
                    "draft": "", "feedback": "", "rounds": 0, "passed": False})
print(final["passed"], final["draft"][:80])

Four things about this block:

1. route is a conditional edge, not a while. That is the entire reason to reach for LangGraph here. The loop lives in the graph, so the retry edge evaluate → generate is a declared part of the structure rather than a line of imperative code — which means the framework can checkpoint state between nodes, resume a run that died on round 2, and stream node-by-node progress. A while loop in optimize() gets you none of that, which is why Tier 3 has to own its own state.
2. passed must be in S. Nodes return partial dicts that get merged into the state, so evaluate writing {"passed": ...} and route reading s.get("passed") only work if the key is declared — a key missing from the TypedDict is a type error, and at runtime route silently reads None and loops until the round cap.
3. route checks rounds >= 3 in the same expression as passed. The cap is part of the edge condition, not a separate guard, so there is no path through the graph that iterates without it. This is the oscillation fix from Evaluatoroptimizer expressed as graph structure rather than as discipline.
4. The cap counts generates here and evaluates in Tier 3 — and lands on the same number. rounds is incremented by generate, and route runs after evaluate, so a cap of 3 buys three generates and three evaluates: six calls, and the last draft is judged before route sees it. Tier 3 counts rounds of judging instead and stops at the same six. Same loop, different fencepost, same bill — and in both, the draft that comes out the end has a verdict attached to it.

Walk one pass through the graph:

generate  rounds 0 -> 1, writes d0     route not consulted (unconditional edge)
evaluate  d0 -> fail                   route: passed False, rounds 1 < 3 -> generate
generate  rounds 1 -> 2, writes d1
evaluate  d1 -> fail                   route: passed False, rounds 2 < 3 -> generate
generate  rounds 2 -> 3, writes d2
evaluate  d2 -> fail                   route: rounds 3 >= 3 -> END, returns d2

Three generates, three evaluates, and the returned draft d2 was evaluated on the step immediately before the graph ended. Same six calls as Tier 3, reached by counting a different thing.

Proving Tier 2’s logic holds

Those four points are claims about what the code does. The block below drives the graph above against the stand-in: it exercises the two node functions and the router, and nothing else. LangGraph is not under test here — the runtime underneath is the stand-in registered before the listing, so each assertion below proves the nodes and the routing are right, not the framework works.

Each assertion names the defect it pins, and every one was checked by reintroducing that defect and confirming the block fails. The exception is a seed cut down to task alone: it never reaches an assertion, because the listing above dies in its first node — the failure the try in assertion 2 catches, and the reason the honest place to catch a partial seed is the KeyError itself.

Before reading the assertions, know what the three recorders hold. CALLS is the ordered list of "generate" / "evaluate" strings, one per model call the graph made. PROMPTS is the text each of those calls was handed. INITIAL holds the state dict each invoke() started from — so INITIAL[0] is the exact seed the listing above ran with, which is what SEED reuses.

SEED = dict(INITIAL[0])       # the exact dict the listing above invoked with

# ---- 1. `passed` is declared in S, so evaluate's verdict reaches route.
#         Undeclared, the update is dropped on the merge, `route` reads None,
#         and a draft that passed on round 1 gets revised twice more anyway.
CALLS.clear(); PROMPTS.clear(); SCRIPT[:] = [canned(True, issues=[])] * 4
final = app.invoke(SEED)                  # 4 verdicts scripted, 1 legitimately used:
                                          # a loop that ignores `passed` runs on, and
                                          # is caught by an assertion, not by the stub
assert final["passed"] is True, "evaluate wrote `passed` and route never saw it"
assert CALLS == ["generate", "evaluate"], ("a passing verdict did not stop the "
                                           "graph", CALLS)
assert final["draft"] == "d0", final["draft"]

# ---- 2. every key in S is seeded before invoke. A partial initial state dies
#         in the first node, before the first model call.
assert set(SEED) == set(S.__annotations__), (
    "app.invoke seeds a different key set than S declares: "
    f"{sorted(set(S.__annotations__) ^ set(SEED))}")
try:
    app.invoke({"task": "write the payments integration guide"})
    raise AssertionError("a partial initial state was accepted")
except KeyError as e:
    assert "draft" in str(e), e

# ---- 3. the cap: three generates, three evaluates, and the draft that comes
#         out was the last one judged. 2N, the same six calls as Tier 3.
CALLS.clear(); PROMPTS.clear(); SCRIPT[:] = [canned(False)] * 8
final = app.invoke(SEED)
assert CALLS == ["generate", "evaluate"] * 3, ("the cap is not 3 rounds", CALLS)
assert final["rounds"] == 3, final["rounds"]
assert (final["passed"], final["draft"]) == (False, "d2"), final
assert "d2" in PROMPTS[-1], "the returned draft is not the one last judged"

# ---- 4. feedback reaches the next prompt as TEXT, not as a repr'd list.
CALLS.clear(); PROMPTS.clear(); SCRIPT[:] = [canned(False), canned(True, issues=[])]
final = app.invoke(SEED)
assert isinstance(final["feedback"], str), type(final["feedback"])
assert "- add caveats\n- cite section" in PROMPTS[2], PROMPTS[2]
assert "['add caveats'" not in PROMPTS[2], "the raw issues list reached the generator"
assert "d0" in PROMPTS[2], "the revision prompt does not carry the previous draft"

# ---- 5. the node calls `accepted`, so the claim alone does not end the loop.
CALLS.clear(); SCRIPT[:] = [canned(True, met={}, issues=[])] * 3
final = app.invoke(SEED)
assert final["passed"] is False, "an empty criteria_met ended the loop"
assert CALLS == ["generate", "evaluate"] * 3, CALLS

print("02a Tier 2: node and routing assertions hold")

What this deliberately does not claim:

• The merge rule is modelled, not verified. Point 2 says an undeclared key leaves route reading None; the stand-in drops undeclared keys because that is the behaviour being described, so assertion 1 proves the nodes depend on passed reaching the state, not that LangGraph drops it. Some versions raise instead — in which case the same edit fails louder, not quieter.
• Checkpointing, resume and streaming are untested. They are point 1’s entire argument for using a graph framework, and nothing here touches them. That claim rests on the library’s documentation.
• No model is called, so nothing here exercises truncation, refusal or a reply with no text in it. Tier 3 tests those on the SDK path, where the code that handles them lives; Tier 2 delegates them to langchain-anthropic.

Tier 3 — Anthropic SDK

Now the same loop with no framework at all — just the vendor’s Python client. Verdict, CRITERION_KEYS, accepted, and as_feedback are repeated here, unchanged, so this listing runs on its own.

It is the longest block in the chapter. Read it in five parts:

1. The schema and helpers, copied from above.
2. JUDGE_SYSTEM and CRITERIA — the judge’s instructions and the five criteria in prose.
3. first_text — the reply reader, which is where three failure modes are caught.
4. generate and evaluate — one model call each.
5. optimize — the loop itself, in eleven lines.

from __future__ import annotations   # `list[str] | None` needs this on 3.9

import anthropic
from pydantic import BaseModel, Field

client = anthropic.Anthropic()

class Verdict(BaseModel):
    reasoning: str = Field(description="Cite specific evidence before judging.")
    criteria_met: dict[str, bool]
    passed: bool
    issues: list[str]

CRITERION_KEYS = ("api_cited", "code_runs", "errors_named",
                  "sentence_len", "no_invention")

def accepted(v: Verdict) -> bool:                  # the claim AND the evidence
    return (v.passed
            and all(k in v.criteria_met for k in CRITERION_KEYS)
            and all(v.criteria_met.values()))

def as_feedback(issues: list[str]) -> str:
    return "\n".join(f"- {i}" for i in issues)

class NotAnAnswer(RuntimeError):
    """The reply stopped for a reason that is not a finished draft."""

JUDGE_SYSTEM = (
    "You are a strict evaluator. Judge the draft against every criterion. "
    "Report each unmet criterion as one concrete, actionable issue. "
    "Do not rewrite the draft."
)

# This is what "clear, articulable criteria" actually looks like. Each line is
# independently checkable and independently reportable — that is what makes
# `criteria_met: dict[str, bool]` meaningful instead of decorative. The five
# names here are CRITERION_KEYS, and the two lists have to move together.
CRITERIA = """<criteria>
api_cited:     Every claim about API behaviour cites a section number from the spec.
code_runs:     Every code sample is valid Python 3.9 and imports only the stdlib.
errors_named:  The error-handling section names all four documented failure codes.
sentence_len:  No sentence exceeds 40 words.
no_invention:  No feature is described that does not appear in the spec.
</criteria>"""

def first_text(r) -> str:
    """The only safe way to read a reply.

    Never `r.content[0]` and never a bare `next(...)` over it: a refusal
    arrives as an ordinary HTTP 200 whose content list is EMPTY, so both
    spellings crash the loop instead of reporting a decline.
    """
    if r.stop_reason == "refusal":
        raise NotAnAnswer(f"model declined: {getattr(r, 'stop_details', None)}")
    if r.stop_reason == "max_tokens":
        raise NotAnAnswer("truncated at max_tokens: raise the ceiling or split the task")
    text = next((b.text for b in r.content if b.type == "text"), None)
    if text is None:                   # thinking blocks only, or nothing at all
        raise NotAnAnswer(f"no text block in the reply ({r.stop_reason})")
    return text

def generate(task: str, draft: str = "", issues: list[str] | None = None) -> str:
    prompt = task if not draft else (
        f"{task}\n\n<previous_draft>\n{draft}\n</previous_draft>\n"
        f"<issues>\n{as_feedback(issues or [])}\n</issues>\n"
        "Produce a revised version that resolves every issue."
    )
    r = client.messages.create(
        model="claude-opus-5",
        max_tokens=8192,                       # truncation ceiling, not a budget
        messages=[{"role": "user", "content": prompt}],
    )
    return first_text(r)

def evaluate(task: str, draft: str) -> Verdict:
    r = client.messages.parse(
        model="claude-opus-5",
        max_tokens=2048,                       # a verdict is four short fields
        system=[{                              # byte-identical every round
            "type": "text",
            "text": JUDGE_SYSTEM + "\n\n" + CRITERIA,
            "cache_control": {"type": "ephemeral"},
        }],
        messages=[{"role": "user",
                   "content": f"<task>{task}</task>\n<draft>{draft}</draft>"}],
        output_format=Verdict,
    )
    return r.parsed_output

def optimize(task: str, max_rounds: int = 3) -> tuple[str, bool]:
    draft = generate(task)
    for round_no in range(1, max_rounds + 1):
        v = evaluate(task, draft)          # every draft that can be returned
        if accepted(v):                    # goes through here first
            return draft, True
        if round_no == max_rounds:
            return draft, False            # judged, and it failed. Never fake success
        draft = generate(task, draft, v.issues)
    return draft, False                    # max_rounds < 1: nothing was judged,
                                           # so this path can only ever be False

Seven points, each with its own subsection below.

1. reasoning is the first field in the schema

Constrained decoding generates fields in order (Structured output is a guarantee not a request), so putting the score first would force the model to commit before reasoning. Order is not cosmetic.

2. Branch on stop_reason before you touch content

stop_reason is the field on a reply that says why generation stopped. content is the list of blocks the model produced. The rule is: ask why generation stopped before you read what it produced — and never index into content blindly.

first_text exists because three of the endings are not a finished draft, and two of those carry no text block at all:

Two rows deserve a note. A refusal is not an HTTP error — it is a normal 200 response whose content list is empty, so the loop dies with a traceback about iteration instead of reporting that the model declined. And on claude-opus-5, adaptive thinking is on by default, so a reply routinely leads with a thinking block; a response truncated before it reaches any text is the same empty-generator crash again.

One helper, three failures.

3. The judge’s system prompt is marked for caching — and as written it will not cache

Start with the mechanism. Prompt caching lets the provider store its precomputed internal state for a block of prompt text, so later requests that begin with the same bytes skip recomputing it. The cache_control marker on the system block is what requests that. Because the block is byte-identical every round, rounds 2 and later would read the criteria at roughly a tenth of the normal input rate (Prompt caching derived).

The mechanism is right. The size is not.

A cached prefix has to clear the model’s minimum, which is 512 tokens on claude-opus-5. Here is the block being cached, measured:

137 against a floor of 512. The ~4-characters-per-token figure is a rule of thumb (Tokens), but it does not have to be exact — 549 characters is nowhere near 512 tokens under any tokenization, because that would require fewer than 1.1 characters per token.

Below the minimum, nothing caches — and there is no error. The only symptom is cache_creation_input_tokens: 0 in the usage block (Evaluatoroptimizer states the same caveat in as many words).

So: keep the marker. It costs nothing, and it starts paying the moment the criteria list grows to the size that makes the pattern worth using. But verify rather than assume — print usage.cache_read_input_tokens on round 2, and if it is zero you are paying full price for every round.

One more thing to know before you rely on it. "ephemeral" is the short-lived cache tier, and its default time-to-live is five minutes — which three claude-opus-5 calls at high effort can genuinely exceed. It is not “far longer than a revision loop lasts”; it is the same order of magnitude. A one-hour tier exists for exactly that reason, at a higher write price: writing a five-minute entry costs about 1.25× the base input rate, and writing a one-hour entry costs about 2×.

4. max_tokens is a truncation ceiling, not a budget

It is the one parameter here that silently changes behaviour. The model is not told about it, so it does not wrap up as it approaches the limit — it is simply cut off, and you get stop_reason: "max_tokens" with a half-finished draft. 8192 for the generator and 2048 for a four-field verdict are sized to that, and first_text is what stops a truncation being mistaken for a draft.

5. output_format guarantees the verdict parses

Invalid output has probability zero, because the runtime never lets an invalid token be chosen. No regular expressions patching up the text afterwards, and no repair loop. It guarantees shape, though, not intent — which is the next point.

6. accepted, not v.passed

Nothing in the schema ties the boolean to the dict, so {"reasoning": "", "criteria_met": {}, "passed": true, "issues": []} is a schema-valid verdict that ends the loop on round 1 having checked nothing. Gate on the claim and the evidence, and require the evidence to name all five criteria.

7. The failure path returns False, and the draft it returns has been judged

An agent that reports success on round 3 without passing is the most dangerous bug in this pattern. A loop that returns a draft generated after the last verdict is a quieter version of the same bug: the call is wasted and the artifact is unreviewed, though the flag stays honest. Exiting after the evaluation rather than after the revision fixes both.

Proving the guards hold

Six of the seven points above are claims about what the code does when something goes wrong. The block below stubs the client out — no network, no credentials — and drives each path.

Each assertion is written so that reverting the guard it pins makes it fail. That is a stricter bar than it sounds: a test that only checked “something was raised” would let all three stop_reason branches be deleted one at a time without ever going red, so each assertion below checks the specific message its branch produces.

The first half of the block is fixtures rather than tests — block, reply, fake, and verdict build canned responses and a client that records traffic into CALLS and JUDGED. The numbered comments mark where the actual assertions start.

import types

CALLS: list = []          # every API call the loop makes, in order
JUDGED: list = []         # the prompt every evaluator actually saw
_real_client = client

def block(kind: str, text: str = ""):
    b = types.SimpleNamespace(type=kind)
    if kind == "text":
        b.text = text                      # a thinking block has no .text at all
    return b

def reply(stop_reason: str, *blocks):
    return types.SimpleNamespace(stop_reason=stop_reason, content=list(blocks),
                                 stop_details=None)

def fake(replies, verdicts=()):
    """A client that hands back canned replies and records the traffic."""
    rs, vs = list(replies), list(verdicts)

    def create(**kw):
        CALLS.append("generate")
        return rs.pop(0)

    def parse(**kw):
        CALLS.append("evaluate")
        JUDGED.append(kw["messages"][0]["content"])
        return types.SimpleNamespace(parsed_output=vs.pop(0))

    return types.SimpleNamespace(
        messages=types.SimpleNamespace(create=create, parse=parse))

def verdict(passed, met=None, issues=("add caveats", "cite section")):
    return Verdict(reasoning="cited section 4.2", passed=passed, issues=list(issues),
                   criteria_met=dict.fromkeys(CRITERION_KEYS, True) if met is None
                                else met)

drafts = lambda n: [reply("end_turn", block("text", f"d{i}")) for i in range(n)]
FAILS = [verdict(False, met=dict.fromkeys(CRITERION_KEYS, False))] * 9

# ---- 1. refusal: HTTP 200, EMPTY content. A bare next() raises StopIteration.
client = fake([reply("refusal")])
try:
    generate("t")
    raise AssertionError("a refusal was read as a draft")
except NotAnAnswer as e:
    assert "declined" in str(e), e

# ---- 2. max_tokens: a truncated draft is not a draft.
client = fake([reply("max_tokens", block("text", "half a dra"))])
try:
    generate("t")
    raise AssertionError("a truncated draft was returned as finished")
except NotAnAnswer as e:
    assert "max_tokens" in str(e), e

# ---- 3. thinking-only content: opus-5 thinks by default, so the first block
#         is routinely not text, and a truncation before the text is empty too.
client = fake([reply("end_turn", block("thinking"))])
try:
    generate("t")
    raise AssertionError("a reply with no text block was read as a draft")
except NotAnAnswer as e:
    assert "no text block" in str(e), e

# ---- 4. the draft that comes out has been judged, and no call is wasted.
CALLS.clear(); JUDGED.clear()
client = fake(drafts(9), FAILS)
assert optimize("t", max_rounds=3) == ("d2", False)
assert CALLS == ["generate", "evaluate"] * 3, CALLS       # 2N, not 2N+1
assert "<draft>d2</draft>" in JUDGED[-1], "returned a draft no evaluator saw"

# ---- 5. `passed` is a claim; `criteria_met` is the evidence.
client = fake(drafts(9), [verdict(True, met={}, issues=[])] * 3)     # empty dict
assert optimize("t", max_rounds=3) == ("d2", False), "empty criteria_met passed"
client = fake(drafts(9), [verdict(True, met={"api_cited": True}, issues=[])] * 3)
assert optimize("t", max_rounds=3) == ("d2", False), "one-of-five criteria passed"
client = fake(drafts(9), [verdict(True, met={**dict.fromkeys(CRITERION_KEYS, True),
                                             "no_invention": False}, issues=[])] * 3)
assert optimize("t", max_rounds=3) == ("d2", False), "a failing criterion passed"
client = fake(drafts(1), [verdict(True, issues=[])])   # and a real pass passes,
assert optimize("t", max_rounds=1) == ("d0", True)     # or the gate is `return False`

# ---- 6. the step cap halts, at every cap including 0 and negative.
for cap in (0, -1, 1, 3):
    CALLS.clear()
    client = fake(drafts(9), list(FAILS))
    _, ok = optimize("t", max_rounds=cap)
    assert ok is False, cap
    assert len(CALLS) == max(1, 2 * cap), (cap, CALLS)

# ---- 7. feedback reaches the prompt as text, not as a repr'd list.
assert as_feedback(["add caveats", "cite section"]) == "- add caveats\n- cite section"
assert isinstance(as_feedback([]), str)

client = _real_client
print("02a guards: all assertions hold")

Three of those blocks are worth a note.

Assertion 4 pins the call sequence, not just the answer. CALLS == ["generate", "evaluate"] * 3 is what catches the wasted call: the loop that revises on its way out produces a seventh entry and returns d3, a draft whose only reader is the caller. Asserting on the returned tuple alone would let that regress the moment someone “simplified” the exit.

Assertion 5 asserts a pass as well as three failures. Three tests that all expect False are equally satisfied by a gate that reads return False, which is not a gate at all. The fourth case — a genuinely passing verdict that must come back True — is what keeps the other three honest.

Assertion 6 covers the step cap. It is the one guard in this file that could not be defeated: for round_no in range(1, max_rounds + 1) halts at every cap tried, including 0 and -1, where the loop body never runs and the function returns after a single generate with passed=False. That is why the expected call count is max(1, 2 * cap) — at cap 0 or -1 there is exactly one call (the pre-loop generate) and no evaluate at all, and at caps 1 and 3 it is the usual 2N. There is no path through optimize that iterates without the cap, and no cap value that turns the honest False into a True.

The criteria are the design decision

The last thing to say is about the CRITERIA block itself. This is where the advice in Evaluatoroptimizer — if a criterion can be checked in code, check it in code — stops being a slogan and becomes a line-by-line decision.

Two of those five criteria should never reach the model.

code_runs is ast.parse — the standard-library call that parses Python and raises if the syntax is invalid — plus a check of what the sample imports. sentence_len is a regular expression, a pattern matched over the text. Both are a few lines of ordinary code.

Checking them in code buys three things. They become deterministic — same input, same verdict, every time. They become free — no tokens, no latency. And they become immune to a judge having an opinion about whether a 41-word sentence is really that bad.

It also shrinks the judge’s job down to the three criteria that genuinely need reading comprehension: api_cited, errors_named, no_invention.

Why the named keys make this incremental

You do not have to move all of it at once, and the criteria_met dict is what lets you move it one key at a time.

The same dict can be filled by code for some keys and by the model for others. accepted reads the merged dict and cannot tell the difference — which is the point. Concretely, a partially migrated round looks like this:

from code:   {"code_runs": True, "sentence_len": False}
from judge:  {"api_cited": True, "errors_named": True, "no_invention": True}
merged:      all five keys present -> accepted() returns False (sentence_len)

This is also why accepted insists every key in CRITERION_KEYS is present. A key that no one filled in is a criterion nobody checked, and silently dropping it would turn a five-criterion gate into a four-criterion one.

Keep the code-checked lines in the prompt anyway. The generator reads the same criteria block, and a criterion the generator never sees is one it will keep violating — you would be catching the violation in code instead of preventing it.

One consistency note

The criteria are about someone else’s code, and this file is code too. The listing above targets Python 3.9 — which is why from __future__ import annotations is there to make list[str] | None legal, and why code_runs asks for valid 3.9 rather than 3.10.

Pick one floor and let every line in the file agree with it. Two floors in one file is how a criterion ends up passing a sample the code cannot import.

What interviewers probe: “How do you know when to stop iterating?” and “What if the evaluator is wrong?” Cap the rounds and prefer code-checkable criteria. Calibrate the judge against ~50 human labels — score the same 50 items by hand, compare, and report how often the judge agreed — before trusting it (Calibration what an agreement rate actually means).

Next: 03 — Tools & MCP. The pattern this file builds is catalogued alongside the other eight in chapter 02.