Case Study 05 — Autonomous Agent

The design problem: an agent that works unattended for eight hours with no human in the loop, and produces a report you can trust in the morning.

An agent, here, is a program that loops. It sends text to a large language model (an LLM — a model that predicts the next chunk of text and can ask for tools to be run on its behalf). It runs the tools the model asks for. It feeds the results back. Then it repeats.

The loop is straightforward. Everything that makes an unsupervised eight-hour run survivable sits outside the loop, in five mechanisms:

This chapter covers how to write each of those five, how to derive the cost of a night from token counts (about $22), and when not to build this at all.

The problem, as input and output

Fix exactly what the agent receives and what it must produce. Every later decision follows from those two things.

What goes in. Three things.

1. A goal, written in plain English.
2. A budget with two dimensions: money, and wall-clock time. Wall-clock time means real elapsed time on a clock on the wall — eight hours is eight hours whether the agent worked hard or sat waiting on a slow test suite.
3. A success predicate. This one decides whether the project is viable at all. A predicate is a function that returns true or false; a success predicate answers “is the goal met?” by running code, not by asking for an opinion.

What comes out. Three things.

1. The work itself: a branch of code changes, files written, data cleaned.
2. A structured report: which of the seven possible endings the run reached, what it spent, every task it completed and verified, every task it gave up on and why, and how far it drifted from the goal over time.
3. An exit code — the small integer a program returns when it finishes — so a script that wakes up at 9am can tell success from failure without reading prose.

Here is all of that filled in for one overnight goal. The IN half is what you hand the harness at submit time; the OUT half is what the harness produces by morning.

IN   goal       "Reduce p95 latency on /api/search to under 200ms."
     budget     $28 hard cap, 8 hours of wall clock
     predicate  a Python function the HARNESS runs:  p95_ms("/api/search") < 200

OUT  status     one of GOAL_MET, BUDGET_EXHAUSTED, TIME_LIMIT, DRIFT_DETECTED,
                NO_PROGRESS, QUEUE_EMPTY_GOAL_UNMET, UNRECOVERABLE_ERROR
     exit code  0 success, 2 partial, 3 failure, 4 unrecoverable
     work       a git branch, one commit per verified task
     report     verified tasks, parked tasks with reasons, spend, drift history

Two terms in that block need definitions before the rest of the chapter.

p95 latency is the response time that 95% of requests come in under. The slowest 5% are worse than this. It measures the bad-but-not-freak case rather than the average, which is why teams set targets on it instead of on the mean.

The harness is the ordinary, non-model program that owns the loop. It holds the goal, keeps the money ledger, writes the checkpoints, and runs the success predicate. When this chapter says “the harness does X, not the agent”, it means X is Python the model cannot reach through any tool. That distinction — harness code versus anything the model can influence — drives every design decision below.

In, on, and out of the loop

“No human in the loop” needs a precise definition, because this chapter is an argument about it.

Human in the loop means a person sits inside the agent’s control flow. The run pauses, a human approves or corrects, and only then does it continue.

No human in the loop means nobody is inside the control flow at all. Every mechanism in this chapter exists to replace one of the things that person would have done.

Human on the loop is a third, weaker thing, and the names are close enough that people mix them up. A person watches the run and can intervene, but the run never waits for them. The supervised hybrid offered in Alternatives is human-on-the-loop with a review queue, and it is the design to reach for whenever the constraint permits.

Why this is hard: nobody is watching for eight hours. With no human correcting it, small errors compound, and the agent drifts toward tasks that feel productive but no longer serve the goal.

First thing to say: “Fully autonomous is usually the wrong choice, and I’d say so to the stakeholder. It’s survivable only when three things are true: actions are reversible, success is machine-checkable, and there’s a hard budget ceiling. If any is missing, I’d build a supervised agent instead.”

Then design it anyway. Flagging the risk and delivering the design is better than refusing.

The three preconditions, and what each one is standing in for

Each precondition substitutes for a specific thing a human supervisor would otherwise provide. Naming that substitution is what makes the argument derived rather than recited.

The middle row decides most cases. Machine-checkable means a program can answer “done?” — a test suite that goes green, a query that returns under 200ms, a checksum that matches. If the only available judge is a person’s taste, the agent is evaluating its own output, and a model asked “did you succeed?” is not consulting evidence; it is producing the text a successful assistant would produce.

If you cannot write the success predicate as code, the task is not a candidate for autonomy. That single test disqualifies most tasks people want to run overnight, and stating it early is the strongest opening in this interview.

Architecture

One lap of the loop, walked end to end, gives every later mechanism — termination, drift, checkpointing, budgets — a place to attach.

Read the diagram top to bottom: a task comes off the queue, runs, is verified, is checkpointed, and then passes three gates (drift, budget, goal) before the next lap starts.

flowchart TD
    G([Goal + budget]) --> INIT[Seed task queue]
    INIT --> Q[[Queue]]
    Q --> PICK[Pop highest-value task]
    PICK --> MARK[Mark IN_PROGRESS<br/>+ checkpoint]
    MARK --> EX[Execute]
    EX --> VER{Machine-checkable<br/>verification}
    VER -->|pass| CP[Checkpoint state]
    VER -->|fail| RETRY{Retries left?}
    RETRY -->|yes| EX
    RETRY -->|no| PARK[Park task + record why]
    PARK --> CP
    CP --> DRIFT{Drift check<br/>vs ORIGINAL goal}
    DRIFT -->|drifted| HALT([Halt + alert])
    DRIFT -->|ok| BUD{Budget}
    BUD -->|over| STOP([Halt + report])
    BUD -->|ok| DONE{Goal predicate<br/>run by the HARNESS}
    DONE -->|yes| FIN([Done + report])
    DONE -->|no, queue empty| FAIL([FAILURE + report])
    DONE -->|no, queue non-empty| REPLAN[Update queue] --> Q

    style DRIFT fill:#bc6c25,color:#fff
    style HALT fill:#9d0208,color:#fff
    style STOP fill:#bc6c25,color:#fff
    style FIN fill:#2d6a4f,color:#fff
    style FAIL fill:#9d0208,color:#fff
    style CP fill:#7209b7,color:#fff
    style MARK fill:#7209b7,color:#fff

The colour legend

Four of these hex values are borrowed from the system-design key, and every one of them means something different here — so read this legend, not that one. The meanings below hold for all five diagrams in this chapter.

Nothing here uses colour to mean irreversible, which is what red means in that key. Every ending in this chapter is a halt, and halting is precisely the reversible thing.

One lap, box by box

The goal and budget arrive, and the harness turns the goal into a starting task queue — a plain list of concrete units of work, held by the harness and refilled as the run learns things.

Each lap pops the task with the highest value to the goal. Before executing anything, the Mark IN_PROGRESS + checkpoint box writes down which task is about to run, so a crash mid-task leaves a record of exactly what was in flight. The matching Checkpoint state box writes the result afterwards. Two purple boxes, two disk writes, one before and one after.

The task then executes, and its result goes to a verification step that is machine-checkable rather than a self-assessment: a test, a query, a file that either exists or does not. A failure is retried while retries remain. When they run out, park task + record why sets the task aside with a written reason instead of looping on it forever.

Whether the task passed or was parked, the state is checkpointed. Then two guards fire before the next lap:

• The drift check compares the queue against the original goal, and halts if the run has wandered.
• The budget check halts if the money is gone.

Only after both decline does the harness — never the model — run the goal predicate. Three outcomes:

• It passes: the run is done.
• It fails and the queue is empty: the run ran out of tasks before it met the goal. That is a failure, not a finish; the empty-queue trap below explains why.
• It fails with the queue non-empty: the replan step (Update queue) adds what was learned, and the loop takes another lap.

What makes this more than a ReAct loop

ReAct is reason and act, the standard pattern where a model alternates between thinking a step and calling a tool (React reason act). The inner loop above is exactly that. Five additions distinguish this design from a bare ReAct loop:

1. Verification after every task — machine-checkable, not the model’s own opinion.
2. A checkpoint before and after every task — the two purple boxes.
3. A drift check against the immutable original goal.
4. Dual budgets — one the model can see, one it cannot.
5. A termination set in which every exit is reported, including the failures.

Termination

Every way the run is allowed to end has a name. Treating those endings as a formal set, rather than as scattered if statements, is what makes the morning report honest.

The seven endings are the whole set:

STOP = { GOAL_MET, QUEUE_EMPTY_GOAL_UNMET, BUDGET_EXHAUSTED,
         TIME_LIMIT, DRIFT_DETECTED, NO_PROGRESS, UNRECOVERABLE_ERROR }

The set needs two properties. Each one is a constraint on how the loop is written, not a comment above it.

Property 1: the set is exhaustive

Exhaustive means every path out of the loop matches exactly one member of that set. No exit that is not on the list.

Writing the loop as while True with a single unconditional stop check at the top gets you most of the way. No break buried in a branch. No loop condition that can quietly end the run.

It does not get you all the way, and this is the part that is usually missed: an exception raised inside the loop body is also a path out. On an eight-hour run it is the likeliest one, because the loop body is where the tools and the model calls live.

while True alone leaves the agent one uncaught BudgetExceeded away from an ending that has an exit code in the table and a traceback in reality. Exhaustive means the exception path is classified too, which is what the try in the code below is for.

Property 2: the set is prioritized, and therefore disjoint

Several members can be true at the same instant. A run can exhaust its budget on the same lap that it meets its goal. So the order of the checks decides which one gets reported.

GOAL_MET is checked first, so a run that reaches the goal on its last dollar reports success rather than BUDGET_EXHAUSTED. Checking in a different order would report a different ending from the same facts, which is why the order is code, not convention.

That order is exactly what the next diagram shows: six checks in sequence, each one falling through to the next.

flowchart TD
    T([Top of loop]) --> C1{goal predicate<br/>passes?}
    C1 -->|yes| S1["GOAL_MET<br/>SUCCESS · exit 0"]
    C1 -->|no| C2{ledger refused, or<br/>spend >= hard cap?}
    C2 -->|yes| S2["BUDGET_EXHAUSTED<br/>PARTIAL · exit 2 · alert"]
    C2 -->|no| C3{elapsed > wall clock?}
    C3 -->|yes| S3["TIME_LIMIT<br/>PARTIAL · exit 2 · alert"]
    C3 -->|no| C4{drift > 0.5?}
    C4 -->|yes| S4["DRIFT_DETECTED<br/>FAILURE · exit 3 · page"]
    C4 -->|no| C5{5 tasks, no change?}
    C5 -->|yes| S5["NO_PROGRESS<br/>FAILURE · exit 3 · alert"]
    C5 -->|no| C6{queue empty?}
    C6 -->|yes| S6["QUEUE_EMPTY_GOAL_UNMET<br/>FAILURE · exit 3 · alert"]
    C6 -->|no| RUN[Run next task]

    style S1 fill:#2d6a4f,color:#fff
    style S2 fill:#bc6c25,color:#fff
    style S3 fill:#bc6c25,color:#fff
    style S4 fill:#9d0208,color:#fff
    style S5 fill:#9d0208,color:#fff
    style S6 fill:#9d0208,color:#fff

The diagram is that priority order, top to bottom: goal met, then money, then time, then drift, then no-progress, then an empty queue. Only when all six checks decline does the loop run the next task.

Each ending box carries three attachments, and they are three different audiences:

• A class — SUCCESS, PARTIAL or FAILURE — which says how bad it is, for a human skimming the report.
• An exit code, for whatever script reads the outcome.
• An alerting action. Alert means a message to a channel someone reads in the morning. Page means waking someone up now. Only drift gets a page, because drift is the one ending where the run was actively spending money on the wrong problem.

The table below is the same seven endings with those attachments spelled out, plus the one thing each report must contain. Three of the rows are bolded because they are the ones interviewers push on.

Four cells in that table use shorthand worth expanding before you read the code.

• The ledger is the harness’s running tally of money spent, built in Budget ceilings below.
• The spend curve is that tally plotted over time. It tells you whether the money went on steady work or on one runaway task.
• The diff is the complete set of changes the run made, as a line-by-line comparison against the state it started from.
• K, in the NO_PROGRESS row, is the streak length: how many consecutive tasks may change nothing verifiable before the run is declared stuck. It is 5 in the code below.

One row in that table has no if anywhere in should_stop, and that is deliberate: UNRECOVERABLE_ERROR is the exception ending. Nothing returns it. It is produced by the except clause in run below, which catches what Goal.assert_intact and both refusal branches of Checkpoint.load raise. Write the loop without that clause and the row is advertising an exit code that no code path can reach — which is what the module-level set-equality check exists to make impossible.

The loop in code

The priority order from the diagram is now literally the function body of should_stop, read top to bottom. Two names to notice on the way past: state.budget.blocked is a flag the ledger sets when it refuses a call (the Budget ceilings section explains why the flag has to exist), and state.harness_goal_predicate is the success test, held by the harness, with no tool pointing at it.

from typing import Optional

STOP = frozenset({"GOAL_MET", "QUEUE_EMPTY_GOAL_UNMET", "BUDGET_EXHAUSTED",
                  "TIME_LIMIT", "DRIFT_DETECTED", "NO_PROGRESS",
                  "UNRECOVERABLE_ERROR"})

DRIFT_EVERY = 5                                 # tasks between drift checks

class BudgetExceeded(Exception):
    """Raised by Ledger.reserve BEFORE a call that would cross the hard cap."""

def should_stop(state) -> tuple[bool, str]:
    """Exhaustive and prioritized. Order is load-bearing."""
    if state.harness_goal_predicate():          # the HARNESS runs it, not the agent
        return True, "GOAL_MET"
    if state.budget.blocked or state.budget.spent >= state.budget.hard_cap:
        return True, "BUDGET_EXHAUSTED"         # `blocked`: reserve() already refused
    if state.elapsed > state.wall_clock_limit:
        return True, "TIME_LIMIT"
    if state.drift_score > 0.5:
        return True, "DRIFT_DETECTED"
    if state.no_progress_streak >= 5:
        return True, "NO_PROGRESS"
    if not state.queue:
        return True, "QUEUE_EMPTY_GOAL_UNMET"   # NOT success — report it as such
    return False, ""

STOP_CLASS = {
    "GOAL_MET": "SUCCESS",
    "BUDGET_EXHAUSTED": "PARTIAL",
    "TIME_LIMIT": "PARTIAL",
    "DRIFT_DETECTED": "FAILURE",
    "NO_PROGRESS": "FAILURE",
    "QUEUE_EMPTY_GOAL_UNMET": "FAILURE",
    "UNRECOVERABLE_ERROR": "FAILURE",
}

# At import, and in BOTH directions. A reason with no class and a class with no
# producer are different bugs, and only the second one catches a stop reason
# that is advertised in the table and returned by nothing.
assert STOP == set(STOP_CLASS), STOP ^ set(STOP_CLASS)

def run(state, drift_check=None):
    while True:
        try:
            stop, reason = should_stop(state)
            if stop:
                assert reason in STOP_CLASS, f"unclassified stop: {reason}"
                return report(state, reason, STOP_CLASS[reason])
            execute_next(state)
            if drift_check and state.step % DRIFT_EVERY == 0:
                drift_gate(state, drift_check(state.goal, state.queue))
        except BudgetExceeded:              # reserve() refused the NEXT call. Do not
            state.budget.blocked = True     # classify it here: let the top of the
            continue                        # loop report BUDGET_EXHAUSTED, exit 2
        except AssertionError:
            raise                           # the tripwire above stays loud
        except Exception as e:              # EVERY other way out, and classified
            return report(state, "UNRECOVERABLE_ERROR",
                          STOP_CLASS["UNRECOVERABLE_ERROR"], detail=repr(e))

Two things in that listing are easy to under-read.

The try is what makes the set exhaustive. Without it, while True is not the only exit: any exception out of execute_next is a second one, and it is the common one, because execute_next is where the tools and the model calls are.

UNRECOVERABLE_ERROR is the seventh advertised ending and it has no other producer. Goal.assert_intact and both refusal branches of Checkpoint.load raise RuntimeError, and this except is the only thing that turns those into a classified, reported ending rather than a traceback in a log nobody is reading at 3am.

BudgetExceeded gets its own branch above the catch-all because it is not unrecoverable. The ledger refused the next call before making it, so nothing is broken — the run is a PARTIAL with a full report, not a crash. That branch sets blocked and continues, letting the top of the loop classify the stop in the one place stops get classified.

The module-level assert STOP == set(STOP_CLASS) is the check that does the work, not the one inside run. The assert inside the loop only fires on the branch the run happens to take, at 3am, in production. The set equality fires at import, on every branch at once, in both directions — which is how you notice a class that has an exit code, a report contract and no code path that returns it.

Running it

The block above references _State, execute_next and report, which are not yet defined. The block below supplies stand-ins and exercises run six times, once per path. The numbered comments are the test names. Case 3 is the defect this design exists to fix: its assertions show that spent stops at $27.75 of a $28.00 cap and never reaches it. The block prints termination: 6 checks passed.

# --- a stand-in harness, so the loop above actually runs ------------------
class _Task:
    def __init__(self, tid): self.id = tid

class _Ledger:                          # the priced one, with PRICES, is below
    def __init__(self, cap):
        self.hard_cap, self.spent, self.blocked = cap, 0.0, False
    def reserve(self, worst: float) -> None:
        if self.spent + worst > self.hard_cap:
            self.blocked = True         # the flag should_stop reads
            raise BudgetExceeded(f"would reach ${self.spent + worst:.2f} "
                                 f"of ${self.hard_cap:.2f}")
    def record(self, actual: float) -> None:
        self.spent += actual

class _State:
    def __init__(self, *, tasks=200, cap=28.00, met=False, boom=None):
        self.budget = _Ledger(cap)
        self.queue = [_Task(f"t_{i}") for i in range(tasks)]
        self.elapsed, self.wall_clock_limit = 0.0, 8 * 3600
        self.drift_score, self.no_progress_streak, self.step = 0.0, 0, 0
        self.goal, self.pruned, self._met, self._boom = "reduce p95", [], met, boom
    def harness_goal_predicate(self): return self._met

def execute_next(state):
    if state._boom: raise state._boom
    state.budget.reserve(1.00)          # PRE-FLIGHT worst case, as the real one does
    state.budget.record(0.75)           # the actual, which is always less
    state.queue.pop(0)
    state.step += 1

def report(state, reason, cls, detail=None): return (reason, cls, detail)

# 1. the common path, and the priority order: goal-met is checked first
assert run(_State(met=True)) == ("GOAL_MET", "SUCCESS", None)

# 2. the queue drains with the goal unmet. A FAILURE, and it gets reported
assert run(_State(tasks=3)) == ("QUEUE_EMPTY_GOAL_UNMET", "FAILURE", None)

# 3. THE ADVERSARIAL ONE. reserve() refuses the call that WOULD cross the cap,
#    so `spent` never reaches `hard_cap` and the bare `spent >= hard_cap` test
#    is unreachable on its own. Before the `blocked` flag and the except
#    branch, this run ended in an uncaught BudgetExceeded: no report, no exit
#    code, and no STOP_CLASS entry for the thing that actually happened.
s = _State()
assert run(s) == ("BUDGET_EXHAUSTED", "PARTIAL", None)      # exit 2, with a report
assert s.budget.spent == 27.75 < s.budget.hard_cap          # never reached, by design
assert s.budget.blocked                                     # which is why the flag exists

# 4. anything else out of execute_next is the SEVENTH ending, which nothing
#    used to produce. Goal.assert_intact and Checkpoint.load both raise here.
reason, cls, detail = run(_State(boom=RuntimeError("goal text mutated")))
assert (reason, cls) == ("UNRECOVERABLE_ERROR", "FAILURE")
assert "goal text mutated" in detail

# 5. and the catch-all does NOT swallow the tripwire it sits under
STOP_CLASS.pop("QUEUE_EMPTY_GOAL_UNMET")
try:
    run(_State(tasks=1))
    raise SystemExit("an unclassified stop was reported as if it were classified")
except AssertionError as e:
    assert "unclassified stop: QUEUE_EMPTY_GOAL_UNMET" in str(e)
STOP_CLASS["QUEUE_EMPTY_GOAL_UNMET"] = "FAILURE"

# 6. the set equality, in both directions
def _agree(stop, classes) -> bool: return set(stop) == set(classes)
assert _agree(STOP, STOP_CLASS)
assert not _agree(STOP | {"TOOL_REVOKED"}, STOP_CLASS)            # reason, no class
assert not _agree(STOP, set(STOP_CLASS) | {"CREDS_REVOKED"})      # class, no producer
print("termination: 6 checks passed")

The empty-queue trap

The most common trap here is that the usual loop shape cannot report its most common failure:

def run_naive(state):
    while state.queue:             # <-- the bug is here, not in the reporting
        execute_next(state)
    return "done"

while queue: makes “empty queue, goal unmet” the normal exit. There is no branch to report it from, because the condition that should be a failure is the loop’s own termination condition. The bug is structural: you cannot report a state your control flow treats as success.

The cost of this shows up in a trace of the last two minutes of a run. Everything from 03:52:41 onward happens in the same second. The three lines that matter are queue length = 0, run.status = "completed", and the morning line underneath.

03:41:12  task t_33  "add index on search_events(created_at)"     PASS
03:52:40  task t_34  "verify p95 < 200ms on /api/search"          FAIL  (p95 = 410ms)
03:52:41  replan: no further optimizations identified
03:52:41  park t_34  reason="verification failed, no remaining ideas"
03:52:41  queue length = 0
03:52:41  loop exits
03:52:41  run.status = "completed"
03:52:41  report: "Completed 33 of 34 tasks. See diff."

morning:  p95 is 410ms, unchanged from 24 hours ago.
          The one task that measured the actual goal is the one that failed,
          and it is a footnote in a report headed "completed".

Thirty-three tasks passed and one failed, and the one that failed is the only one that measured the goal. The loop drained its queue, so it exited the way it always exits, and the status field says completed because that is the only thing a drained queue can mean in this shape.

An empty queue with an unmet goal is a failure, and an agent that reports it as success is worse than one that crashes, because a crash gets investigated and a false success does not.

Two further rules follow from the same idea. Both keep the verdict away from the thing being judged.

1. The harness runs the goal predicate, never the agent. state.harness_goal_predicate is a Python callable defined at submit time, and the agent has no tool that can influence it. If the agent could declare success, the entire termination set collapses to whatever it feels like saying.
2. Parked tasks are part of the report, not a log line. The most informative artifact of a failed run is the list of tasks it gave up on, each with its reason. That list is what tells you in the morning whether to retry, re-scope, or abandon.

Drift detection

A characteristic failure of autonomous agents is that the run slowly stops working on the thing you asked for, and only one kind of check catches it.

Drift is what happens when each self-generated task looks locally reasonable but the chain of them does not. Twenty steps in, the agent is optimizing something nobody asked for, having never taken a step you could point at as wrong.

The chain below is six tasks the agent generated for itself, left to right, starting from a goal of reducing p95 latency. The colour shifts gradually from on-goal green to off-goal red, and no single arrow is obviously wrong.

flowchart LR
    G["Original goal:<br/>reduce p95 latency"] --> T1["Profile endpoints ✓"]
    T1 --> T2["Add caching ✓"]
    T2 --> T3["Refactor cache layer ~"]
    T3 --> T4["Add cache metrics ~"]
    T4 --> T5["Build a metrics dashboard ✗"]
    T5 --> T6["Add dashboard auth ✗✗"]

    style T1 fill:#2d6a4f,color:#fff
    style T2 fill:#2d6a4f,color:#fff
    style T3 fill:#bc6c25,color:#fff
    style T4 fill:#bc6c25,color:#fff
    style T5 fill:#9d0208,color:#fff
    style T6 fill:#9d0208,color:#fff

Follow the chain one link at a time, and try to name the wrong step:

• Profile endpoints — exactly right. You cannot reduce latency you have not measured.
• Add caching — follows directly from what the profile showed.
• Refactor the cache layer — defensible cleanup of the thing just added.
• Add cache metrics — how you would confirm the cache is actually working.
• Build a metrics dashboard — a natural way to look at the metrics just added.
• Add dashboard auth — a login gate, which is what you do before anyone else opens a dashboard.

Every arrow is defensible. The chain is not. Adding a login page to a dashboard has nothing to do with p95 latency, and the run got there in six locally sensible moves.

No local check catches this, because drift is a property of the path, not of any step on it. A checker that looks at one task and asks “is this reasonable?” says yes to all six. That is why the comparison has to be against the origin — and why the origin has to be beyond the agent’s reach.

Why the goal must be immutable, shown as a trace

Put the goal somewhere the agent can edit and the detector stops working. It does not fail loudly; it keeps reporting healthy numbers while the run goes nowhere.

The trace below is that failure in four timestamps. t is the task number. drift_check(goal, queue) returns a score from 0.0 (fully aligned) to 1.0 (entirely unrelated), and anything under 0.3 is considered fine. Watch what the goal text says at t=0 versus t=17.

t=0    goal.txt  "Reduce p95 latency on /api/search to under 200ms."
       drift_check(read("goal.txt"), queue) -> 0.02   aligned

t=17   agent task t_17: "Update goal.txt to reflect refined understanding"
       goal.txt  "Improve observability of the search subsystem."
       (locally reasonable! it just spent four tasks on metrics)

t=20   queue: [build metrics dashboard, add dashboard auth, add SSO to dashboard]
       drift_check(read("goal.txt"), queue) -> 0.05   "fully aligned"

t=40   p95 on /api/search: 410ms, unchanged.
       drift_score never exceeded 0.08 for the entire run.
       Every alert threshold was respected. Nothing fired.

The goal here lives in a file the agent’s tools can write. At step 17 the agent edits it, and that edit is itself a locally reasonable task given the four metrics tasks it had just finished.

From then on the detector is comparing the queue against the new goal, which the queue matches perfectly. (SSO in that queue is single sign-on, the corporate login flow. It is three steps removed from anything about latency.)

Drift detection compared the queue against a goal that had already drifted, and correctly reported 0.0 the whole way down. The detector was not broken. It was pointed at a moving target.

Three defenses, and which one actually holds

There are three defenses, and all three are structural rather than instructional. None of them ask the model to behave.

They are built on a Goal object with no setter, hashed at construction so a change to the text becomes detectable. A hash here is a SHA-256 digest: a short fixed-length fingerprint of some text, with the property that different text almost certainly produces a different fingerprint. Store the fingerprint at construction, recompute it later, compare — if they differ, the text changed.

import hashlib

class Goal:
    """Loaded once at submit time. There is no setter."""
    def __init__(self, text: str):
        self._text = text
        self._sha = hashlib.sha256(text.encode()).hexdigest()

    @property
    def text(self) -> str:
        return self._text

    def assert_intact(self) -> None:
        if hashlib.sha256(self._text.encode()).hexdigest() != self._sha:
            raise RuntimeError("UNRECOVERABLE_ERROR: goal text mutated")

The block below runs four checks against assert_intact. The third one matters most: it defeats the hash in two lines, which is the limit of the mechanism. The block prints goal: 4 checks passed.

import pathlib, tempfile

g = Goal("Reduce p95 latency on /api/search to under 200ms.")
g.assert_intact()                                    # the clean case

# CONFIRMING: an in-process poke at the text alone is caught
bad = Goal("Reduce p95 latency on /api/search to under 200ms.")
bad._text = "Improve observability of the search subsystem."
try:
    bad.assert_intact()
    raise SystemExit("a mutated goal passed assert_intact")
except RuntimeError as e:
    assert "goal text mutated" in str(e)

# ADVERSARIAL, and this is the honest limit of the mechanism. `_sha` is derived
# from `_text` at construction and compared against a rehash of `_text`, so the
# check is self-referential: it catches a mutator that forgets `_sha`, and
# nothing else. Two lines defeat it.
worse = Goal("Reduce p95 latency on /api/search to under 200ms.")
worse._text = "Improve observability of the search subsystem."
worse._sha = hashlib.sha256(worse._text.encode()).hexdigest()
worse.assert_intact()                                # PASSES
assert worse.text != g.text                          # and the goal is now the wrong one

# ADVERSARIAL: the threat the trace above dramatises is an agent editing a
# FILE. assert_intact never opens one, so it cannot see that edit at all — and
# it does not need to, because defense 1 means there is no file to edit.
p = pathlib.Path(tempfile.mkdtemp()) / "goal.txt"
p.write_text("Improve observability of the search subsystem.")
g.assert_intact()                                    # unaffected: not on disk
assert p.read_text() != g.text
print("goal: 4 checks passed")

1. The goal lives in the harness, not on disk. If it is a file, it is inside a tool’s reach; if it is inside a tool’s reach, a locally reasonable task will eventually edit it. Load it once from the submission record. This is the defense that actually works, and the block above is why. A goal held in the harness is unreachable by any tool the agent has, and unreachability is a property of the tool allowlist, not of a check.
2. Hash it and assert on every drift check. Cheap, and it converts one class of silent corruption — an in-process write to _text that forgets _sha, from a bad merge or a buggy library — into UNRECOVERABLE_ERROR. It is not a defense against the agent: no tool call can reach a private attribute, and anything that can reach one can set both fields. It is a cheap secondary check behind the real lock, which is that no tool exists to reach the goal at all.
3. The drift check runs in a fresh context. Fresh context means the judging model call starts from an empty conversation: it receives only (original_goal, queue), never the agent’s reasoning, never the transcript. This is the same mechanism as the evaluator in Evaluatoroptimizer. A judge holding the agent’s justifications in context is predicting the continuation of a conversation in which those justifications were persuasive, and it will rationalize the chain it just read.

The check itself

The check is one model call with a constrained output shape. Structured output means the model is forced to emit fields matching a declared schema rather than free text, so the score comes back as a float you can compare against a threshold instead of a paragraph you would have to parse.

DriftCheck below is that schema, and check_drift is the call. Note the system prompt: it defines “off-goal” for the judge and tells it explicitly that it has not been given the agent’s reasoning, so it cannot assume the reasoning was sound.

from pydantic import BaseModel, Field

class DriftCheck(BaseModel):
    reasoning: str                              # first field, on purpose
    off_goal_task_ids: list[str]
    drift_score: float = Field(ge=0, le=1)

def check_drift(goal: Goal, queue: list) -> DriftCheck:
    goal.assert_intact()
    r = client.messages.parse(                  # FRESH context: no agent history
        model="claude-opus-5", max_tokens=2048,
        system=("Score how far the task queue has drifted from the ORIGINAL goal. "
                "A task is off-goal if completing it would not measurably advance "
                "the original goal, even if it seems useful. "
                "Judge each task against the goal only. You have not been given "
                "the agent's reasoning and must not assume it was sound. "
                "0.0 = fully aligned, 1.0 = entirely unrelated."),
        messages=[{"role": "user", "content":
                   f"<original_goal>{goal.text}</original_goal>\n"
                   f"<queue>{fmt(queue)}</queue>"}],
        output_format=DriftCheck,
    )
    return r.parsed_output

reasoning is the first field on purpose. Constrained decoding — the mechanism that forces output to match the schema — emits fields in schema order (Structured output is a guarantee not a request). Put drift_score first and you force the model to commit to a number before it has written a word of analysis. Put reasoning first and the number is conditioned on the analysis.

Cadence and thresholds

Cadence. Run the check every 5 tasks, not every task. It is a full Opus call — the most expensive model tier — and drift is a slow-moving property that will not appear and vanish inside five tasks. DRIFT_EVERY = 5 in the termination code is that cadence.

Thresholds. Two bands, and they do different things:

Both bands need code, and the prune half is the one that usually gets quoted and never written.

check_drift returns a score. drift_gate is the harness half that acts on it, and it is the only thing in the whole design that ever assigns state.drift_score — the value should_stop reads at the top of the next lap. run calls it every DRIFT_EVERY tasks.

The tasks it sheds go onto state.pruned, and they belong in the morning report next to the parked ones. A task the harness removed for drifting is at least as informative as one the agent gave up on.

The block below defines drift_gate and then runs four scenarios through the full run loop with a fake detector. The last scenario is the boundary case, and it is the one that catches quoting errors: both thresholds use a strict >, so exactly 0.5 prunes rather than halts. The block prints drift: 4 checks passed.

PRUNE_AT, HALT_AT = 0.3, 0.5

def drift_gate(state, dc: DriftCheck) -> None:
    """The harness half of the check: record the score, then act on the band."""
    state.drift_score = dc.drift_score           # nothing else assigns this
    if dc.drift_score > HALT_AT:
        return                                   # should_stop halts on the next lap
    if dc.drift_score > PRUNE_AT:                # prune band: shed and continue
        off = set(dc.off_goal_task_ids)
        state.pruned += [t for t in state.queue if t.id in off]
        state.queue = [t for t in state.queue if t.id not in off]

def _verdict(score, off=()):
    return DriftCheck(reasoning="...", off_goal_task_ids=list(off), drift_score=score)

OFF = ["t_5", "t_6", "t_7"]

# aligned: nothing pruned, nothing halted, the run ends the ordinary way
s = _State(tasks=8)
assert run(s, drift_check=lambda goal, q: _verdict(0.05))[0] == "QUEUE_EMPTY_GOAL_UNMET"
assert s.pruned == [] and s.drift_score == 0.05

# PRUNE band: the off-goal tasks leave the queue and the run keeps going
s = _State(tasks=8)
assert run(s, drift_check=lambda goal, q: _verdict(0.4, OFF))[0] == "QUEUE_EMPTY_GOAL_UNMET"
assert [t.id for t in s.pruned] == OFF and s.step == 5

# HALT band: the score is recorded, and should_stop reports it on the next lap
s = _State(tasks=8)
assert run(s, drift_check=lambda goal, q: _verdict(0.6, OFF))[:2] == ("DRIFT_DETECTED",
                                                                     "FAILURE")
assert s.pruned == []                    # a halted run does not silently reshape itself

# ADVERSARIAL: both thresholds are strict `>`. Exactly 0.5 does NOT halt, and
# exactly 0.3 does NOT prune. A detector calibrated to "0.5" and a loop that
# fires at 0.5 are two different systems; assert which one you shipped.
s = _State(tasks=8)
run(s, drift_check=lambda goal, q: _verdict(0.5, OFF))
assert s.drift_score == 0.5 and [t.id for t in s.pruned] == OFF   # pruned, not halted
s = _State(tasks=8)
run(s, drift_check=lambda goal, q: _verdict(0.3, OFF))
assert s.pruned == []                                             # not even pruned
print("drift: 4 checks passed")

The detector has its own failure mode, worth stating up front. It has a false-negative bias — it misses real drift more often than it invents fake drift — because a plausible task chain reads as reasonable. Calibrate it against 20 hand-labeled queues (10 aligned, 10 drifted at known severity), where calibrate means running the detector on inputs whose right answers you already know and measuring how often it agrees. Report that agreement rate before trusting a threshold. If it cannot separate the labeled sets, lower the threshold rather than shipping a detector you have not measured.

Checkpointing

Surviving interruption means saving the run’s state so that a crash costs minutes rather than the whole night. Less obviously, it means ensuring a half-written save file can never pass for a whole one, which is more dangerous than having no save at all.

An 8-hour run will be interrupted. A host reboot, a spot-instance reclaim (a cheap cloud machine taken back at short notice), an out-of-memory kill, a deploy. Design for resume from step one.

A checkpoint is a single file holding everything needed to pick the run back up: the queue, what finished, what was parked, what was in flight, and how much has been spent.

The two rows in the diagram are alternatives, not a sequence. The top row is the safe write: four steps ending in a committed checkpoint. The bottom row is the naive one-line write, which ends in a resume that is wrong without knowing it.

flowchart TD
    S[State in memory] --> W1["1. write state.json.tmp<br/>same filesystem"]
    W1 --> W2["2. f.flush() + os.fsync(fd)<br/>bytes are on the device"]
    W2 --> W3["3. os.replace(tmp, final)<br/>ATOMIC rename"]
    W3 --> W4["4. fsync the directory<br/>rename itself is durable"]
    W4 --> OK([Checkpoint committed])

    B1["Naive: open(final,'w').write(...)"] --> B2["crash at byte 184,301"]
    B2 --> B3["file exists, is truncated,<br/>and may still parse"]
    B3 --> BAD([Confident wrong resume])

    style W3 fill:#2d6a4f,color:#fff
    style OK fill:#2d6a4f,color:#fff
    style BAD fill:#9d0208,color:#fff

The top row, step by step:

1. Write the new state to a temporary file on the same filesystem as the real one.
2. Flush, then fsync. Flush pushes the program’s buffered bytes down to the operating system. fsync forces the operating system to push them onto the physical device. Both are needed: a flush alone leaves the bytes in the OS cache, where a power cut loses them.
3. Rename the temp file over the real one with os.replace. This is atomic, meaning any reader sees either the entire old file or the entire new one, never a mixture. This is the step that buys the whole guarantee.
4. fsync the directory, so the rename itself is durable and not just the bytes.

Only after step 4 is the checkpoint committed.

The bottom row is open(final, 'w').write(...). A crash partway leaves a file that exists, is truncated, and may still parse as valid JSON. That is the dangerous outcome, and it is the subject of the next subsection.

Why a torn checkpoint is worse than no checkpoint

A torn write is one that stopped partway, leaving a file that is neither the old state nor the new one.

There are two ways a non-atomic write dies, and only one of them is survivable. Both are shown below: case A is the file refusing to parse, case B is the file parsing into a lie.

--- case A: it fails loudly (the lucky case) ---
$ python -c "import json; json.load(open('state/checkpoint.json'))"
json.decoder.JSONDecodeError: Unterminated string starting at:
  line 2841 column 18 (char 184301)

resume: refuses to start. You lose 8 hours. You know you lost 8 hours.

--- case B: it parses (the case that costs money) ---
the write was cut between records and the tail happened to close cleanly:

{"step": 34,
 "goal_sha": "9f2c...",
 "completed": [ ...22 entries... ],     <-- 12 entries lost
 "queue": [],                            <-- flushed before the queue was written
 "spent_usd": 11.40}

In case A the crash landed inside a string, at byte 184,301. The file no longer parses, so every attempt to resume fails immediately and loudly. You lost eight hours and you know it.

In case B the crash happened to land between records and the remaining braces closed cleanly. The file parses. Twelve of the 34 completed tasks are missing and the queue is empty, and nothing about the file says so. Resume proceeds:

resume: loads fine. queue is empty. goal predicate fails.
        -> QUEUE_EMPTY_GOAL_UNMET, or worse, with the naive loop, "completed".
        12 completed tasks are now invisible; the agent may redo them, and
        anything non-idempotent runs twice.

Idempotent means running an operation twice has the same effect as running it once; anything non-idempotent — a payment, a message, an append — does damage on the second run.

No checkpoint fails fast; a torn checkpoint that parses produces a confident wrong resume. That asymmetry is the argument for atomicity.

The implementation

Checkpoint.save below is the four-step write from the diagram. Checkpoint.load is its inverse plus two refusals. Three things to watch for as you read:

• The payload is framed: the state is serialized to a string, then wrapped in an outer object carrying a SHA-256 of that string. load rehashes and compares.
• Every refusal raises RuntimeError with UNRECOVERABLE_ERROR in the message, so a bad checkpoint arrives at the loop as one of the seven endings rather than as a stray JSONDecodeError.
• artifacts stores paths, not contents, and history_summary stores a compacted summary rather than the raw transcript. Rule 4 below explains why.

import hashlib, json, os, pathlib, tempfile

SCHEMA_VERSION = 3

class Checkpoint:
    def __init__(self, path: str):
        self.path = pathlib.Path(path)
        self.path.parent.mkdir(parents=True, exist_ok=True)

    def save(self, state) -> None:
        payload = {
            "schema_version": SCHEMA_VERSION,
            "goal": state.goal.text,               # original, never rewritten
            "goal_sha": state.goal._sha,
            "queue": [t.model_dump() for t in state.queue],
            "in_progress": state.in_progress.model_dump() if state.in_progress else None,
            "completed": [t.model_dump() for t in state.completed],
            "parked": [t.model_dump() for t in state.parked],
            "spent_usd": state.budget.spent,
            "step": state.step,
            "artifacts": state.artifact_paths,     # paths, not blobs
            "history_summary": state.summary,      # COMPACTED: the run's history
                                                   # summarized down by a model
                                                   # call, not the raw transcript
        }
        body = json.dumps(payload, indent=2, sort_keys=True)
        framed = json.dumps({"sha256": hashlib.sha256(body.encode()).hexdigest(),
                             "body": body})

        # temp file on the SAME filesystem, or the rename is not atomic
        fd, tmp = tempfile.mkstemp(dir=self.path.parent, suffix=".tmp")
        try:
            with os.fdopen(fd, "w") as f:
                f.write(framed)
                f.flush()
                os.fsync(f.fileno())               # bytes on the device
            os.replace(tmp, self.path)             # atomic rename
            dirfd = os.open(self.path.parent, os.O_DIRECTORY)
            try:
                os.fsync(dirfd)                    # the rename itself is durable
            finally:
                os.close(dirfd)
        except BaseException:
            pathlib.Path(tmp).unlink(missing_ok=True)
            raise

    def load(self) -> Optional[dict]:
        if not self.path.exists():
            return None
        try:                                            # a TORN file dies here,
            framed = json.loads(self.path.read_text())  # not at the checksum:
            body = framed["body"]                       # there is no body to hash
            body_sha = hashlib.sha256(body.encode()).hexdigest()
        except (ValueError, KeyError, TypeError, AttributeError) as e:
            raise RuntimeError(
                f"UNRECOVERABLE_ERROR: checkpoint unreadable: {e!r}") from e
        if body_sha != framed["sha256"]:
            raise RuntimeError("UNRECOVERABLE_ERROR: checkpoint checksum mismatch")
        payload = json.loads(body)
        if payload["schema_version"] != SCHEMA_VERSION:
            raise RuntimeError("UNRECOVERABLE_ERROR: checkpoint schema mismatch")
        return payload

The block below does a round trip, then chops the saved file at 200 random byte positions and asserts that every one is refused. It finishes with the mechanism’s limit: a body that was already wrong before it was serialized frames, hashes, and reloads without error — case B from above, reproduced exactly. It prints checkpoint: 200 truncations refused, 4 checks passed.

import random

class _CTask:
    def __init__(self, tid): self.id = tid
    def model_dump(self): return {"id": self.id}

class _CkState:
    def __init__(self):
        self.goal = Goal("Reduce p95 latency on /api/search to under 200ms.")
        self.queue, self.completed, self.parked = [_CTask("t_34")], [_CTask("t_33")], []
        self.in_progress = None
        self.budget = type("B", (), {"spent": 11.40})()
        self.step, self.artifact_paths, self.summary = 34, ["out/0042.sql"], "..."

ck = Checkpoint(tempfile.mkdtemp() + "/state/checkpoint.json")
ck.save(_CkState())
assert ck.load()["step"] == 34 and ck.load()["spent_usd"] == 11.40   # round trip

# CHAOS: 200 random truncations, and not one of them loads. This is the one
# guard in this chapter that survives adversarial testing intact — say so.
whole = ck.path.read_text()
random.seed(0)
refused = 0
for _ in range(200):
    ck.path.write_text(whole[:random.randrange(1, len(whole))])
    try:
        ck.load()
    except RuntimeError as e:
        refused += "UNRECOVERABLE_ERROR" in str(e)
assert refused == 200

# ...and note WHICH branch refuses. Every one of the 200 dies in json.loads,
# never at the checksum comparison, because a truncated frame has no body to
# hash. Same safety, different branch — and before the try above, it arrived as
# a bare JSONDecodeError, which is not one of the seven endings.
ck.path.write_text(whole[:len(whole) // 2])
try:
    ck.load()
except RuntimeError as e:
    assert "unreadable" in str(e) and "checksum mismatch" not in str(e)

# ADVERSARIAL: the checksum covers what happened AFTER serialization and
# nothing before it. A body that was already wrong when it was written frames,
# hashes and reloads without a murmur — case B, verbatim.
wrong = json.dumps({"schema_version": SCHEMA_VERSION, "queue": [], "completed": [],
                    "step": 34, "spent_usd": 11.40}, sort_keys=True)
ck.path.write_text(json.dumps({"sha256": hashlib.sha256(wrong.encode()).hexdigest(),
                               "body": wrong}))
assert ck.load()["queue"] == []                      # loads fine. The hash is silent.
print("checkpoint: 200 truncations refused, 4 checks passed")

Five rules are doing the work in that code, and each one earns its place by ruling out a specific way the resume goes wrong.

1. Temp file on the same filesystem. os.replace across filesystems is a copy, not a rename, and a copy is not atomic. /tmp is usually a different mount — a separate storage volume — which is exactly why the temp file is created next to the checkpoint instead.
2. fsync the file, then fsync the directory. Without the second one, the rename can be lost in a power failure even though the data was written.
3. Checksum the body — and be exact about which half does what. This one needs three sentences, because the checksum gets more credit than it earns.
   • os.replace is what defeats case B, not the checksum. An atomic rename means a reader sees the whole old file or the whole new one, so a torn frame never reaches disk under the real name at all. And when a truncated file is handed to load() directly, as the chaos block does, it dies in json.loads before the checksum is ever compared.
   • What the checksum adds is everything the rename cannot cover: bit rot, a bad disk, a frame corrupted after it was committed.
   • What neither covers is a body that was already wrong when it was serialized. That one frames, hashes and reloads without complaint — the last assertion in the block above.
   • The refusal is worth having regardless, and it must arrive as UNRECOVERABLE_ERROR rather than as a raw JSONDecodeError, or the run ends in an exception that is not one of the seven endings.
4. Artifacts by path, never by content. An artifact is any file the run produced. A checkpoint that embeds file contents grows with the work and turns resume into a large prefill — a large block of text sent into the model’s context on the first call, paid for by the token.
5. Checkpoint after every task, and again before starting one — not on a timer. The checkpoint is a few kilobytes; a timer just chooses how much work you are willing to lose.

IN_PROGRESS: resume verifies, it does not re-execute

The dangerous window is a task that was mid-execution when the process died. Its side effects are half-applied and the checkpoint cannot know how far it got.

That is what the IN_PROGRESS marker is for, and the rule for handling it is counter-intuitive: on resume you check whether the task already happened, and you never simply run it again.

The trace below shows why. The first half is a real interruption — an index migration killed mid-flight. The second half compares three ways of resuming from it. Read the two naive branches first; they are what you get by default.

03:11:02  t_27  status=IN_PROGRESS  "apply index migration 0042"
03:11:02  checkpoint saved (in_progress=t_27)
03:11:19  psql: CREATE INDEX CONCURRENTLY idx_search_ts ON search_events(created_at);
03:11:41  <SIGKILL — spot instance reclaimed>

--- resume at 07:02:10 ---
load()  -> in_progress = t_27

  naive (re-execute):
    psql: CREATE INDEX CONCURRENTLY idx_search_ts ...
    ERROR:  relation "idx_search_ts" already exists
    -> agent sees an error, "fixes" it by dropping and recreating,
       burns 40 minutes rebuilding an index that was already valid

  naive on a non-idempotent task class:
    t_27 = "post the summary to the #eng channel"
    -> posted twice. No error. Nobody notices it was the agent's fault.

  correct (verify, do not execute):
    t_27.verify()  ->  SELECT indisvalid FROM pg_index WHERE ... -> True
    mark t_27 DONE, continue at t_28

The task is a database migration — a schema change applied by running SQL — and SIGKILL is the signal that terminates a process instantly, with no chance to clean up. The index was in fact built before the kill, so re-executing produces an error that the agent then “fixes” destructively. The second case is worse, because posting a message twice raises no error at all.

The rule and its consequence:

Every task carries a cheap, idempotent verify(). On resume, an IN_PROGRESS task is verified, never re-executed. A task that cannot be verified is not eligible for autonomous execution.

In code, resume is the first thing the harness calls after loading a checkpoint. It reads in_progress out of the checkpoint dict, runs that task’s verify(), and routes on the answer. There is no execute call anywhere in it — that absence is the entire point of the function.

def resume(state, ckpt: dict):
    t = ckpt.get("in_progress")
    if t is None:
        return state
    task = Task(**t)
    outcome = task.verify()                  # cheap, idempotent, read-only
    if outcome.done:
        state.completed.append(task)
    elif outcome.partially_applied:
        state.parked.append(task.with_reason(
            f"resumed into a partially applied state: {outcome.detail}"))
    else:
        state.queue.insert(0, task)          # nothing happened; safe to re-run
    state.in_progress = None
    return state

Here is that function called on the checkpoint from the trace above, once for each of the three verdicts verify() can return. The state on the left is what the harness holds afterwards:

ckpt = {"in_progress": {"id": "t_27", "kind": "migration", "sql": "0042"}, ...}

--- verify() says the index exists and is valid ---
resume(state, ckpt)
  outcome.done = True
  -> state.completed  = [..., t_27]
     state.queue      = [t_28, t_29, ...]        # unchanged
     state.in_progress = None
  the run continues at t_28. The index is NOT rebuilt.

--- verify() says the index does not exist at all ---
resume(state, ckpt)
  outcome.done = False, outcome.partially_applied = False
  -> state.queue      = [t_27, t_28, t_29, ...]  # pushed back to the FRONT
     state.in_progress = None
  the run re-executes t_27, which is safe: nothing happened the first time.

--- verify() says the index exists but is marked INVALID ---
resume(state, ckpt)
  outcome.partially_applied = True
  -> state.parked = [..., t_27 ("resumed into a partially applied state:
                                 idx_search_ts exists with indisvalid=false")]
     state.in_progress = None
  the run continues at t_28 and the morning report names t_27 for a human.

The three-way outcome is where most implementations are only two-way, and the missing third is the expensive one.

• done means the work landed. Mark it complete and move on.
• Nothing happened means the task never took effect, so it is safe to put back at the front of the queue.
• partially_applied is the case implementations forget, and the one that must go to a human. A half-applied migration is not something an agent should improvise against unsupervised.

Budget ceilings

One spending limit is not enough. The two used here fail in opposite directions, so each covers the other.

The budget ledger is the harness’s running total of money spent. It is consulted before every model call, and what it does depends on how much of the cap is gone. Four bands, shown below.

flowchart LR
    B[Budget ledger] --> W1["< 70%<br/>run freely"]
    B --> W2["70-90%<br/>tell the agent to prioritize"]
    B --> W3["90-100%<br/>finish current task only"]
    B --> W4["> 100%<br/>halt + report partial"]

    style W2 fill:#bc6c25,color:#fff
    style W3 fill:#bc6c25,color:#fff
    style W4 fill:#9d0208,color:#fff

With a $28 cap, those percentages are concrete dollar figures:

• Below $19.60 (70%) — run freely.
• $19.60 to $25.20 (70–90%) — the harness injects a notice so the agent knows to prioritize what matters most.
• $25.20 to $28.00 (90–100%) — finish the current task only, start nothing new.
• Past $28.00 — halt and report partial progress.

Two ceilings, and you need both

Each one fails in a way the other covers.

The difference is enforcement versus pacing.

The hard cap is a control. It is plain harness code, invisible to the model, and it refuses the API call — one request to the model provider — before the request is made. The model has no say in it.

The task budget is a request. It is a number placed in the model’s own configuration so the model can pace itself and wrap up gracefully. A model stuck in a tool loop sails straight past it, because nothing enforces it.

Neither alone is enough, so run both.

The ledger, in code

Ledger has two methods and they run at different times. reserve runs before a call and can refuse it. record runs after a call and books what it actually cost. PRICES[model] returns a (input_price, output_price) pair in dollars per million tokens.

class Ledger:
    def __init__(self, hard_cap_usd: float):
        self.hard_cap = hard_cap_usd
        self.spent = 0.0
        self.blocked = False               # read by should_stop; see below

    def reserve(self, model: str, est_in: int, max_out: int) -> None:
        """Pre-flight. Check the WORST case before the call, not the actual after."""
        pin, pout = PRICES[model]                       # $/MTok
        worst = (est_in * pin + max_out * pout) / 1e6
        if self.spent + worst > self.hard_cap:
            self.blocked = True            # the ONLY thing that makes
            raise BudgetExceeded(          # BUDGET_EXHAUSTED reachable
                f"would reach ${self.spent + worst:.2f} of ${self.hard_cap:.2f}")

    def record(self, usage, model: str) -> None:
        pin, pout = PRICES[model]
        self.spent += (usage.input_tokens * pin
                       + usage.cache_read_input_tokens * pin * 0.1
                       + usage.cache_creation_input_tokens * pin * 1.25
                       + usage.output_tokens * pout) / 1e6

Prices are quoted in $/MTok, dollars per million tokens. A token is the few-character chunk models read and write text in (Tokens). Dividing by 1e6 at the end of each expression converts a token count times a per-million price back into dollars.

reserve checks the worst case before the call, not the actual after it. The worst case is the whole prompt plus the maximum output the call is allowed to produce, because max_tokens is the only bound you have on output before the call happens.

Work one out. On Opus 5 at $25 per million output tokens, a call with max_tokens=64000:

64,000 output tokens x $25 / 1,000,000  =  $1.60

That is $1.60 of exposure from a single call, on output alone, before you count a token of input. Checking after the fact makes the ceiling an observation rather than a control — you find out you blew it once you already have.

Why spent >= hard_cap is not enough on its own

That pre-flight has a consequence people miss, and it is the defect this section exists to fix.

reserve refuses the call that would cross the cap. So the run stops with money still on the ledger, and spent never reaches hard_cap. In the termination test earlier, the run ends at $27.75 of a $28.00 cap: the next call would have taken it to $28.75, so it was refused.

Read literally, then, spent >= hard_cap inside should_stop is a condition that never fires. BUDGET_EXHAUSTED becomes an ending that nothing produces — a row in the table with an exit code and no code path.

The fix is one flag, wired across three places:

1. reserve sets blocked = True before it raises.
2. run catches BudgetExceeded, sets nothing else, and continues.
3. should_stop reads blocked at the top of the next lap and reports BUDGET_EXHAUSTED — PARTIAL, exit 2, with the report the table promises instead of a traceback.

The alternative is to make the stop test spent + typical_call_cost >= hard_cap, which works but puts an estimate in the control path. The flag puts the decision in exactly one place.

The ledger, exercised

The block below prices two turns and then shows the pre-flight refusing a call without spending. The second case matters most: a turn billed by input_tokens alone comes out 13.75x cheaper than it really is, because the tokens went to the cached input fields. The block prints ledger: 4 checks passed.

PRICES = {"claude-opus-5": (5.0, 25.0), "claude-sonnet-5": (3.0, 15.0)}

class _Usage:
    def __init__(self, i, o, read=0, write=0):
        self.input_tokens, self.output_tokens = i, o
        self.cache_read_input_tokens, self.cache_creation_input_tokens = read, write

# the number the paragraph above quotes, derived rather than asserted
assert 64_000 * PRICES["claude-opus-5"][1] / 1e6 == 1.60

led = Ledger(28.00)
led.reserve("claude-opus-5", 20_000, 4_000)          # $0.20 worst case: allowed
led.record(_Usage(20_000, 4_000), "claude-opus-5")
assert round(led.spent, 4) == 0.2000 and not led.blocked

# ADVERSARIAL: a cached turn. A ledger counting `input_tokens` alone bills this
# at $0.0025. It is $0.0344 — 13.75x more — and the ceiling is then nowhere
# near where you think it is.
before = led.spent
led.record(_Usage(500, 0, read=20_000, write=3_500), "claude-opus-5")
assert round(led.spent - before, 6) == round((500 * 5 + 20_000 * 0.5
                                              + 3_500 * 6.25) / 1e6, 6)
assert round(led.spent - before, 4) == 0.0344 != round(500 * 5 / 1e6, 4)

# the pre-flight refuses BEFORE the call, and records that it refused
tight = Ledger(1.00)
try:
    tight.reserve("claude-opus-5", 10_000, 64_000)
    raise SystemExit("the ceiling is an observation, not a control")
except BudgetExceeded as e:
    assert str(e) == "would reach $1.65 of $1.00"
assert tight.blocked and tight.spent == 0.0      # nothing spent: that is the point
print("ledger: 4 checks passed")

That 13.75x is worth deriving, because it is the difference between a ceiling that holds and one that does not. Prompt caching lets a repeated prefix be stored once and re-sent cheaply: cache reads bill at ~0.1x the input price and cache writes at 1.25x (Prompt caching derived). So a turn with 500 fresh input tokens, 20,000 cache reads and 3,500 cache writes, on Opus at $5/MTok input:

fresh input   500 x $5.00                     =    $0.0025
cache reads   20,000 x $5.00 x 0.1  = x $0.50 =    $0.0100
cache writes  3,500 x $5.00 x 1.25  = x $6.25 =    $0.0219
                                                 ---------
true cost                                          $0.0344
counting input_tokens only                         $0.0025

                        0.0344 / 0.0025  =  13.75x under-reported

A ledger that counts only input_tokens under-reports a cached run by that factor. Count all four usage fields or your ceiling is not where you think it is.

The visible half: task_budget

The second ceiling is set per call, as a budget the model can see. One detail is easy to get wrong.

task_budget takes a token count, not a dollar amount. Converting remaining dollars into it requires a price, and the price depends on which model the next call uses — so the harness does that conversion and the model never sees a dollar figure at all.

The ledger is denominated in dollars. The visible budget is denominated in tokens. They are two different instruments, and confusing their units is a sign the design has not been built.

with client.beta.messages.stream(
    model="claude-opus-5",
    max_tokens=64000,
    betas=["task-budgets-2026-03-13"],
    output_config={"effort": "high",
                   "task_budget": {"type": "tokens", "total": remaining_tokens}},
    tools=TOOLS,
    messages=messages,
) as stream:
    resp = stream.get_final_message()

At 70% spend, inject a budget notice into the conversation so the agent reprioritizes rather than discovering the ceiling by hitting it.

Put that notice at the end of the message list, not in the system prompt. The system prompt is the standing instruction block sent at the front of every call. The notice is volatile — its dollar figure changes on every call — so putting it at the front would invalidate the cached prefix each time.

Concretely: it belongs after the last cache breakpoint, the marker up to which the prefix is reused. The end of the window is also where recall is best (Why quality degrades in long contexts), so the placement is right for two independent reasons.

Memory

“Memory” in an eight-hour agent is not one thing: it is five different stores with five different lifetimes, and each exists because some mechanism above needs data that lives exactly that long.

The table names the five, and the third column says which need each one serves.

Layer by layer, ordered by how long each one lives:

• Working memory is the message list of the current model call. It dies with the call.
• The checkpoint survives a crash. That is its whole job.
• The original goal survives everything, including the agent’s own opinions about it.
• Episodic memory is the run’s accumulated lessons. Episodic means drawn from specific past episodes rather than from general instructions.
• The filesystem holds the actual outputs, and this is what keeps context flat. The context window is the text the model is given on a single call: finite, and paid for by the token. Because the work lives in files and only the paths are carried forward, the context window stops growing even as the pile of work grows.

The original goal must be stored immutably and compared against literally. If the agent can rewrite it, drift detection compares the queue against a goal that has already drifted, and reports 0.0 while the run goes off the rails. That is the trace above, and it is the central failure this case study guards against.

The episodic layer needs the same discipline as Reflexion lessons (Reflexion), where an agent writes down what it learned from a failure and reads it back on later attempts. A lesson must be specific and falsifiable, or it is a permanent tax on every future prompt:

✗  "Be careful with database migrations."
✓  "CREATE INDEX CONCURRENTLY cannot run inside a transaction block;
    the migration runner wraps everything in one. Use raw_sql_outside_tx()."

The first costs tokens on every call and changes no decision. The second names the constraint (this database command refuses to run inside a transaction block, meaning a group of statements applied all-or-nothing), the cause (the migration runner wraps everything in one), and the workaround, and it can be proved wrong, which a vague warning cannot.

How many API calls does this actually make?

The ceiling should be a number you can defend rather than a guess, which means deriving the cost of an overnight run from token counts.

The run to be priced: 8 hours, 40 tasks, roughly 6 model calls per task. Prices: Opus 5 at $5 per million input tokens and $25 per million output tokens, written $5/$25. Sonnet 5, the cheaper mid-tier model, at $3/$15.

Step 1 — per-task input, derived

Each task is a short ReAct loop with its own context, and that context grows on every turn, because the whole conversation is re-sent each time. The model is stateless; call 4 does not remember calls 1 through 3, so you resend them.

Two constants set the growth:

• The first call carries the system prompt (3.0k tokens) plus the task brief (0.5k) = 3.5k.
• Every later call adds the previous tool result (3.0k) plus the model’s own previous reply (0.3k) = 3.3k more than the call before it.

So the size of call t is:

input(t) = 3.5k + (t - 1) x 3.3k

Substituting, for a six-call task:

input(1) = 3.5 + 0 x 3.3 =  3.5k
input(2) = 3.5 + 1 x 3.3 =  6.8k
input(3) = 3.5 + 2 x 3.3 = 10.1k
input(4) = 3.5 + 3 x 3.3 = 13.4k
input(5) = 3.5 + 4 x 3.3 = 16.7k
input(6) = 3.5 + 5 x 3.3 = 20.0k
                           -----
sum per task               70.5k input,  ~4.2k output
x 40 tasks                 2.82M input,  168k output

The 70.5k is the sum of all six calls, not the size of the last one. You pay for the whole context on every call, so a 20.0k final context does not cost 20.0k — it cost 70.5k to get there.

That sum is why the total grows with the square of the loop length. Adding a seventh call would add its own 23.3k, not 3.3k. This is the lever the optimization table pulls hardest on.

The last two lines scale to the whole run: 70.5k x 40 tasks = 2,820k = 2.82M input tokens, and 4.2k x 40 = 168k output tokens.

Step 2 — bill it

Now price every kind of call the run makes, each at its own model’s rate.

Read the arithmetic column as: (input in thousands x $/MTok) + (output in thousands x $/MTok), all divided by 1000 to land in dollars. The division by 1000 is what converts “thousands of tokens at a per-million price” into dollars. Spelled out for the first row:

2,820k input  x $5/MTok  =  2820 x 5  = 14,100
  168k output x $25/MTok =   168 x 25 =  4,200
                                        ------
                                        18,300 / 1000  =  $18.30

Where the call counts come from: 40 tasks x 6 calls = 240 task-execution calls; one verification per task = 40; a drift check every 5 tasks over 40 tasks = 8; plus 12 replans and 6 compactions. Total 240 + 40 + 8 + 12 + 6 = 306 calls.

Compaction is the periodic call that summarizes the history so far so the context stops growing. Six of them across the night cost $1.80.

The total: $18.30 + $0.78 + $0.35 + $0.78 + $1.80 = $22.01.

83% of the run is the 240 task-execution calls — $18.30 of $22.01. Everything else is rounding error, including the drift checks people worry about. Eight Opus calls cost $0.35, which buys the one detector that catches the failure mode defining this architecture.

What each optimization is worth, in isolation

Each row below changes one thing and leaves everything else at the baseline. The savings are comparable to each other but do not add up — apply two and you get less than the sum, because they compete for the same tokens. Everything is measured against the $22.01 baseline.

The New total column is computed the same way every time: take $22.01, subtract the affected row’s old cost, add its new cost.

Four of those rows compress arithmetic that is worth unpacking.

The Sonnet row. Sonnet 5 is $3/$15 against Opus 5’s $5/$25 — and 3/5 = 15/25 = 0.6, the same ratio on both input and output. Move 60% of 40 tasks (24 of them) to Sonnet and the task row becomes 16 tasks at full price plus 24 at 0.6x:

$18.30 x [ (16/40) + (24/40) x 0.6 ]  =  $18.30 x (0.40 + 0.36)
                                      =  $18.30 x 0.76  =  $13.91
new total  =  $22.01 - $18.30 + $13.91  =  $17.62

The effort: low row. effort is a setting that controls how much the model reasons before answering; low cuts output tokens. Trimming those 24 mechanical tasks by roughly 40% takes total output from 168k to 128k, and output is priced 5x input, so 40k tokens is real money:

40k saved output x $25/MTok  =  40 x 25 / 1000  =  $1.00
new total  =  $22.01 - $1.00  =  $21.01

The loop-cap row, which is the biggest single win. It wins because of the quadratic term: dropping the 5th and 6th calls removes their own input and every re-send those calls would have carried. Summing input(t) for four calls instead of six:

input, 4 calls  =  3.5 + 6.8 + 10.1 + 13.4      =  33.8k   (vs 70.5k)
output          =  4.2k x 4/6                    =   2.8k

input  33.8k x 40 tasks = 1,352k x $5/MTok   =  $6.76
output  2.8k x 40 tasks =   112k x $25/MTok  =  $2.80
                                                -------
task row                                         $9.56
new total  =  $22.01 - $18.30 + $9.56  =  $13.27,  saving $8.74 (39.7%)

The caching row is worth seeing in full. A cache write costs 1.25x the normal input price and a cache read costs 0.1x, so each call re-reads the whole previous prefix cheaply and pays full freight only on the 3.3k of new material. Effective tokens below means “tokens you are billed for at the plain input price” — a 3.5k write bills as 3.5 x 1.25 = 4.375k effective.

call 1:  write 3.5k x 1.25                        =  4.375k effective
call 2:  read 3.5k x 0.1  + write 3.3k x 1.25     =  4.475k
call 3:  read 6.8k x 0.1  + write 3.3k x 1.25     =  4.805k
call 4:  read 10.1k x 0.1 + write 3.3k x 1.25     =  5.135k
call 5:  read 13.4k x 0.1 + write 3.3k x 1.25     =  5.465k
call 6:  read 16.7k x 0.1 + write 3.3k x 1.25     =  5.795k
                                          total   = 30.05k  (vs 70.5k)

Feed 30.05k back through the task row and the saving is real money:

input  30.05k x 40 tasks = 1,202k x $5/MTok   =  $6.01
output   4.2k x 40 tasks =   168k x $25/MTok  =  $4.20   (unchanged)
                                                 -------
task row                                          $10.21
new total  =  $22.01 - $18.30 + $10.21  =  $13.92,  saving $8.09 (37%)

The precondition that schedule quietly depends on: a prefix is cached only above the model’s minimum cacheable length, and that floor is per model rather than per generation.

The first call writes a 3.5k prefix. On claude-opus-5 the floor is 512 tokens, and 3,500 / 512 = 6.8x, so it clears comfortably and call 1 can write at all. On claude-haiku-4-5 the floor is 4,096 — above 3.5k — and this entire schedule collapses to zero. Every write is refused, every read is a miss, and all six calls bill at full input price (Prompt caching the highest leverage lever). Check usage.cache_creation_input_tokens on the first call rather than assuming the marker did something.

The caveat that decides whether this works: the cache TTL — time to live, how long an entry survives before it is dropped — is five minutes.

A task whose tool calls are slow (a test suite, a migration, a build) will exceed five minutes between calls. Then every turn pays a 1.25x write with no read to amortize it, which is worse than not caching at all — you have raised your input bill by 25% and gained nothing.

Measure the inter-call gap per task class, cache only the fast ones, or buy the extended TTL. This is the most common way a caching “optimization” silently raises the bill.

The sentence to say out loud

All of the above compresses into one answer, and every number in it traces back to a step above.

“About $22 for an overnight run, so I’d set the hard ceiling at $28 — $5.99 of headroom over the derived $22.01, so the ledger refuses the next call instead of cutting the agent mid-edit. The visible task budget is a token count, not a dollar figure, and I’d seed it per task from the derivation above: 70.5k in and 4.2k out for a six-call task. The number that actually matters is cost per completed and verified task: at a 75% verification pass rate that’s $22.01 / 30 = $0.73. If the pass rate drops below ~70% the agent is burning budget on work that gets thrown away, and I’d rather halt and alert than keep paying for it.”

Two of those figures are worth being able to reproduce on demand:

headroom          $28.00 - $22.01              =  $5.99
verified tasks    40 tasks x 75% pass rate     =  30 tasks
cost per verified $22.01 / 30                  =  $0.73

Cost per completed and verified task is the number to lead with, because it is the only one that moves when quality moves. A cheaper run that verifies less is not cheaper.

Failure modes

Everything above compresses into one table: each way the run goes wrong, the signal that reveals it, and the mechanism that contains it. Bolded rows are the ones worth memorizing. If you can reconstruct the Guard column from the Failure column, you can rebuild the design from scratch.

Two rows have a dash in the detection column, and that is deliberate: a crash and an irreversible mistake cannot be detected after the fact, so they are handled entirely by prevention — checkpoint everything, keep production credentials out of the process, and give the agent no tool that can do damage it cannot undo.

The last two rows are the ones to volunteer unprompted. The harness evaluates the goal predicate itself. The agent’s claim of success is an input to the report, never the basis for it.

Alternatives considered and rejected

The argument against your own design is the part interviewers weigh most heavily. Each row below names why the alternative is tempting before saying why it loses.

Three terms in that table deserve plain definitions.

• A cron job is a script run on a fixed schedule by the operating system, with no model involved.
• Fan-out means splitting work across several agents running at once. Reading in parallel is safe because nothing is changed. Writing in parallel means two agents editing the same state with nobody awake to reconcile them.
• A tool allowlist is the explicit list of tools the agent is permitted to call. The constraint is enforced by what exists, not by what the prompt asks for, which is why “no publish verb in the allowlist” is a stronger guarantee than “do not publish anything” in a system prompt.

Evals

Before trusting a night to this design, you need evidence that it works. Eval is short for evaluation: an automated test whose subject is the agent’s behavior rather than a single function’s return value.

The first column names four layers of test, defined underneath the table. The bolded rows are the ones most easily written in a form that cannot fail, and the mistake they share has a specific shape.

An eval that restates the implementation cannot fail. That is the shape, and it generalises well past this design.

“Mutating the goal text raises UNRECOVERABLE_ERROR” is the assert_intact body written out in English. It passes on the one input the author had in mind and says nothing about the input an attacker has in mind. That is why every eval row above that names a mechanism also names the case that defeats it and asserts the defeat — the row records what the guard does not do as well as what it does.

The four layers in that first column do different jobs:

• Unit tests a single function in isolation.
• Component tests one mechanism, such as the drift detector, against inputs whose right answers you already know.
• Integration runs the whole agent end to end in a sandbox — a throwaway environment with no access to anything real.
• Chaos is the odd one out: it deliberately breaks the machine mid-run to see whether recovery works.

Run integration evals at 10x speed with a shrunken budget: 4 tasks, a $2 cap, a 20-minute wall clock. The shape of every failure above appears at that scale, and you can iterate twenty times a day. Full overnight runs are for release candidates only.

The chaos tests catch what design review cannot. kill -9 sends the same uninterruptible termination signal a cloud provider does when it reclaims your machine, and firing it at a random step is a two-line test harness. It is the only thing that proves your atomic write is actually atomic on the filesystem you deploy to.

Interviewer pushback

These are the questions this design attracts, what each one is testing, and the answer that lands. The italic line names the question behind the question.

“Isn’t this just a ReAct loop with extra steps?” Testing: whether you can name what autonomy actually costs to build. The loop is identical. Everything that makes it survivable is outside the loop: verification per task, atomic checkpointing with IN_PROGRESS semantics, drift detection against an immutable goal, dual budgets, and a prioritized termination set where every exit is reported. Without those, an 8-hour ReAct loop is an expensive mess nobody can audit. The loop is the easy part; the termination set is the hard part.

“Walk me through your termination conditions.” Testing: whether you treat this as a set with properties, or a pile of ifs. There are seven members, and the set is both exhaustive and prioritized. Exhaustive because the loop is while True with one stop check and a try around the body — that second half is the one people leave out, and without it an exception from a tool is a way out with no class, no exit code and no report. BudgetExceeded is classified as BUDGET_EXHAUSTED, everything else as UNRECOVERABLE_ERROR, and a module-level STOP == set(STOP_CLASS) asserts at import that no member of either side is missing from the other. Prioritized because several can be true at once and the order decides which one you report — goal-met first, so a run that finishes on its last dollar reports success. And the member people forget is QUEUE_EMPTY_GOAL_UNMET, which is a failure class.

“Why is an empty queue a failure?” Testing: the single sharpest trap in this case study. Because it means the agent ran out of ideas before it ran out of goal. The deeper point is that the bug is in the loop shape, not the reporting: while queue: makes “queue empty, goal unmet” the normal exit, so there is no branch to report it from. You cannot report a state your control flow treats as success. The trace runs like this: 33 of 34 tasks pass, the one that measured the actual goal fails, queue empties, report says “completed”, p95 is unchanged, and nobody reads the footnotes on a green run.

“Your drift detector is an LLM call. What if it’s wrong?” Testing: whether you calibrate detectors or just add them. There are two separate concerns here. Reliability: calibrate against 20 hand-labeled queues and report agreement before trusting a threshold; it has a known false-negative bias because a plausible chain reads as reasonable. Attack surface: it runs in a fresh context with only the goal and the queue, never the agent’s reasoning, for the same reason a code reviewer isn’t the author — a judge holding the agent’s justifications will rationalize the chain it just read.

“What if the agent decides the goal was imprecise and refines it?” Testing: whether you have seen this one. Then drift detection compares the queue against a goal that already drifted and reports 0.0 the whole way down. The trace is the one above: goal rewritten at step 17, score never exceeds 0.08, p95 unchanged at step 40, no alert fires. The fix is structural, and I’d be precise about which half does the work: the goal lives in the harness rather than on disk, so no tool can reach it — that is the lock. The hash asserted on every drift check is a smoke alarm behind it, and only catches an in-process write that forgets to update the digest; anything that can set _text can set _sha too. If the agent believes the goal is wrong, the correct output is a halt with a proposed revision, not a silent edit.

“What happens if the process dies mid-task?” Testing: whether “checkpointing” means something concrete to you. The task was marked IN_PROGRESS and checkpointed before execution, so resume knows exactly which task was in flight. It then runs that task’s verify() — never re-executes it. Three outcomes: done (mark complete), not started (requeue), partially applied (park for a human). Re-executing is how you get a duplicate Slack post or a second refund, and neither raises an error.

“Why not just write the checkpoint directly?” Testing: whether you know why atomicity is not pedantry. Because a torn checkpoint that still parses is worse than no checkpoint. No checkpoint fails fast and you know you lost the run. A truncated file whose tail happens to close cleanly loads fine with twelve completed tasks missing and an empty queue — a confident wrong resume. So: temp file on the same filesystem, fsync the file, os.replace, fsync the directory, and a SHA-256 over the body so anything the rename didn’t cover becomes UNRECOVERABLE_ERROR instead of silence.

“Why two budgets?” Testing: whether you understand the difference between a request and a control. They fail in opposite directions. The hard cap is enforcement — a pre-flight check on the worst-case cost of the next call, raising before it is made, invisible to the model. Alone, it cuts the agent mid-edit. The task budget is pacing — visible to the model so it wraps up gracefully. Alone, it is advisory and a runaway tool loop sails past it. They are not even denominated in the same unit — the hard cap is dollars on the ledger, the task budget is tokens in output_config, because tokens are what the API accepts. The headroom lives on the dollar side: a $28 ceiling over a derived $22.01 spend, which is what buys the clean wrap-up. And the two have to be wired together or one is dead code: reserve() refuses before the call, so spent never reaches the cap, so the stop check reads a blocked flag rather than the balance.

“You said $22. Where does it go, and what would you cut first?” Testing: whether the number is derived or quoted. 83% is the 240 task-execution calls, and per-task input is 70.5k because a 6-call loop resends its history each turn. Two levers dominate: capping the loop at 4 calls instead of 6 saves $8.74 of $22.01, which is 39.7%, because the term is quadratic in calls; incremental prefix caching saves $8.09, or 37%. Model tiering on the mechanical 60% is another 20%. I’d cap the loop first — it’s free and it also reduces drift surface. The caching one has a trap: the TTL is five minutes, so on tasks with slow tools you pay 1.25x writes with no reads and the “optimization” raises the bill.

“How do you know it did the right thing overnight?” Testing: whether you’ll accept the agent’s word for anything. You don’t take its word. Success is a machine-checkable predicate the harness runs. The report contains the trace, the diff, the spend curve, every parked task with its reason, and the drift score history. If the predicate can’t be written, the task isn’t a candidate for autonomy — that test is the gate, not a nice-to-have.

“What would you actually let it do unsupervised?” Testing: calibrated risk judgment. Reversible work in an isolated environment with a test suite as ground truth: dependency upgrades gated on continuous integration, mechanical refactors, test backfilling, data cleaning on a copy. Not: anything touching production, sending external communications, spending money, or any action whose verify() cannot distinguish “not done” from “half done.” The tool allowlist has no publish verb in it, so the constraint is enforced by what exists rather than by what the prompt asks for.

“How do you test an 8-hour agent?” Testing: whether you have actually shipped one. At 10x speed with a shrunken budget — 4 tasks, a $2 cap, a 20-minute clock. Every failure shape reproduces at that scale and you get twenty iterations a day. Then two chaos tests that design review cannot substitute for: kill -9 at a random step with a resume assertion, and a checkpoint truncated at a random byte with a refuse-to-start assertion. Full overnight runs are for release candidates.

“Why not just use a cron job?” Testing: whether you will argue against your own design. If the steps are known in advance, use a cron job — cheaper, deterministic, auditable, and it cannot drift. And be careful how you phrase the comparison: a cron job’s model cost is zero, so there is no multiple to quote. The honest framing is absolute, not relative — this design costs about $22 a night against a script that costs nothing, so autonomy has to be worth $22 a night on its own terms. It is, but only when the next step genuinely depends on what the previous one found. And if I could get a human to look at a review queue each morning, I’d build the supervised hybrid instead: the agent parks anything it’s unsure about and continues on the rest. That costs nothing and removes most of the irreversible-action risk.

Next: 06 — Customer Support Agent.