11 — Case Studies

Eight fully worked answers to the interview question “design an agent that does X.” Each file is one complete design round from end to end: the architecture, the arithmetic that prices it, the ways it breaks, and the interviewer’s objections with each one answered. Each file covers enough to defend its design in a forty-five-minute interview, and the eight share a template you can apply to a ninth problem that nobody wrote up.

These are the applied end of the agentic systems track. Each file restates the idea it depends on before using it and links to the chapter that derives it, so you can read them without any other chapter first and follow the links back when you want the mechanism underneath a number.

The vocabulary every file below uses

A handful of terms recur in all eight studies and are defined once here.

An agent is a program that puts a large language model (LLM) in a loop: the model proposes an action, your code runs it, the result is appended to the conversation, and the loop repeats until the model says the task is finished. A tool is one of the actions you permit — a function with a name, a description and typed arguments, which the model invokes by emitting a structured request rather than by running anything itself. The harness is all the code around the model: the loop, the tool implementations, the budget checks. A turn is one pass around the loop, and one turn costs one call to the model’s application programming interface (API), which is why “how many API calls does this make?” is a cost question.

Everything the model can see at once is its context window, measured in tokens — the sub-word chunks text is split into, roughly three-quarters of a word each. Prices are quoted per MTok, meaning one million tokens, and output tokens cost several times what input tokens cost. Prompt caching appears in every cost table: you mark a stable prefix of the conversation, the provider keeps its computed state, and re-sending that prefix on the next turn bills at about a tenth of the input price in exchange for a one-time write premium of 1.25× (Prompt caching derived). Finally, evals are the automated test suite for an agent — fixed inputs, a scoring function, and a threshold you can gate a release on.

What goes in and what comes out

Two input/output pairs matter here and are easy to confuse: the exercise has one, and the agent you design inside it has another.

The interviewer’s sentence is the only thing you are given. Everything in the right-hand column is what you are expected to produce, roughly in that order, inside forty-five minutes. The method for running that clock is chapter 10; these eight files are that method already executed on eight problems.

The eight designs

Each row names what the running system consumes and emits, and the one idea the file exists to teach. Terms of art in the last column are defined where they first appear.

Three terms in that table carry the designs and are worth defining. Grounding means checking every claim in a drafted reply against a document actually retrieved, so the model cannot state a policy that does not exist. Drift detection means scoring the agent’s current task queue against the goal it was given, to catch a run that is still busy but no longer working on the right thing. A subagent is a second model instance with its own context window, given a written brief and returning a summary — the mechanism, and the one good reason for it, are derived in chapter 06.

In what order to read them

Nothing here depends on anything else here, so what follows is a recommendation rather than a prerequisite chain. Read 01 through 08 straight through if you are working the whole track. If you have limited time before a specific interview, the order below front-loads the designs that get asked most and the judgments that transfer furthest.

Each file’s interviewer pushback section is the last section of all eight, and it is the only part written as dialogue.

The shape they share

All eight files follow one template, so you can navigate any of them without re-reading it and can compare two designs section by section.

Eight sections appear in every file, always in this order — only the evals and alternatives pair ever swaps:

flowchart LR
  A[Problem] --> B[Architecture]
  B --> C[Design-specific core]
  C --> D[Memory]
  D --> E[How many API calls]
  E --> F[Failure modes]
  F --> G[Evals and alternatives]
  G --> H[Interviewer pushback]

The design-specific core is where each design does its own thing, and it is the bulk of the file. What varies is worth knowing before you use these to compare designs:

The API-calls section is not the ending. It lands between two-fifths and four-fifths of the way into each file — 45% through 03, 74% through 06, around 75% typically — because everything after it exists as a consequence of it. Failure modes, evals and alternatives are all things an interviewer starts probing once that table is on the board, not things they move on from.

What the money section contains

Every file includes one thing a typical design writeup omits, and it is what you are being asked to reproduce.

The section is a table of turns, input tokens, output tokens, cache hits and dollars, with the arithmetic shown rather than asserted, followed by a ranked list of the levers that would cut it. The levers land in a consistent order — prompt caching first, then how many turns the loop is allowed, then how much each observation costs, and only then which model you picked (Optimization order). Here is what each study’s table comes out to, which is also a fast index to how the eight problems differ in scale:

In half these studies the model bill is not the largest number on the page, and saying so is what separates a senior answer from a competent one.

Failure traces, shown rather than described

Every study names what it rejected and why, and most of them print the failing artifact instead of describing it.

Case study 07 is the cleanest example. A one-to-many join between orders and their line items makes each order’s total repeat once per line, so summing that column counts it several times — a fan-out double count. The join is correct, the query runs without error, and revenue reads $2,300 instead of $1,000. Both queries are printed side by side and they differ in a single aggregate expression. The reported number is not 2× or 3× but 2.3×, so no magnitude check catches it and a finance team can run on it for two quarters.

Case study 08 prints an extraction that is schema-valid and arithmetically self-consistent and still wrong by $15. The model computed a line total from quantity times unit price instead of reading the discounted total printed on the page, so the validator checked the model’s arithmetic against itself and learned nothing. That is a harder failure than a malformed record, and it is the reason every numeric field in that schema is nullable — a field that must contain a number is a guarantee the model will invent one. (This study’s own failure modes section is a table rather than a trace.)

The rest carry traces too. Study 01 has a prompt-injection trace, in which text on the screen is read by the model as an instruction, alongside a second failure worth memorizing; 02 and 03 each carry one; 04 has two, an averaging failure and an injection through a fetched page; and 05 traces a goal the agent rewrote at step 17, after which drift detection kept scoring the run as aligned — never above 0.08 — because it was comparing the queue against a goal that had already moved.

Pricing used throughout: claude-opus-5 $5/$25 per MTok input/output, claude-sonnet-5 $3/$15, claude-haiku-4-5 $1/$5. Cache reads bill at about 10% of the input price and cache writes at 1.25×. The ordered queue that interleaves these eight with the rest of the material is Study_plan.