09 — Text-to-Image

A text-to-image system takes a sentence and returns images.

The generator that turns random noise into an image is a settled engineering choice. What decides product quality is how the sentence reaches that generator: which model reads the text, how its output enters the image generator, what the guidance knob at sampling time does to the probability distribution being sampled, how to measure any of it, and what one image costs.

This chapter covers how to:

• draw the system end to end;
• derive classifier-free guidance from Bayes’ rule;
• price an image to four decimal places;
• explain mechanically why “a red cube on a blue sphere” often comes out purple.

The input and output. A sentence of English goes in — say "a red cube on top of a blue sphere, in a sunlit room". Four PNG images at 1024 × 1024 pixels come out, about six seconds later, all four different from one another. That is the contract. Everything below exists to make the images match the sentence.

The machinery this chapter stands on

The generator is a latent diffusion model, which has three parts: a compressor, a denoiser, and a loop that runs the denoiser.

Part one: the VAE, which shrinks the image. A variational autoencoder, or VAE, is a pair of small neural networks trained together. Its encoder squeezes a 1024 × 1024 image down to a much smaller grid of numbers, here 128 × 128 positions with 4 numbers at each. That compressed grid is called a latent. Its decoder expands a latent back into pixels.

Working in the latent instead of in pixels is what makes the whole thing affordable. Count the positions:

pixels   1024 · 1024  =  1,048,576 positions
latent    128 ·  128  =     16,384 positions
                         1,048,576 / 16,384  =  64x fewer

Then patching, which shrinks it again. One more step happens before the denoiser sees that grid, and every cost figure below depends on it. A patch size of 2 groups each 2 × 2 block of latent cells into a single token:

latent cells        128 · 128  =  16,384
patch 2 -> tokens    64 ·  64  =   4,096      a further 16,384 / 4,096 = 4x

That token count, 4,096, not the cell count, is what every FLOP and dollar figure in this chapter is computed from. Serving restates it where the arithmetic is done.

(A note on c = 4, the four numbers per position: The ceiling the autoencoder sets concludes that c = 16 is worth taking, at no change in token count and therefore no change to any figure here. This chapter keeps c = 4 because it is what public checkpoints ship; nothing below depends on the choice.)

Part two: the denoiser, which does the generating. This is the large network. Training works by taking a real image’s latent, corrupting it with a known amount of random Gaussian noise, and asking the denoiser to predict the noise that was added. How much noise was added is indexed by a number t called the timestep: t near 1 means almost pure noise, t near 0 means almost a clean image.

Part three: the sampler, the loop that runs the denoiser over and over. Generation starts from pure noise and subtracts a little of the predicted noise at a time — about 28 rounds — until a clean latent falls out. Then the VAE decoder turns that latent into an image. Each of those rounds is one forward pass through the denoiser, and the count of forward passes is the entire compute bill.

The model family comparison derived argues why diffusion beat the alternatives for open-domain image generation, and Latent diffusion derived derives the latent-versus-pixel choice and its cost; Sampling ddpm ddim and the step count curve covers samplers and step counts. Read them for depth. Nothing here requires them.

Those three parts leave exactly one thing undecided: how a sentence gets into the denoiser at all.

That path is called the conditioning path. “Conditioning” is the machine-learning word for extra information a model is given alongside its main input — here, the prompt alongside the noisy latent.

The questions that separate a strong answer from a memorized one all live on that path:

• which model reads the text;
• why the text reaches the denoiser through cross-attention rather than by being glued onto its input;
• what classifier-free guidance is doing to the sampled distribution;
• why attributes swap between objects.

1. Problem framing

Fix the contract first: what the system must do, what makes it hard, and three reframes that carry every later decision.

• Input: a free-text prompt, 1-200 words, from an open-domain user population. Open-domain means you do not get to restrict what people ask for — no fixed vocabulary, no category list.
• Output: 4 images at 1024 · 1024, in under ~6 seconds.
• Constraints: cost per image must sit far below the revenue per image; nothing illegal or defamatory may leave the box; the same prompt should be reproducible on request but not identical across the four samples. Why it is hard: there is no label. There is no “correct” image for “a lonely lighthouse in a storm.”

So the objective is a distribution match. You are asking the model to produce images that look like plausible draws from the space of images that fit the sentence, not to reproduce one target. And every metric you can compute is a proxy for a judgment you cannot compute.

First thing to say: “The generative backbone is a solved-ish choice — latent diffusion. The decisions that move the product are on the conditioning path: which text encoder, how it reaches the denoiser, and what guidance scale you sample at. Those three decide prompt adherence, and prompt adherence is what users mean when they say a model is good.”

Prompt adherence is the term this chapter uses constantly, so pin it down now: it is how faithfully the image contains what the sentence asked for — the right objects, the right count, the right colours on the right objects, the right spatial arrangement. It is a separate thing from aesthetics, which is how good the image looks regardless of what was asked. The two are measured differently and they trade off, which is the single most repeated idea in this chapter.

Three reframes carry the design:

Three terms in that table are used before they are defined:

• FID stands for Fréchet Inception Distance, the standard one-number image-quality score. Offline two axes never one number derives it and lists four things wrong with it.
• Recaptioning means throwing away the text that came with your training images and writing new descriptions of them with another model. Caption quality is the dominant lever shows why that is the highest-leverage money in the budget.
• Capacity is a loose word for how many parameters a network has, and therefore how much it can represent.

Assumptions in this stage. Every section of this chapter ends with a block like this one, sorted into three bins: things you state out loud and move on, things you ask because the answer changes the design, and the load-bearing claim that the section collapses without.

State out loud:

• Four images at 1024 × 1024 in about six seconds, prompts of 1-200 words, an open-domain consumer audience.
• Each is a product decision you are free to pick. Being wrong costs you a re-derivation of the cost arithmetic in Serving, not a different architecture.

Ask, never assume:

• What the product is for. A stock-photo tool, a design tool with layout controls, and a consumer toy have different tolerances for the adherence-versus-aesthetics trade in The tradeoff derived rather than asserted. The answer sets the guidance scale you ship, which is the single most visible knob in the system.
• Whether four samples per request is fixed. It is the multiplier on every cost figure in Serving, and it is also the product’s main defence against sampling variance.

Load-bearing: that no single number can express quality here, because there is no correct image.

Everything downstream depends on that — two-axis human evaluation, gating on question-by-question (VQA) prompt adherence rather than FID, splitting the regenerate metric by whether the prompt was edited. If a single trustworthy quality score existed, you would optimize it directly and could delete most of Metrics.

2. ML objective

What is the model trained to do? Two properties of the training objective cause most of the problems later in the chapter.

Latent diffusion trains one network to predict the noise that was added to a latent, given the text. Writing the training target as the noise itself, rather than as the clean image, is called the epsilon parameterization — “epsilon” (eps) is the conventional symbol for that noise:

z_0    = E(x)                       VAE encoder, image -> latent
z_t    = sqrt(a_t)·z_0 + sqrt(1 - a_t)·eps        eps ~ N(0, I)
loss   = E_{z_0, c, t, eps}  || eps - eps_theta(z_t, t, c) ||^2

Line by line:

• Line 1. x is a training image and E is the VAE encoder, so z_0 is that image’s clean latent.
• Line 2 corrupts it. eps ~ N(0, I) means “draw eps from a standard Gaussian” — each number is an independent random draw with mean 0 and variance 1. a_t is a fixed schedule between 1 and 0 that says how much clean signal survives at timestep t.
• Line 3 is the loss. eps_theta is the denoiser, theta standing for its trainable weights, and it is handed the corrupted latent z_t, the timestep t, and the conditioning c. || · ||^2 is squared distance summed over every number in the latent. E_{...} means “average over” — over images, captions, timesteps and noise draws.

The whole thing reads: on average, over every image and every noise level, the denoiser’s guess at the noise should match the noise that was actually added.

c is the text conditioning — the vectors the prompt was turned into. The entire text-to-image problem is the question of what c is and how eps_theta reads it — everything else in the loss is the unconditional generative model, the plain picture-maker that ignores text.

Two properties of this loss matter downstream and are worth stating before anyone asks:

It is a regression, not a likelihood you can read. A regression loss just measures the distance between a prediction and a target; a likelihood would tell you how probable the data is under the model. This is the former, so the loss value tells you almost nothing about sample quality. A model with lower validation loss can produce visibly worse images, because the loss is dominated by high-noise timesteps where every model predicts roughly the mean. Never gate a release on training loss.

It has no term for prompt adherence. The model is asked to denoise, and the caption is a hint that makes denoising easier. If the caption is wrong, ignoring it is the correct thing for the loss to learn. That single sentence is why caption quality is the dominant data lever in Data and labels, and why classifier-free guidance in Classifier free guidance derived exists at all — sampling has to force the conditioning to matter more than training did.

Assumptions in this stage.

State out loud: the epsilon parameterization and a standard variance-preserving noise schedule. Flow-matching and v-prediction are the common alternatives. They change the arithmetic inside the sampler and the guidance scales in The tradeoff derived rather than asserted by a point or two, and they change nothing about the conditioning argument that is this chapter’s subject.

Ask: whether the base model is being trained from scratch or adapted from an existing checkpoint. That decides whether Data and labels and Training are a project or a paragraph.

Load-bearing: that the loss contains no adherence term. The recaptioning spend in What recaptioning costs, the existence of classifier-free guidance in Classifier free guidance derived, and the attribute-binding failure in Failure modes are all consequences of it. If the objective did score whether the image matched the caption, the honest design is to train against that score directly and delete the guidance machinery entirely.

3. Conditioning: the centerpiece

The model that reads the prompt matters more than the model that draws the image. This section gives the mechanism by which text reaches the denoiser, and rules out three alternatives to it.

3.1 Why the text encoder outranks the denoiser

The text encoder is the binding constraint on prompt adherence, and the reason is the encoder’s training objective rather than its size.

A text encoder is a neural network that turns a sentence into a sequence of vectors — one vector per token, a token being a chunk of text a bit smaller than a word. Frozen means its weights are downloaded, never updated during your training, and used as a fixed function.

The denoiser never sees your prompt. It sees a sequence of vectors that a frozen text encoder produced. Structure the encoder discarded is structure the denoiser cannot recover, at any parameter count, because there is no path in the architecture by which it could.

Compare the candidates on the axis that matters. There are four, and they need naming before the table:

• CLIP (Contrastive Language-Image Pretraining) is a pair of models trained to make an image and its caption land near each other in one shared vector space. Its text half is called the “text tower.”
• OpenCLIP bigG is a larger open reimplementation of the same idea.
• T5 (Text-To-Text Transfer Transformer) is a general-purpose language model; its encoder half is what gets used here, and “XXL” is its largest size.
• Decoder LM hidden states means the internal activations of an ordinary text-generating model, such as a 7-billion-parameter chat model.

The table below compares them on parameter count, how many tokens they accept, what they were trained to do, and — the column that decides the argument — what their per-token output vectors actually record:

Seven pieces of vocabulary in that table, each of which the argument turns on:

• An embedding is a list of numbers a network produces to stand for something — a token, a sentence, an image — arranged so that things the network treats as similar get similar lists. The whole of this subsection is an argument about what a particular embedding does and does not record.
• ViT-L is “Vision Transformer, Large,” the size label of the CLIP variant.
• Context is the maximum number of tokens the encoder will accept. CLIP’s 77 is a hard truncation, so a long prompt simply loses its tail.
• A contrastive objective trains a model to score matching pairs higher than mismatched ones.
• Pooled means the whole sentence is collapsed into one summary vector before the loss looks at it — typically by averaging, or by reading one designated position.
• Span corruption is T5’s training task: blank out random runs of words and make the model reconstruct them.
• Attachment is the grammatical question of which word modifies which. In “a red cube on a blue sphere,” attachment is what says red belongs to cube.

The size argument

Read the Params column against the denoiser’s 2.6B:

CLIP ViT-L text tower     0.123B
T5-XXL encoder            4.7B      a 38x jump
the entire denoiser       2.6B      so the swap adds 4.7 / 2.6 = 1.8x
                                    the denoiser's whole parameter count

Swapping a 123M CLIP tower for a 4.7B T5 encoder adds nearly twice the conditioning capacity that the entire denoiser has. That is the resource-allocation argument in one line, and it is why the modern default is a large frozen LM encoder with a comparatively modest denoiser rather than the reverse.

The objective argument, which is the deeper one

The size is not actually the main reason. The objective is.

CLIP’s contrastive loss is computed on a single pooled vector per caption, matched against a single pooled vector per image. The gradient — the signal that says how each weight should change to lower the loss — that reaches the token at position 3 exists only insofar as position 3 changes the pooled vector.

Nothing in the loss requires the pooled vector to distinguish these two captions:

"a red cube on a blue sphere"
"a blue cube on a red sphere"

Both contain the same words. Both describe scenes with a red thing, a blue thing, a cube, and a sphere.

A bag of words is a representation that records which words appeared and forgets the order. A pooled representation that is close to a bag of words matches both images about equally well. So the contrastive objective is nearly indifferent between encoding the binding — which adjective goes with which noun — and not encoding it. There is no gradient pressure either way.

CLIP is not bad at composition by accident; its training objective never asked for composition. (Composition here means getting multiple objects and their properties correct together rather than one at a time.)

T5’s span-corruption objective is the opposite case. It cannot be solved without representing which adjective modifies which noun — you cannot fill in a deleted word without knowing what the surrounding phrase is about. So T5’s token features carry attachment as a side effect of having been trained at all, not because anyone asked for it.

3.2 Cross-attention is the mechanism

The operation by which text vectors reach the denoiser has a specific shape, and three consequences can be read straight off it.

Attention is the operation by which one set of vectors looks up information in another. Each vector on the asking side emits a query (Q); each vector on the answering side emits a key (K) and a value (V). Every query is compared against every key, the comparison scores are turned into weights that sum to 1 by a softmax — a function that exponentiates a list of scores and divides by their total, so the result reads as a set of proportions — and each query gets back the weighted average of the values. When a set of vectors attends to itself it is called self-attention; when it attends to a different set it is called cross-attention. That distinction is the whole of what follows.

The conditioning enters through cross-attention layers interleaved with the denoiser’s self-attention blocks: the image asks, the text answers. Attention and why context costs what it does works the same machinery through in a language-model setting.

It has one asymmetry: Q and K,V come from different places, and there are far fewer of the latter.

Q  comes from the image latent tokens      (64 · 64 = 4,096 of them at 1024px)
K, V come from the text encoder outputs    (up to 512 of them)

attention(Q, K, V) = softmax( Q · K^T / sqrt(d) ) · V

Q · K^T is every query scored against every key. d is the width of those vectors, and dividing by sqrt(d) keeps the scores from growing with width. Multiplying by V collects the answer.

The latent tokens are the 64 × 64 grid positions of the latent, flattened into a sequence. Each one is a small patch of the image asking the prompt what should be drawn there.

The diagram traces one prompt through that path to the denoiser:

flowchart LR
    P["Prompt<br/>a red cube on<br/>a blue sphere"] --> ENC["Frozen text encoder<br/>FIXED CAPACITY<br/>structure lost here<br/>is lost forever"]
    ENC --> KV["K, V<br/>one vector<br/>per text token"]
    NZ["Noisy latent z_t<br/>4,096 tokens"] --> QQ["Q<br/>one vector<br/>per spatial location"]
    QQ --> XA["Cross-attention<br/>softmax over TEXT<br/>independently per location"]
    KV --> XA
    XA --> OUT["eps prediction"]

    style ENC fill:#1d3557,color:#fff
    style XA fill:#bc6c25,color:#fff
    style OUT fill:#2d6a4f,color:#fff

The prompt reaches the denoiser through exactly one component, and it is frozen. Everything the rest of the system can do with the prompt is bounded by what came out of that component.

Three consequences fall straight out of that shape:

Every spatial location queries the whole prompt independently. Latent token i computes its own softmax over the text tokens. There is no coupling between what location i attends to and what location j attends to. The architecture has no representation of “these two regions are the same object.”

The attention map is a soft segmentation. The attention map for a text token is that token’s column of softmax weights, reshaped back onto the image grid — for each text token you can read off, per layer per step, a 64 · 64 heatmap of where in the picture it is being applied. A segmentation is an assignment of image regions to things; this is a soft one because the weights are fractions rather than yes/no. This is your primary debugging instrument and Failure modes uses it.

Cost is linear in prompt length, not quadratic. The Q · K^T product has shape 4,096 · 512 — image tokens by text tokens. Only one of those two numbers is the prompt, so the cost grows in direct proportion to prompt length rather than with its square:

128-token prompt   4,096 · 128  =   524,288 scores
256-token prompt   4,096 · 256  = 1,048,576 scores    2x the prompt, 2x the work

Doubling the prompt costs almost nothing in absolute terms. Doubling the resolution costs a lot, because that number goes up on both sides of every self-attention product. That is the opposite of the intuition people import from large language models, where every token attends to every other token and cost grows with the square of the length. It is why “just write a longer prompt” is free while “just render at 2048” is not.

Note that this claim is about cross-attention only. The denoiser’s own self-attention over its 4,096 image tokens is still quadratic in the image, which is where the 5.5 TFLOP attention term in Serving comes from.

3.3 Alternatives to cross-attention, and why they lose

Text could reach the denoiser three other ways, and each has a cost. MMDiT is not a bad idea; it is a real competitor, rejected on cost rather than capability.

Reading that table needs four definitions:

• The timestep embedding is the small vector the denoiser already receives that tells it how noisy z_t is. “768-d” means 768 numbers long, and broadcast means the same vector is added at every one of the 4,096 spatial positions — which is exactly why it cannot say different things in different places.
• MMDiT is a Multimodal Diffusion Transformer. It stops treating text as a read-only reference and instead glues the text tokens onto the image sequence, so both get updated by the same self-attention. The catch is cost: the sequence is now 4,096 + 512 = 4,608 long, and self-attention costs the square of that, so you pay (4608/4096)^2 = 1.27x on every attention layer.
• FiLM is Feature-wise Linear Modulation: given one summary vector, scale and shift each of the network’s channels by a learned amount. It conditions the whole image uniformly, which is right for “how noisy is this” and “how pretty should this be” and useless for “red on the left.”
• A class label is the one-of-N conditioning older models used — “dog,” “cathedral” — where a single vector genuinely is the whole message.

Assumptions in this stage.

State out loud: a frozen T5-XXL-class encoder at 4.7B parameters and 512 tokens of context, feeding a 2.6B denoiser through cross-attention. Those exact sizes set every FLOP figure in Serving and nothing structural.

Ask:

• Whether the product needs spatial control — bounding boxes, sketches, region masks. If it does, the conditioning path grows a second channel and several failure modes in Failure modes stop being failures. That is an architecture change, not a tuning one.
• Whether prompts in the target market are long and syntactic or short and keyword-ish. CLIP’s 77-token truncation is only a defect if prompts exceed it.

Load-bearing: that the encoder is frozen, so structure it discarded is unrecoverable downstream.

That one claim carries three later arguments: spending parameters on the encoder rather than the denoiser, the explanation for attribute binding in Failure modes, and the highest-impact rejection in Alternatives considered and rejected. If you trained the encoder jointly with the denoiser, the bottleneck argument weakens and the resource-allocation conclusion has to be re-derived — though Alternatives considered and rejected explains why nobody does that anyway.

4. Classifier-free guidance, derived

One knob controls how hard the image follows the prompt. What raising it buys, and the three ways it goes wrong, follow from a short derivation.

Classifier-free guidance (CFG) is a sampling-time trick that makes the image follow the prompt harder than the trained model would on its own. It has one dial, the guidance scale w, and turning it up buys prompt adherence and sells realism, diversity and colour fidelity. Classifier free guidance introduces it in a setting with no text, where the tradeoff is fidelity against coverage. Text conditioning changes what the trick is doing and adds a second, purely numerical failure unrelated to probability, so it is worth deriving here in full.

4.1 The setup

The whole method rests on a single fact: a classifier you never trained is already sitting inside the model, as the difference between two of its predictions.

Start with one fact. Diffusion’s noise prediction is a scaled score. The score of a probability distribution is the gradient of its log-density with respect to the data — in plain terms, the direction in which the data would have to move to become more probable.

Three symbols before the line: grad_z means “the gradient with respect to z,” p(z_t | c) is the probability density of noisy latents given the caption, and sigma_t is the noise standard deviation at timestep t.

eps_theta(z_t, t, c)  ≈  -sigma_t · grad_z log p(z_t | c)

So the denoiser, trained only to predict noise, also reports which direction increases probability. Everything after this is algebra.

Step 1: rearrange Bayes’ rule. Bayes’ rule is the identity that relates the probability of the caption given the image to the probability of the image given the caption. Take logs of it, differentiate in z, then move terms so the classifier p(c | z) sits alone on the left. The term log p(c) has no z in it, so its gradient is zero and it drops out:

log p(z | c)  =  log p(c | z) + log p(z) - log p(c)

grad_z log p(c | z)  =  grad_z log p(z | c)  -  grad_z log p(z)

Step 2: convert scores back into things you can compute. Multiply the second line through by -sigma_t, and use the fact from the top of this subsection to replace each score with a denoiser call. The first term on the right becomes the denoiser run with the caption, written eps_cond; the second becomes the same denoiser run with no caption, written eps_uncond:

-sigma_t · grad_z log p(c | z)   =   eps_cond  -  eps_uncond

The difference between the conditional and unconditional noise predictions is exactly the score of an implicit classifier p(c | z) — a classifier you never trained and never have to. p(c | z) is “how well does this image match this caption,” which is precisely a classifier’s job; the name classifier-free guidance comes from getting it without ever building one. Saying it in this form is what distinguishes an answer that derived the method from one that memorized it.

4.2 The extrapolation

That observation turns into the formula everyone quotes — and into the reason a second forward pass exists at all.

Now ask to sample not from p(z|c) but from a sharpened distribution that overweights the classifier. “Proportional to” is all that is needed, since the constant that makes it a valid probability distribution never affects the gradient.

Four lines, and each one follows from the one above it. The left margin says what operation got you there:

define          p_w(z | c)  proportional to  p(z | c) · p(c | z)^(w - 1)

log, then       grad log p_w  =  grad log p(z|c) + (w - 1)·grad log p(c|z)
differentiate

multiply by     eps_cfg  =  eps_cond + (w - 1)·(eps_cond - eps_uncond)
-sigma_t, then
substitute
step 1's result

collect terms   eps_cfg  =  eps_uncond + w·(eps_cond - eps_uncond)

The last two lines are the same expression written two ways. Expand the third line and the eps_cond terms cancel:

eps_cond + (w-1)·(eps_cond - eps_uncond)
  =  eps_cond + w·eps_cond - w·eps_uncond - eps_cond + eps_uncond
  =            w·eps_cond - w·eps_uncond + eps_uncond
  =  eps_uncond + w·(eps_cond - eps_uncond)

That is the CFG formula, and now every term in it has a meaning:

• w = 1 recovers plain conditional sampling — the correction term is multiplied by zero.
• w = 0 gives eps_uncond, which is unconditional sampling.
• w > 1 raises the implicit classifier to a power, which sharpens it: probabilities near 1 stay near 1 while everything else is pushed toward 0.

Getting eps_uncond is a training-time trick, not a second model. During training, replace the caption with a learned null embedding — a single vector, trained like any other weight, that stands for “no caption” — with probability ~10%. One network learns both behaviors; at sample time you run it twice, once with c and once with null. That habit of blanking the conditioning during training is called conditioning dropout, and it reappears in Training.

The 10% is set by two pressures pushing in opposite directions. The unconditional branch only needs to be good enough to define a direction, so it does not need much data. But every dropped caption is a training step that teaches nothing about conditioning, so it is not free either.

• Below ~5%, the unconditional branch is undertrained. eps_uncond is then noisy, the difference vector is noise rather than a score, and guidance becomes unstable at high w.
• Above ~20%, you are paying real adherence for a term you only subtract.

4.3 The tradeoff, derived rather than asserted

Three separate things go wrong as w rises, and they go wrong for three unrelated reasons — which is why no single fix addresses all three.

1. Diversity falls, because the target distribution is mode-seeking

A mode is a peak of a probability distribution. Mode-seeking means the sampler is drawn toward peaks and away from everything else.

p(c|z)^(w-1) concentrates mass wherever the implicit classifier is most confident. Raising a number below 1 to a large power shrinks it far faster than it shrinks a number near 1, so the gap between “very confident” and “somewhat confident” widens with every increment of w.

As w grows, the sampled distribution collapses toward the argmax — the single most confident point — of p(c|z), within the region the model considers a plausible image at all. At w = 12, four samples from “a golden retriever” are four near-copies of the most prototypical golden retriever the model knows.

2. Realism falls, because the tilted distribution is not the data distribution

p(z|c) · p(c|z)^(w-1) is a product of two densities. Multiplying two densities together is called tilting one by the other, and the peaks of the product need not sit anywhere near the peaks of either factor.

The data manifold is the thin, curved sheet inside the space of all possible latents on which real images actually live. The overwhelming majority of latents decode to garbage. Nothing in the tilted expression constrains the sharpened optimum to lie on that sheet, so the sampler happily walks off it.

3. Saturation appears, because the sampler’s step size assumes a calibrated eps

Saturation is the visible symptom: colours pushed to their maximum, blown-out highlights, crunchy edges. Calibrated here means the predicted noise has the statistical size the sampler was designed around.

The update rule was derived under the assumption that eps_theta has roughly unit variance per component — that is what the schedule was built for. CFG adds the difference vector w times, so the magnitude grows with w:

|| eps_cfg ||  ≈  || eps_uncond ||  +  w · || eps_cond - eps_uncond ||

|| · || is the norm, the overall magnitude of a vector. So that line says the predicted noise gets bigger roughly in proportion to w.

The chain from there to the visible artifact has three links:

1. An over-large eps makes each sampling step over-shoot toward the predicted clean latent z_0.
2. Repeated over-shooting drives the latent’s numbers to magnitudes the VAE decoder never saw in training.
3. The decoder maps those to clipped, blown-out red-green-blue values.

The “deep-fried” look at w = 15 is a decoder operating out of distribution — being fed inputs unlike anything in its training set — not an aesthetic choice anyone made.

What the numbers look like

The table below is one measured sweep of w on a fixed 2,000-prompt evaluation set. Read it for the disagreement between columns, not for the absolute values.

The three offline columns are all defined properly in Offline two axes never one number. In brief: FID compares the statistics of a batch of generated images against a batch of real ones and is lower-is-better; CLIPScore measures caption-image similarity using CLIP and is higher-is-better; VQA adherence decomposes the prompt into yes/no questions, asks a vision-language model each one, and scores the fraction answered correctly. The last column is the fraction of human raters preferring that setting over w = 5 in a head-to-head.

Three metrics, three different optima: FID at 2.0, CLIPScore at 12.0, humans at ~4-5. An optimum is the setting at which a metric is best.

Notice what that means in practice. If you shipped the FID-optimal setting of w = 2, only 31% of raters would prefer it to w = 5. If you shipped the CLIPScore-optimal setting of w = 12, only 9% would. That table is the argument against reporting one number, and it is also the answer to “what guidance scale should I use”: the one that maximizes the metric you are actually being paid for.

The diagram below is the same argument in one picture. One cause on the left, three independent mechanisms in the middle, four consequences on the right.

flowchart TD
    W["Raise guidance w"] --> S1["p of c given z raised to w-1<br/>-> mode-seeking"]
    W --> S2["tilted product density<br/>-> modes off the data manifold"]
    W --> S3["norm of eps grows ~linearly in w<br/>-> sampler over-shoots"]
    S1 --> D1["DIVERSITY falls<br/>4 samples become 4 copies"]
    S2 --> D2["REALISM falls"]
    S3 --> D3["SATURATION<br/>VAE decodes out-of-range latents"]
    W --> A1["ADHERENCE rises<br/>up to a plateau"]

    style W fill:#1d3557,color:#fff
    style A1 fill:#2d6a4f,color:#fff
    style D1 fill:#bc6c25,color:#fff
    style D2 fill:#bc6c25,color:#fff
    style D3 fill:#9d0208,color:#fff

Raising w fires all four arrows at once, so there is no setting at which you get the green box without the orange ones. That is why the choice is an operating point rather than a fix.

Where to set w in practice

The table below gives working ranges. The pattern across the first three rows is one idea: the weaker the conditioning signal, the more you have to amplify it.

Two names in it need expanding first. Flow-matching is an alternative training formulation to the epsilon prediction of Ml objective; it learns a straight-line velocity field instead of a noise estimate, and tends to need less guidance. Guidance distillation is the technique from What cfg costs that trains a model to imitate a guided model in a single pass.

Two mitigations worth naming, because they are what a practitioner reaches for before lowering w:

• Guidance rescale. After computing eps_cfg, rescale it to match the standard deviation of eps_cond — the standard deviation being the usual measure of how spread out a set of numbers is, which is what “the size of eps” means in practice. This directly cancels the norm growth derived above without touching the direction, so you keep adherence and lose the saturation.
• Guidance interval. Apply CFG only in the middle band of timesteps rather than at all of them. At very high noise the conditional and unconditional predictions barely differ, so guidance is wasted. At very low noise it mostly amplifies texture into crunch.

The guidance interval changes the bill, so it is off by default in this chapter. Here is the arithmetic if you turn it on. With t walking from 1 down to 0 across 28 steps, restricting CFG to t in [0.10, 0.85] leaves 20 steps inside the band and 8 outside it:

20 guided steps   · 2 passes  =  40
 8 unguided steps · 1 pass    =   8
                                ---
                                 48   against 56 with CFG everywhere

saving   (56 - 48) / 56  =  14%

Everything priced in Serving assumes the default — CFG at every step, NFE = 56 — so switching the interval on is a saving you would re-derive, not an assumption the chapter has already made.

Both mitigations appear in the function below, which is one guided step written out. The two things to read for are the lo <= t <= hi early return (the interval) and the eps_c.std() / eps.std() ratio (the rescale). The rest is scaffolding that makes the block runnable.

def guided_eps(model, z_t, t, c_text, c_null, w=5.0, rescale=0.7,
               lo=0.0, hi=1.0):
    """One CFG evaluation, with the two standard corrections applied.

    The cond/uncond pair is one batched forward of size 2, not two calls --
    same FLOPs, roughly the same wall time on an under-utilised GPU.

    lo/hi are the guidance interval, and they default to the whole schedule
    because that is what section 8 prices: NFE = 28 x 2 = 56. Pass
    lo=0.10, hi=0.85 to turn the interval on, and re-derive the bill.
    """
    if not (lo <= t <= hi):                 # guidance interval: skip the ends
        return model(z_t, t, c_text)

    eps_c, eps_u = model.batched(z_t, t, [c_text, c_null])
    eps = eps_u + w * (eps_c - eps_u)       # the extrapolation derived above

    # The guard matters. section 2 argues that ignoring a bad caption is the
    # loss-minimising behaviour, so eps_c == eps_u is an input this model is
    # *trained* to produce -- and a prediction with no spread at all puts a
    # zero in this denominator. Rescaling by a ratio of standard deviations
    # is a no-op in that case anyway, so skip it rather than divide by it.
    if rescale and eps.std() > 0:           # cancel the norm growth in w
        scaled = eps * (eps_c.std() / eps.std())
        eps = rescale * scaled + (1.0 - rescale) * eps
    return eps


def nfe(steps: int = 28, lo: float = 0.0, hi: float = 1.0) -> int:
    """Network evaluations for one image: 2 per guided step, 1 per skipped one."""
    ts = [1.0 - i / (steps - 1) for i in range(steps)]   # t walks 1 -> 0
    return sum(2 if lo <= t <= hi else 1 for t in ts)


# --- run the function, so the block is not decoration -------------------
class _Stub:
    """A minimal stand-in: returns a fixed pair, and counts its own calls."""
    def __init__(self, eps_c, eps_u):
        self.eps_c, self.eps_u, self.calls = eps_c, eps_u, 0

    def __call__(self, z_t, t, c):
        self.calls += 1
        return self.eps_c

    def batched(self, z_t, t, conds):
        self.calls += len(conds)
        return self.eps_c, self.eps_u


class _Vec(tuple):
    """Just enough vector arithmetic to run the function above."""
    def _z(self, o, f):
        o = o if isinstance(o, _Vec) else _Vec([o] * len(self))
        return _Vec(f(a, b) for a, b in zip(self, o))
    def __add__(self, o):  return self._z(o, lambda a, b: a + b)
    __radd__ = __add__
    def __sub__(self, o):  return self._z(o, lambda a, b: a - b)
    def __mul__(self, o):  return self._z(o, lambda a, b: a * b)
    __rmul__ = __mul__
    def std(self):
        m = sum(self) / len(self)
        return (sum((x - m) ** 2 for x in self) / len(self)) ** 0.5


plain = _Stub(_Vec((1.0, -0.5, 0.25, 2.0)), _Vec((0.8, -0.2, 0.10, 1.4)))
guided_eps(plain, None, 0.5, "a red cube", None)
print(f"one guided step costs {plain.calls} network evaluations")

# The exact input section 2 predicts: the model ignores the caption, so the two
# predictions are identical. Before the guard this divided by zero when the
# prediction was also flat; it must now be a no-op that returns eps unchanged.
same = _Vec((0.4, -0.3, 0.9, 0.1))
out = guided_eps(_Stub(same, same), None, 0.5, "a caption the model ignores", None)
assert all(abs(a - b) < 1e-12 for a, b in zip(out, same)), out

flat = _Vec((0.0, 0.0, 0.0, 0.0))           # zero spread -> a zero denominator
out = guided_eps(_Stub(flat, flat), None, 0.5, "ignored, and flat", None)
assert all(abs(a - b) < 1e-12 for a, b in zip(out, flat)), out

print(f"NFE, CFG every step        {nfe()}")
print(f"NFE, interval [0.10, 0.85] {nfe(lo=0.10, hi=0.85)}")

assert plain.calls == 2                     # one batched forward of size 2
assert nfe() == 56                          # what section 8 prices
assert nfe(lo=0.10, hi=0.85) == 48          # what the interval would cost
assert round(56 * 26.8) == 1501             # TFLOP per image, section 8

The rescale term is the derivation in code. eps.std() grows with w, the sampler was built for eps_c.std(), so you divide the growth back out. Blending at 0.7 rather than replacing outright is empirical: full rescaling costs a little adherence along with the saturation.

4.4 What CFG costs

The second forward pass has a cost. Separate its compute cost from its latency cost before naming the technique that removes it.

CFG doubles the number of function evaluations. The number of function evaluations (NFE) is the count of forward passes through the denoiser needed to make one image, and it is the standard unit of cost for a diffusion model because everything else is rounding error. 28 sampling steps become 56 forward passes. This is the single largest line item in Serving, so it is the first thing an interviewer will poke.

You do not pay double latency, though. The conditional and unconditional passes are independent — neither needs the other’s output — so they batch into one forward pass of batch size 2, meaning the graphics card processes both at once rather than one after the other. On an under-utilized GPU that is nearly free in wall-clock time and exactly 2x in FLOPs, floating-point operations, the count of arithmetic operations performed.

The real fix for the FLOPs is guidance distillation. Distillation is training a small or fast “student” model to imitate the outputs of a slower “teacher.” Here the teacher is the two-pass guided model and the student is a one-pass model that takes w as an input. That recovers the 2x. Combined with step distillation — a student trained to take one big sampling step where the teacher took several — it is how 4-step models exist.

Assumptions in this stage.

State out loud: 28 sampling steps, CFG applied at every one of them (so NFE = 56), a rescale blend of 0.7, and 10% caption dropout during training. All four are operating points you would sweep, and moving them re-prices Serving without changing any mechanism. The guidance interval is the clearest example, and it is deliberately not assumed here: switching it on cuts NFE to 48 and moves every figure in Serving with it.

Ask: which of the three metrics the business is paid on. The table above has three different optima, and the answer is the guidance scale you ship.

Load-bearing: that eps_cond - eps_uncond is a usable estimate of the implicit classifier’s score. That is what makes guidance a principled sharpening rather than a hack, and it is what predicts all three failure directions at once. If the unconditional branch is undertrained — the reason the 10% floor exists — the difference vector is noise rather than a score, guidance amplifies that noise, and none of this section’s predictions hold.

5. Data and labels

The training data has two halves: which image-text pairs survive filtering, and what the text attached to them says. The second decides model quality.

5.1 The pairs

The ingest filters below run in order of how much they pay off, and one argument about ordering is easy to get backwards.

The raw material is image-text pairs scraped from the web: a picture together with whatever text was near it, usually its alt-text, the description an HTML page supplies for screen readers. On the order of 2-5 billion pairs after collection, filtered down hard.

The table lists the filters in the order they pay off. The “Keeps” column is what survives each one, and the rows compound — the ~55% and the ~60% multiply out to roughly a third of the original corpus before the later filters even run.

Five terms in that table:

• An aesthetic predictor is a small model trained on human ratings that scores how good-looking an image is.
• Perceptual dedup is deduplication by visual similarity rather than by exact file match, so that a photo republished at three sizes counts once.
• Memorization is the failure dedup prevents: a model reproducing a training image closely enough to be recognizable.
• NSFW is “not safe for work,” the industry shorthand for sexual and graphic content. CSAM is child sexual abuse material, which is illegal to possess and is removed by matching against hashes of known material rather than by any classifier.
• A stock overlay is the semi-transparent watermark stock-photo sites stamp across previews.

One ordering argument that is easy to get backwards

Aesthetic filtering is a fine-tuning tool, not a pretraining tool. Pretraining is the long, broad, expensive first training run. Fine-tuning is a short additional run on a narrower dataset that adjusts an already-trained model.

Filter hard at pretraining and you delete most of the world’s visual diversity — diagrams, product photos, ordinary rooms. The model then loses the ability to render anything unglamorous, because it never saw any.

The standard recipe is broad pretraining, then a short high-aesthetic fine-tune. The asymmetry is the reason: you cannot recover coverage — the range of things the model can render at all — that you never trained on, but you can always add polish later.

5.2 Caption quality is the dominant lever

The text half of the training data matters more than the image half, and the right mix of original and rewritten captions can be derived from what each source uniquely contains.

Raw alt-text is terrible. It is written for accessibility, for SEO — search engine optimization, the practice of writing text to rank well in search results rather than to describe anything — or for nothing at all. Four real shapes of it, with the only useful one marked:

alt="IMG_20190412_113255.jpg"
alt="Click here to buy"
alt="dog"
alt="Golden retriever puppy sitting on a red picnic blanket in a park,
     shallow depth of field, late afternoon light"      <- rare

Recall from Ml objective that the loss has no adherence term: if the caption is uninformative, learning to ignore it is optimal. A model trained on alt-text learns to condition weakly, because weak conditioning was the loss-minimizing behavior on most of its data.

The fix is synthetic recaptioning: run a VLM — a vision-language model, a model that takes an image and produces text about it — over the whole training corpus and have it write a dense, structured description of each image. “Synthetic” here means machine-written rather than found on the web.

The table below sweeps the mixture ratio between original and synthetic captions, on the same architecture and the same compute. The first two columns go up as you add synthetic captions. The third column goes the other way, and that is the whole point of the table.

Rare proper-noun recall is the fraction of prompts naming a specific landmark, brand or character that the model renders correctly.

Keep a slice of original alt-text. The last row is the trap: VLM captions describe what is visible and rarely name it. Alt-text says “Eiffel Tower”; a VLM says “a tall iron lattice tower at dusk.” Drop alt-text entirely and the model forgets proper nouns — landmarks, brands, characters — which is exactly the vocabulary real users type. The 10% slice is also what keeps the style of user prompts (short, sloppy, keyword-ish) inside the training distribution.

5.3 What recaptioning costs

Price recaptioning against the training run it feeds, because the ratio between them is what turns “clean the captions” from a chore into the best purchase in the budget.

The arithmetic uses one unit and one standing rule.

• A TFLOP is a trillion floating-point operations.
• A forward pass through a transformer costs about 2 · parameters · tokens of them, because each parameter participates in one multiply and one add per token.
• Training costs roughly 3x a forward pass, since it adds a backward pass of about twice the cost.

Everything below is that rule applied twice: once to the captioner, once to the training run it feeds.

Side one: the captioning job

corpus              500M images
captioner           7B VLM, ~600 input tokens (image tokens + prompt), 120 output

FLOPs per image  ≈  (600 + 120) · 2 · 7e9         =  1.01e13 = 10.1 TFLOP
effective rate      ~300 TFLOP/s on an H100
                    decode is memory-bound, so apply a 3x penalty overall

GPU-seconds      =  500e6 · 10.1e12 · 3 / 300e12  =  5.05e7 s
GPU-hours        =  5.05e7 / 3600                 =  14,028   -> call it 14,000
dollars          =  14,028 · $2.50                =  $35,069  -> call it $35,000
wall time        =  14,028 / 96 GPUs / 24 hours   =  6.1 days

Two lines there deserve a note:

• Effective rate means the throughput you actually get, not the number on the spec sheet. An H100 is quoted at 300 TFLOP/s here, meaning 300 trillion floating-point operations per second sustained on this kind of work.
• Decode is memory-bound means that when a model emits text one token at a time, the graphics card spends most of its time moving weights out of memory rather than doing arithmetic, so the achieved rate is far below peak. Hence the 3x penalty applied across the whole job.

Side two: the pretraining run it feeds

An image-step is one image passing through the denoiser once during training, so 2 billion of them is the size of the pretraining run.

denoiser            2.6B params, 4,096 latent tokens per image
per forward pass    parameters 21.3 + attention 5.5    =  26.8 TFLOP   (§8)
FLOPs per step   ≈  3 · 26.8                           =  80.4 TFLOP

total FLOP       =  2e9 · 80.4e12                      =  1.61e23
GPU-seconds      =  1.61e23 / 300e12                   =  5.36e8 s
GPU-hours        =  5.36e8 / 3600                      =  148,889 -> ~149,000
dollars          =  148,889 · $2.50                    =  $372,222 -> ~$372,000

Use the same per-pass cost here that Serving uses. The 2 · parameters · tokens rule counts only the work every parameter does on every token. It does not count attention, whose cost grows with the square of the 4,096-token sequence and which Serving puts at 5.5 TFLOP a pass.

That omission is not small. Attention is 5.5 / 26.8 = 21% of the real per-pass number, so pricing training on the parameter term alone understates the run by 26.8 / 21.3 = 1.26x. It understates it in the direction that flatters the recaptioning ratio, which is exactly the direction to be suspicious of.

The ratio between them

$35,069 / $372,222  =  9.4%

Recaptioning is about 9% of the pretraining budget, and it moves prompt adherence more than any architecture change you could buy for the same money. That ratio is the number to have ready. It reframes captioning from “a data-cleaning chore” to “the highest-leverage compute you will spend,” and it is the answer to “how would you improve this model” that does not require a new idea.

Assumptions in this stage.

State out loud: a 500M-image corpus surviving the filters, a 7B captioner at 600 input and 120 output tokens, a 3x penalty for memory-bound decoding, a 2.6B denoiser, and a 2-billion-image-step pretraining run. Every dollar figure above is a product of those, and none of them changes the conclusion’s shape — the ratio stays near 8-12% across a wide range.

Ask, never assume:

• Whether you have the legal right to train on the crawled images at all, and whether the captioner’s licence permits using its output to train a competing model. Both are answers that delete a section of this design rather than adjusting it, and neither has a technical workaround.
• The target market’s languages. A captioner that writes only English silently narrows the model to English prompts.

Load-bearing: that alt-text and synthetic captions are not nested — that alt-text is the only place rare proper nouns live, and synthetic captions are the only place dense grounded description lives.

The 90/10 mix is a union of two disjoint capabilities on that assumption. If synthetic captions were simply better on every axis, the right answer is 100% synthetic and the mixing argument is wasted complexity. The fact that proper-noun recall falls from 0.72 to 0.31 in the last row is the measurement that says they are not.

6. Training

Five parts of the training recipe are specific to text-to-image, as opposed to the diffusion mechanics any image model shares.

Chapter 08 covers the shared latent-diffusion mechanics — noise schedules, samplers, the autoencoder. What is specific to text-to-image:

Resolution curriculum. A curriculum is a training schedule that presents easy work before hard work. Train at 256 pixels until the model has semantics — until it reliably puts the right kinds of things in the picture — then 512, then 1024.

Cost scales with token count, which scales with area, so the gap between the two ends is large:

 256px image ->  32 · 32 latent -> patch 2 ->  16 ·  16  =    256 tokens
1024px image -> 128 ·128 latent -> patch 2 ->  64 ·  64  =  4,096 tokens
                                               4,096 / 256  =  16x per step

cost of 80% at 256 + 20% at 1024, relative to all at 1024:
        0.80 · (1/16)  +  0.20 · 1   =  0.05 + 0.20  =  0.25   -> a 4x saving

That 4x comes at no measured quality cost, because composition is learned at low resolution and only texture needs the high one.

Aspect-ratio bucketing. Aspect ratio is the width-to-height proportion of an image, and users want 16:9 and 9:16, not squares.

Center-cropping — cutting a square out of the middle of a wider image — teaches the model that heads are cropped and text is cut off, because that is what its training images looked like. The model is not making a mistake; it learned the crop.

Instead, bucket the data. Sort every image into one of a handful of aspect-ratio groups. Size the groups so the total token count is roughly constant across them — 1024 · 1024, 1152 · 896, 1344 · 768 all land near a million pixels — and draw each training batch from a single bucket, so every batch is a uniform shape and the GPU never pads.

Conditioning dropout at 10% for CFG (The extrapolation), plus independent dropout of any secondary conditioning signals so each can be supplied or omitted separately at generation time.

EMA of the weights. An exponential moving average (EMA) is a second copy of the weights that is continuously nudged toward the live weights. Here with decay ~0.9999, meaning each update moves the copy one ten-thousandth of the way toward the live weights, so it behaves like an average over the last several thousand steps.

Diffusion sample quality is visibly better from the EMA copy than from the live weights. This is one of the few free wins in the recipe. Budget the VRAM — the graphics card’s onboard memory — for a second copy of the model.

Micro-conditioning on nuisance variables. A nuisance variable is a property of a training image that you do not want the model to associate with the caption. Feed the original resolution, the crop offset, and the aesthetic score in as extra conditioning alongside the text.

That buys two things. The model can explain away “this image is blurry because it was upscaled from 300px” instead of learning that blur is a property of the caption. And you can ask for high aesthetic at generation time without having filtered the training set for it.

Assumptions in this stage.

State out loud: an 80/20 split of steps between low and high resolution, EMA decay of 0.9999, and a handful of aspect-ratio buckets at roughly constant token count. These are recipe settings. Changing them re-prices the run and changes nothing structural.

Ask: what aspect ratios the product actually serves. Bucketing is cheap only if the bucket list is short, and a product promising arbitrary ratios is a different training job.

Load-bearing: that composition is learned at low resolution and only texture needs high resolution. The entire 4x saving of the curriculum rests on it. If high-resolution training turned out to teach composition too, the curriculum is a quality regression rather than a free saving, and the pretraining budget in What recaptioning costs roughly quadruples.

7. Metrics

The measurement stack has three layers: what you can compute offline, what only humans can judge, and which online signals separate the two failure axes.

7.1 Offline: two axes, never one number

Four offline metrics matter, each blind to something different — and the one to lead with is the one that decomposes the prompt.

FID, the Fréchet Inception Distance, is the standard headline number. It works in four steps:

1. Push 50,000 generated images and 50,000 real reference images through Inception-v3, an old image classifier, and take the internal feature vectors it produces.
2. Fit a Gaussian to each set — a bell curve, summarized by a mean vector mu and a covariance matrix S describing how the dimensions vary together.
3. Measure the distance between the two Gaussians using the Fréchet distance, a standard formula for exactly that.
4. Lower is better. 0 would mean the two sets of features are statistically indistinguishable.

In the formula below, subscript g is the generated set and r the reference set, and Tr is the trace, the sum of a matrix’s diagonal entries:

FID = || mu_g - mu_r ||^2  +  Tr( S_g + S_r - 2·(S_g · S_r)^(1/2) )

Four things wrong with it, and you should volunteer at least two:

1. It is a distribution metric with no per-prompt semantics. It cannot tell you whether this image matches this prompt. A model that ignores prompts entirely and generates beautiful images from the reference distribution scores excellently.
2. It moves with the reference set. FID against COCO and FID against a curated aesthetic set rank models differently. An FID without its reference set is not a number.
3. Inception features are ImageNet-shaped. They over-weight object texture and under-weight layout, faces, and text.
4. It prefers low guidance (see the The tradeoff derived rather than asserted table), so optimizing FID actively degrades the thing users notice.

CLIPScore is 100 · cos(CLIP_img(x), CLIP_txt(c)). Run the image through CLIP’s image half and the caption through its text half, take the cosine similarity between the two resulting vectors — the cosine of the angle between them, which is 1 when they point the same way and 0 when they are unrelated — and scale by 100.

It is cheap and it correlates with adherence. It also carries a structural flaw that Why the text encoder outranks the denoiser already set up: it is the same model family whose pooled objective could not represent binding in the first place.

So CLIPScore is close to blind to exactly the errors you most need to detect — attribute swaps, red cube and blue sphere trading colours. It also saturates, meaning it stops discriminating above a certain value; look back at the The tradeoff derived rather than asserted sweep and note that 33.4 and 33.6 are treated as a tie. Everything above ~33 is noise.

VQA-based adherence is the one to lead with. VQA is visual question answering: decompose the prompt into atomic yes/no questions, ask a vision-language model each one, then score the fraction answered correctly. “Atomic” means each question checks exactly one claim, so a wrong answer localizes the failure.

The worked example below uses the chapter’s running prompt. Note that both caught errors are attribute-binding errors — the objects are all present and only the colours are wrong, which is precisely the case CLIPScore would pass:

prompt    "a red cube on top of a blue sphere, in a sunlit room"
questions is there a cube?              yes
          is the cube red?              no    <- caught
          is there a sphere?            yes
          is the sphere blue?           no    <- caught
          is the cube above the sphere? yes
          is the room sunlit?           yes
score     4/6 = 0.67

This catches binding, counting, and spatial relations, which is exactly the set CLIPScore misses. Cost is one VLM call per question, so budget it as an eval-set metric — computed on a fixed few-thousand-prompt benchmark — rather than something you run on every image a user generates.

Human preference is the decision metric. Show a rater two images from two models and force a choice — that is pairwise forced choice — then aggregate. Bradley-Terry is the standard statistical model for turning many such pairwise wins and losses into a single per-model strength score; an Elo rating is that score expressed on the familiar chess scale.

Size the study rather than guessing. The formula below is the standard sample size for detecting a difference in a proportion. Three of its pieces:

• alpha = 0.05, two-sided means you accept a 5% chance of calling a difference that is not there.
• power = 0.80 means an 80% chance of catching one that is.
• z_{a/2} = 1.96 and z_b = 0.84 are the standard-normal values those two choices imply.

Substituting for a 55/45 win rate, so p = 0.55:

detect a 55/45 win rate against 50/50, alpha = 0.05 two-sided, power = 0.80

n  =  (z_{a/2} + z_b)^2 · p(1-p) / (p - 0.5)^2

       (1.96 + 0.84)^2       =  2.80^2         =  7.84
       p(1-p) = 0.55 · 0.45                    =  0.2475
       (p - 0.5)^2 = 0.05^2                    =  0.0025
       0.2475 / 0.0025                         =  99

n  =  7.84 · 99                                =  776 comparisons

Now price it. About 800 pairwise judgments per model pair, times 3 raters for agreement, at ~$0.05 each:

800 · 3 · $0.05  =  $120 per model comparison

That is cheap enough to be the release gate — the check a model must pass before it ships — which is the point of computing it. People skip human evaluation because they assume it is expensive. At this scale it is cheaper than the GPU time that produced the samples.

Always collect preference on two separate questions — “which follows the prompt better” and “which looks better” — because The tradeoff derived rather than asserted already showed those have different optima. A single “which do you prefer” collapses the axes and hands you back a number you cannot act on.

This is also the failure mode that aesthetic fine-tuning walks straight into. A reward model is a model trained on human comparisons to predict which of two outputs a human would prefer, then used as an automatic stand-in for the human.

Train one on pairwise “which looks better” comparisons and it raises the aesthetic score while lowering adherence. The reason is mundane: raters comparing two images rarely re-read the prompt, so “better” quietly means “prettier.” Prompt adherence versus aesthetic quality measures it.

Two questions at collection time is the cheapest possible guard against training on the wrong one.

7.2 Online metrics and A/B

Live traffic teaches you what the offline stack cannot, and one piece of instrumentation separates the two failure axes at essentially no cost. An A/B test is the standard live experiment: send some users the current system, some the new one, and compare outcomes.

In the table below, the two middle rows are the ones to read carefully — they are the same event split two ways, and the split is what makes it diagnostic.

Splitting regeneration by whether the prompt changed separates the two failure axes using a signal you already log. It tells you whether to work on the encoder or on the sampler.

A/B design notes that get asked about:

• Randomize by user, not by request. Outcomes within a session are heavily correlated; per-request assignment understates variance and manufactures significance.
• A new model changes the style prior, so existing users regress on contact. Their saved prompts were tuned against the old model. Read the new-user cohort separately: if new users prefer the new model and existing users do not, you have a migration problem, not a quality problem.
• Novelty effects run about two weeks. A novelty effect is the temporary lift you get simply because something changed and users poked at it. Do not call a win before then.
• Guard on the safety metrics as a gate, not a tradeoff. Blocked rate is the fraction of requests the filters refuse and leak rate is the fraction of prohibited content that got through anyway; both are pass/fail, not part of the preference arithmetic.

Assumptions in this stage.

State out loud: a fixed 2,000-prompt offline evaluation set, ~800 pairwise comparisons at three raters and $0.05 each, and a two-week novelty window. All are dials, and the first is the one most worth enlarging over time.

Ask:

• What the offline evaluation set is supposed to represent — live traffic, a curated hard set, or a marketing demo. A gate stratified over the wrong prompt distribution passes models that are broken on the prompts users actually type, which is worse than no gate because it is believed.
• Whether the product logs enough to distinguish a regenerate with an edited prompt from one without. That single distinction is this section’s main recommendation.

Load-bearing: that human preference, collected on two separate questions, is a trustworthy decision metric.

Every offline number in this chapter is calibrated against it. If raters cannot tell the two questions apart, or if the rater pool does not resemble the user base, the whole measurement stack is anchored to nothing and the guidance scale in The tradeoff derived rather than asserted has no defensible setting.

8. Serving

The request path runs from prompt to delivered image. Walking it box by box lets us derive the cost of one image and rank the levers that move it.

The diagram below is that path, top to bottom. It contains two independent safety checks, one before generation and one after, and the argument for both is made right after the diagram.

Seven labels in it need defining first:

• A request queue holds arriving work until a graphics card is free. Priority by tier means paying customers are pulled from it first.
• Refuse with a reason means a blocked request returns an explanation rather than a silent failure.
• Hold or degrade is the middle option between shipping and refusing: send the image to a human reviewer, or blur it, or return it without the face.
• C2PA is the Coalition for Content Provenance and Authenticity, an industry standard for attaching a signed record of how a file was made.
• A CDN is a content delivery network, a fleet of caches near users that serves the finished image.
• Prompt expansion is an optional step where a small language model rewrites a terse prompt into a richer one before the encoder sees it.

flowchart TD
    P([Prompt]) --> TXTMOD{Text moderation<br/>classifier}
    TXTMOD -->|block| REJ([Refuse + reason])
    TXTMOD -->|pass| ENH[Optional prompt<br/>expansion · small LM]
    ENH --> ENC["Text encoder service<br/>T5-XXL · 9.4 GB<br/>cache by prompt hash"]
    ENC --> Q[[Request queue<br/>priority by tier]]
    Q --> GPU["Sampler pool<br/>2.6B DiT · 28 steps<br/>CFG batched as 2"]
    GPU --> VAE[VAE decode<br/>latent -> 1024px]
    VAE --> IMGMOD{Image NSFW<br/>classifier}
    IMGMOD -->|block| REJ
    IMGMOD -->|pass| FACE{Public-figure<br/>face match}
    FACE -->|match| REVIEW[Hold / degrade]
    FACE -->|clear| WM[C2PA manifest +<br/>invisible watermark]
    WM --> CDN([CDN])

    style TXTMOD fill:#bc6c25,color:#fff
    style IMGMOD fill:#2d6a4f,color:#fff
    style GPU fill:#1d3557,color:#fff
    style REJ fill:#9d0208,color:#fff
    style CDN fill:#2d6a4f,color:#fff

The text encoder is a separate service, and the reason is VRAM rather than latency.

Latency first, to rule it out. Encoding 128 tokens through T5-XXL costs 2 · 4.7e9 · 128 = 1.2 TFLOP, which at 300 TFLOP/s is about 4 ms — 0.08% of the 5.0 s of sampling. Caching by prompt hash saves nothing you would notice.

The real reason is memory. The encoder is 9.4 GB in bf16 — bfloat16, a 16-bit number format, so two bytes per parameter, and 4.7e9 · 2 bytes = 9.4 GB. That is 12% of an 80 GB card you would rather spend on sampler batch size, the number of images the card works on at once.

Work out what splitting it off buys. The denoiser itself is 2.6B params at 2 bytes = 5.2 GB, and whatever is left over after the model weights is what batch can grow into:

with encoder on the card    80 - 9.4 - 5.2  =  65.4 GB free for batch
encoder split out           80 - 5.2        =  74.8 GB free for batch

                            74.8 / 65.4     =  1.14

So the sampler fleet gets ~14% more batch for free. Note that 14% is larger than the encoder’s 12% share of the card. That is because batch comes out of what is left after the 5.2 GB of denoiser weights, and 9.4 GB is a bigger fraction of 65.4 than it is of 80.

The image classifier is the control; the text classifier is a mitigation. The distinction is worth stating plainly: a control is a check on what actually happens, and a mitigation is a check on what someone appears to intend.

The prompt filter catches stated intent and is defeated by euphemism. The output filter looks at what was actually produced. One CLIP-class forward pass on the decoded image is ~0.6 ms against 5.0 s of sampling — 0.012% of the bill. There is no cost argument for skipping it.

Cost per image, derived

Derive the per-image bill from first principles, and the levers that follow can be ranked rather than listed.

Reference backbone: a 2.6B-parameter DiT — a Diffusion Transformer, meaning the denoiser is built from transformer blocks rather than from the older U-Net convolutional design — with hidden width d = 2048, 40 layers, and patch size 2, meaning each 2 × 2 square of the 128 · 128 · 4 latent becomes one token. That is what turns a 128 × 128 latent grid into a 64 × 64 = 4,096-token sequence.

The block below runs the whole bill in one column: tokens, then the two sources of work inside one forward pass, then how many passes, then seconds, then dollars.

tokens          (128/2)^2                       =  4,096

parameter term per forward pass
                2 · 2.6e9 · 4,096               =  21.3 TFLOP

attention term per forward pass
  per layer     4 · T^2 · d = 4 · 4,096^2 · 2048 =  1.37e11
  40 layers                                      =  5.5 TFLOP

per pass        21.3 + 5.5                       =  26.8 TFLOP
NFE             28 steps · 2 (CFG)               =  56
per image       56 · 26.8                        =  1,501 TFLOP

H100 bf16, effective at batch 16                 =  300 TFLOP/s
wall time       1,501 / 300                      =  5.00 s of one GPU
at $2.50/GPU-hour  5.00 · 2.5 / 3600             =  $0.0035 per image

Three notes on that block:

• The two work terms are different in kind. The parameter term is every parameter touched once per token, so it grows linearly in tokens. The attention term grows with the square of the sequence length T, which is why the 4 · T^2 · d line has a T^2 in it and why resolution is expensive.
• NFE is the number of function evaluations from What cfg costs: 28 steps, doubled by CFG.
• The last two lines convert FLOPs to seconds at the card’s effective rate, then seconds to dollars at the rental price. $2.50 / 3600 = $0.000694 per GPU-second.

The rest of the pipeline is rounding error, and it is worth showing that rather than asserting it.

The VAE decode of a 1024 · 1024 latent is 2.2 TFLOP (Cost per image end to end) — 2.2 / 1501 = 0.15% of the denoiser. The reason is a mismatch in both size and repetition: a ~50M-parameter convolutional decoder (one built from small sliding filters rather than attention) runs once, against a 2.6B transformer run 56 times.

The safety classifiers are one CLIP-class pass each. Egress — the charge cloud providers levy for data leaving their network — is ~1.5 MB per image at $0.09/GB.

Adding it all up:

sampler                              $0.0034740
VAE decode  2.2 TFLOP, 0.15%         $0.0000051
safety classifiers  0.6 ms, 0.012%   $0.0000005
egress 1.5 MB at $0.09/GB            $0.0001350
                                     ---------
                                     $0.0036146  -> call it $0.0036 per image
at 60% fleet utilization             $0.0060
four images per request              $0.024 per request

Fleet utilization is the fraction of your rented GPU-seconds that are doing useful work rather than idling between requests. At 60% you pay for 100 seconds to get 60 seconds of sampling, so every figure grows by 1 / 0.6 = 1.67x.

Four images for under three cents, and 96% of it is the denoiser (0.0034740 / 0.0036146 = 96.1%).

Now the levers. Two terms in the table below need defining:

• fp8 is an 8-bit number format, half the width of bf16. Per-tensor scaling is the standard trick for keeping it accurate: give each weight matrix its own multiplier so its numbers land in the narrow range 8 bits can represent.
• p99 latency is the time by which 99 out of 100 requests have finished — the number that describes the unlucky tail rather than the typical case.

Read the table for the ratio between rows, not the absolute cents. The paragraph after it explains which baseline the third column is measured against, and getting that wrong is the usual mistake.

Read the third column against $0.0060, not against $0.0036. Every row is the 60%-utilization figure — the bottom line of the block above — because that is the number you actually pay. The $0.0036 two paragraphs up is the same image at 100% utilization, which no fleet reaches.

Against $0.0060, the fp8 row is a 1.7x saving rather than the no-op it looks like, and the larger-batch row at $0.0039 is a saving too, not a surcharge. Only the denominator changed.

One worked row, so the column is reproducible. Only the sampler share scales with these levers; egress and the classifiers do not:

sampler at 60% util      $0.0034741 / 0.6  =  $0.0057902
everything else at 60%   $0.0001406 / 0.6  =  $0.0002343
                                              ---------
baseline                                      $0.0060245

fp8 at 1.8x throughput   $0.0057902 / 1.8  =  $0.0032168
                              + everything else  0.0002343
                                              ---------
                                              $0.0034511  -> $0.0035

The 768 row is the one people get wrong. Tokens fall from 4,096 to 2,304, which is only 1.78x, but the attention term falls quadratically, so the per-pass cost falls 26.8 / 13.7 = 1.95x and the saving is larger than the token count alone suggests.

The right architecture is two tiers, not one. A 4-step distilled model renders a preview grid in ~0.7 s at $0.0011 each, the user picks one, and the full 28-step model renders the keeper — the image the user chooses to save.

two-tier    4 previews · $0.0011  +  1 keeper · $0.0060  =  $0.0104
single-tier 4 full renders · $0.0060                     =  $0.0240

                                        $0.0240 / $0.0104  =  2.3x cheaper

It is also a 7x improvement in time-to-first-pixel — 8 NFE at 26.8 TFLOP is 0.71 s against 5.0 s — which matters because most generations are discarded before anyone looks closely.

Assumptions in this stage.

State out loud: an H100 at 300 TFLOP/s effective and $2.50/GPU-hour at batch 16, 60% fleet utilization, four images per request, 1.5 MB per image at $0.09/GB of egress, and a 2.6B DiT at 28 steps. An A100 at 150 TFLOP/s and $2.00/GPU-hour doubles the seconds and lands at a higher dollar figure; every row of the cost table moves together and the 96%-is-the-denoiser conclusion does not move at all.

Ask: the traffic shape — requests per second, and how peaky. It sets fleet size and therefore the utilization figure, which is the second-largest multiplier in the bill after the denoiser itself.

Load-bearing: that the denoiser is 96% of the cost. Every optimization in the lever table targets it, and the two-tier preview architecture exists only because of it. If the safety classifiers or the VAE were a comparable share, the ranking of levers inverts and the “there is no cost argument for skipping the output classifier” claim in Safety loses its arithmetic.

9. Failure modes

The four characteristic failures of text-to-image systems are consequences of the conditioning path derived in Conditioning the centerpiece, and each can be explained mechanistically.

Attribute binding — the one to explain mechanistically

Attribute binding is the problem of attaching each property to the right object: red to the cube, blue to the sphere. This one you should be able to explain from mechanism, because the mechanism is exactly the architecture from Cross attention is the mechanism.

The block below has two halves. The top is what you see in the outputs. The bottom is the instrument reading that explains it — look for how sharply the noun rows separate (0.58 vs 0.06) and how flat the adjective rows are (0.31 vs 0.27).

PROMPT   "a red cube on top of a blue sphere"

OBSERVED across 8 samples
  3x   blue cube, red sphere            attributes swapped
  2x   purple cube, purple sphere       attributes merged
  2x   red cube, blue sphere            correct
  1x   red cube, red sphere             one attribute duplicated

cross-attention mass, averaged over layers, step 12 of 28:
  token "red"      0.31 on cube region   0.27 on sphere region
  token "blue"     0.29 on cube region   0.33 on sphere region
  token "cube"     0.58 on cube region   0.06 on sphere region
  token "sphere"   0.04 on cube region   0.61 on sphere region

The numbers under “cross-attention mass” are read off the attention maps of Cross attention is the mechanism: for each text token, how much of its softmax weight lands on the cube’s part of the image versus the sphere’s.

Nouns localize; adjectives do not. “Cube” puts 0.58 on the cube and 0.06 on the sphere — it knows where it goes. “Red” puts 0.31 and 0.27 — it is spread across both, essentially indifferent.

Two mechanisms compound to produce that:

1. The encoder never bound the adjective. From Why the text encoder outranks the denoiser, a contrastively-trained encoder’s feature for “red” barely encodes what it modifies, so the value vector it contributes is roughly “redness,” unattached to any noun.
2. The architecture cannot bind it either. From Cross attention is the mechanism, each spatial location softmaxes over the prompt independently, so nothing prevents the sphere region from drawing on “red.”

The failure is not a bug in the denoiser. It is the absence of any component whose job is binding.

One term before the fixes: regularization is any extra penalty added to an objective to steer a model away from a behaviour you do not want. Four mitigations, in descending order of how much they actually help:

1. A token-level-trained encoder. The largest effect by a wide margin, for the reason in Why the text encoder outranks the denoiser.
2. Attention-map regularization at sample time. A penalty applied during sampling that pushes the attention maps of tokens in the same noun phrase to overlap, and the maps of different noun phrases to separate.
3. Explicit region conditioning. The user or a layout model supplies bounding boxes and each box gets its own text. This sidesteps the problem rather than solving it, by giving spatial structure a channel of its own.
4. Prompt rewriting into separate clauses. Helps least, because it does not change the mechanism.

Counting

This failure is simpler than binding and harder to fix, because the missing piece is not a coupling but a whole faculty.

PROMPT   "exactly seven apples on a wooden table"
OBSERVED 5, 6, 6, 8, 6, 9, 6, 7    (one correct in eight)

Nothing in the conditioning path carries a count the denoiser can check against, and nothing in the sampling loop can count. The model generates a texture of apples at a plausible density and stops.

Training data compounds it. Captions are rarely numerically accurate, so the token “seven” is weakly associated with seven of anything.

Reliable counting needs either an external layout stage that places N boxes, or a verifier loop that counts the output and resamples. Both are system design, not model design — which is why this failure is harder to fix than binding even though it is simpler to describe.

Text rendering — the conditioning half

Text rendering fails for two independent reasons, and this chapter owns one of them. Text rendering traced traces the autoencoder half and measures it: at 8x compression the latent cannot hold a 12-point glyph — the drawn shape of a single character — and the character error rate, the fraction of characters wrong when you read the image back, is set before the denoiser is ever involved. What that section leaves on the table is the conditioning half, which is this chapter’s remit — and it explains a pattern the autoencoder argument alone does not.

PROMPT   "a neon sign that says OPEN LATE"
OBSERVED "OPEN LATF"  "OPFN LATE"  "OPEN LAIE"  "0PEN LATE"

Note the shape of the errors. They are not random noise. They are plausible letters in the right slots: the model knows a sign has eight glyph-shaped things in a row, and roughly which shapes, and it is wrong on one or two.

That is exactly what you get from conditioning that carries the idea of the string but not its characters. A subword tokenizer — the standard text-splitting scheme that breaks words into common fragments rather than into letters — turns LATE into one or two numeric IDs. The embedding the model looks up for each ID encodes a word, not a sequence of letter shapes.

The denoiser is not misdrawing letters it was given. It was never given letters.

That also predicts a size dependence, and gives you a budget you can check:

sign occupies 200 px of a 1024 px image, 8 characters
                        200 / 8   =  25 px per character
VAE downsamples 8x   ->  25 / 8   =  3.1 latent cells per character

A latent cell is one position in the 128 × 128 latent grid. The threshold is roughly 4 cells per character:

• Under ~4 cells per character, the autoencoder ceiling binds. The letterform is not representable in the space the denoiser works in, and nothing on the conditioning path can help.
• Above it, the errors are conditioning errors. The fix is a character-aware encoder run alongside the semantic one — a byte-level model such as ByT5, meaning one whose tokens are individual bytes rather than subwords, so the letters really are in the conditioning.

Diagnose which regime you are in before choosing a fix. Encode a real photo of the sign through the VAE and decode it straight back, with no diffusion in the loop at all. Measure the character error rate of that round trip. If it is already high, the text encoder is not your problem.

Spatial relations

The same shape of argument applies to position, and it produces a result close to chance for every relation except left and right.

PROMPT   "a cat to the left of a dog"
OBSERVED correct in ~55% of samples; near chance for "behind", "under"

Position reaches the denoiser only through the content of text-token values. There is no spatial channel. “Left” is a word whose embedding must somehow bias a softmax over 4,096 positions into a half-plane, and nothing trained it to. Layout-conditioned variants fix this by construction; prompt engineering does not.

The rest

The remaining failures are less about the conditioning path and more about data, sampling and numerics. The table pairs each with a detector — how you find out it is happening — and a guard — what you do about it.

Three notes on that table:

• Anatomy failures are the well-known hand and limb errors. A pose estimator is an off-the-shelf model that locates body joints, and its confidence collapsing is a cheap automatic detector for them.
• Style collapse is the diversity loss from The tradeoff derived rather than asserted showing up concretely, as four near-identical images in one grid.
• NaN is “not a number,” the value a floating-point computation produces when it overflows or divides by zero. At fp8’s narrow range a guidance-inflated eps can overflow, and one NaN spreads through the rest of the arithmetic, so you check for it before decoding rather than shipping a grey rectangle.

Assumptions in this stage.

State out loud: the specific rates quoted — attribute binding correct in 2 of 8, counting correct in 1 of 8, spatial relations ~55%. They are illustrative of the shape rather than fixed constants, and they move with the encoder and the prompt distribution.

Ask: which of these failures the product actually cares about. The fixes are not interchangeable. Reliable counting needs an external layout stage or a verifier loop, which is a different system; binding is largely bought with the encoder choice already made in Conditioning the centerpiece.

Load-bearing: that these are architectural absences rather than training deficiencies — no component whose job is binding, none that carries a count, none that carries position.

If they were merely undertrained, more data and more steps would fix them and every mitigation listed here is wasted effort. The attention-map evidence is what says they are not: an undertrained model would put attention mass in the wrong place, not spread it evenly across both regions.

10. Safety

Three problems usually get grouped together under “safety,” and only the first has a clean technical answer.

Prohibited content. This is the one with a clean technical answer, and it runs at two points.

At ingest: classifier plus hash matching for known illegal material, applied before the data ever reaches a training node, with the removal logged and auditable.

At inference, on both sides: a text classifier on the prompt and an image classifier on the output.

The layering argument is the standard one. The prompt filter is a mitigation; the output filter is the control. A prompt filter can be walked around with euphemism, misspelling, or a foreign language. An output classifier looks at the pixels that would actually ship.

Measure them separately — a text-filter block rate and an image-filter catch rate. The reason is diagnostic: a rising image-filter catch rate means the text filter has been figured out, and you would not see that in a combined number.

Likeness. Likeness is the legal and ethical problem of generating a recognizable image of a real person without their consent. Two directions, and they need different machinery.

Prompt-side: detect public-figure names in the prompt.

Output-side: run a face embedding on any detected face — a vector produced by a face-recognition model, built so that two photos of the same person land close together — and match it against a gallery, a stored set of such vectors for known public figures.

The output check is the one that matters, because “the 45th president” and a plain physical description both route around a name blocklist without ever typing a name.

The threshold is a business decision with arithmetic attached:

Lowering the threshold catches more and annoys more people; raising it does the reverse. State the operating point and who chose it, because “we have a face filter” without a threshold is not an answer. Chapter 10 builds this machinery out properly, since it is a system whose entire purpose is generating one specific person’s face.

Style mimicry. The honest answer is that the architecture cannot resolve this. You can blocklist artist names, and it accomplishes very little: the style was learned from the images, and it is reachable by description.

blocked   "in the style of <living artist>"
works     "thick impasto brushwork, swirling cobalt night sky,
           cypress silhouette, heavy visible palette-knife texture"

Removing the name does not remove the learned style, because the name was never the mechanism — it was an index into a region of the model that description also indexes. The only intervention that actually works is exclusion at training time, honored via an opt-out registry, and even that is imperfect against reproductions in the corpus that the registry does not cover. Say this plainly in an interview rather than pretending a filter solves it; the follow-up is a policy and licensing question, and knowing that it is one is the point.

Provenance. Provenance is the record of where a file came from. Two mechanisms, and they fail in different places:

• A C2PA manifest is a signed block of metadata saying this image was machine-generated and by what. It is metadata, so it dies on the first screenshot.
• An invisible watermark is a pattern embedded in the pixels themselves that a detector can read but a viewer cannot see. It survives resize, crop, and JPEG re-compression at reasonable quality. It dies against a screenshot re-encoded through another generative model.

Be precise about that when asked. Provenance is a supply-chain signal for good-faith platforms, not an adversarial defense, and overclaiming it is a bad look.

Assumptions in this stage.

State out loud: the face-match operating points (0.65 catching ~94% at ~2% false flags, 0.75 catching ~81% at ~0.3%) and the claim that the output classifier costs 0.012% of a generation. Both are measurable and both would be re-measured on your own traffic.

Ask, never assume:

• The jurisdictions you ship in, and the platform’s own policy line. “What counts as prohibited” and “what counts as a public figure” are answers you are given, not ones you derive — and they set the thresholds above, which are not technical choices.
• Whether an opt-out registry for artists is something the business will actually honour, since it is the only style-mimicry intervention with a real mechanism.

Load-bearing: that the output classifier is a control while the prompt classifier is only a mitigation.

That is why the output check is never the thing cut under cost pressure, and it is why a rising image-filter catch rate reads as “the text filter has been figured out” rather than as noise. If prompt filtering were reliable, the layering argument collapses and the cheaper single-sided design would be defensible.

11. Alternatives considered and rejected

Name and price the designs that lost, with the specific number that kills each one. Bolded rows are the ones an interviewer is most likely to raise. Each rejection is a number, not a preference.

Three rows use model families this chapter has not otherwise introduced:

• A GAN, a generative adversarial network, trains a generator against a discriminator that tries to tell real images from fake ones. It makes an image in a single forward pass, against this design’s 56, which is where the ~50x comes from. Mode coverage — how much of the real variety it can produce — is where it fails at open-domain scale.
• An autoregressive image model turns the image into a sequence of discrete tokens and predicts them one at a time like text, using a VQ tokenizer (vector quantization: map each patch to the nearest entry in a learned codebook). Predicting each token conditioned on all the previous ones is why it composes and counts well; doing it 4,096 times in strict sequence is why it is slow.
• A cascade generates a small image, then feeds it to a chain of super-resolution models that enlarge it in stages.

12. Interviewer pushback

The questions this design is most often asked, what each is probing, and the answer to say out loud. Cover the answers and try them first; if you cannot produce one, reread the section it comes from.

“Why does the text encoder matter more than the denoiser?” Testing: whether you understand the conditioning path or just the diffusion path. Because the denoiser never sees the prompt — it sees the encoder’s output vectors through cross-attention, and structure the encoder discarded is unrecoverable downstream. CLIP’s contrastive loss is computed on a pooled embedding, so “a red cube on a blue sphere” and “a blue cube on a red sphere” produce nearly the same pooled vector and there is no gradient pressure to encode the binding. A T5-style token-level objective cannot be solved without representing attachment, so it carries it for free. Measured on our eval set, swapping encoders moved VQA adherence 0.58 to 0.74; doubling the denoiser moved it 0.74 to 0.76.

“Derive classifier-free guidance.” Testing: whether the formula is memorized or understood. The noise prediction is a scaled score. By Bayes, grad log p(c|z) = grad log p(z|c) - grad log p(z), so eps_cond - eps_uncond is the score of an implicit classifier you never trained. To sample from p(z|c)·p(c|z)^(w-1) you add (w-1) copies of that classifier score, which gives eps_uncond + w(eps_cond - eps_uncond). You get eps_uncond by dropping the caption 10% of the time in training, so one network serves both branches.

“So set guidance to 15 and get perfect prompt adherence.” Testing: whether you can derive the tradeoff rather than quote a range. Three things break, for three reasons. p(c|z)^(w-1) is mode-seeking, so diversity collapses — four samples become four copies. The tilted product density has modes that need not lie on the data manifold, so realism drops. And the sampler’s step size assumes eps has roughly unit variance; CFG adds the difference vector w times, so the norm grows linearly, the step over-shoots, and latents leave the range the VAE decoder ever saw — that is the deep-fried look. Practical range is 3-5 with a strong encoder, 6-9 with CLIP only, and use guidance rescale if you want the high end without the saturation.

“Your FID improved. Ship it?” Testing: whether you take a good number at face value. Not on its own. FID prefers low guidance, so an FID improvement is often just a guidance change that made adherence worse — on our sweep, FID minimizes at w=2 and human preference peaks at w=5. FID also has no per-prompt semantics: a model that ignores prompts entirely and samples beautiful images from the reference distribution scores well. I would look at VQA adherence and CLIPScore alongside it, and gate on ~800 pairwise human comparisons, which is about $120 and is the only metric with the right sign.

“What is your cost per image, and where does it go?” Testing: whether you can do the arithmetic live. A 2.6B DiT over 4,096 latent tokens is 21.3 TFLOP of parameter work plus 5.5 of attention, so 26.8 per forward. 28 steps with CFG is 56 passes, so 1,501 TFLOP per image. At 300 TFLOP/s effective that is 5.0 seconds of an H100, about $0.0035 at $2.50 an hour, roughly $0.0036 all-in and $0.0060 at realistic utilization. CFG is exactly half of it, which is why guidance distillation is the first optimization, and step distillation to a 4-step preview tier is the second.

“Explain the red-cube-blue-sphere failure without hand-waving.” Testing: mechanism, and whether you have actually debugged one of these. Two causes that compound. The encoder’s feature for “red” barely encodes what it modifies, for the pooling reason above. And in cross-attention every one of the 4,096 spatial locations softmaxes over the prompt independently — there is no coupling that says “the region attending to red must be the region attending to cube.” Read the attention maps and you see it directly: nouns localize at ~0.6 mass on their region, adjectives sit near 0.3 on both. The fixes that work are a token-level encoder, attention-map regularization within noun phrases, or an explicit layout channel. Rewriting the prompt does not touch the mechanism.

“Why can it write a poster headline but not a street sign?” Testing: whether you reason about the latent, or treat the model as a black box. Resolution budget. A sign occupying 200 pixels of a 1024-pixel image with 8 characters is 25 pixels per character, and the VAE downsamples 8x, so 3.1 latent pixels per glyph. Under about 4, the letterform is not representable in the space the denoiser works in. That is why the same model nails a headline at 300 pixels tall. Fixes follow from the derivation: generate larger, use a 4x-compression VAE for a text-region refinement pass, or add character-aware conditioning.

“How much of your budget goes to data work?” Testing: whether you know where the leverage is. Recaptioning 500M images with a 7B VLM is about 14,000 GPU-hours, roughly $35k, against a ~149,000 GPU-hour pretraining run. So about 9% of the budget, and it moves adherence more than any architecture change I could buy for the same money. The reason is in the loss: there is no adherence term, so if captions are uninformative, learning to ignore them is optimal. Fix the captions and conditioning starts to matter during training instead of only at sample time.

“You went 100% synthetic captions. What breaks?” Testing: whether you have run this or read about it. Proper nouns. A VLM describes what is visible and rarely names it — it says “a tall iron lattice tower at dusk,” not “the Eiffel Tower.” Go fully synthetic and rare-entity recall falls from about 0.72 to 0.31, and that vocabulary is exactly what users type. Keep 10% original alt-text, which also keeps the short, keyword-ish prompt style inside the training distribution.

“A user reports the model copies a living artist’s style. What do you do?” Testing: whether you overclaim on safety. I would be honest that a name blocklist mostly does not work. The style was learned from images and is reachable by description — block the name and “thick impasto, swirling cobalt night sky, palette-knife texture” gets you there. The only intervention with a real mechanism is exclusion at training time via an opt-out registry, and even that misses reproductions in the corpus. So: honor the registry, log the exclusions, and treat the rest as a licensing question rather than pretending a classifier resolves it.

“Which safety control would you cut under cost pressure?” Testing: whether you know which controls are load-bearing. None of the output-side ones, and the arithmetic is why: the image classifier is one CLIP-class forward at roughly 0.6 ms against 5.0 seconds of sampling — 0.012% of the bill. The prompt classifier is even cheaper. If something has to go it would be prompt expansion, which costs a small LM call and measurably raises the regenerate-with-edit rate on prompts it rewrote. The output classifier is the control; the prompt classifier is a mitigation. Cutting a control to save 0.012% is not a cost decision.

“How do you A/B a new base model?” Testing: experiment design under a distribution shift you caused. Randomize by user, not request, because within-session outcomes are correlated. Track keeper rate as the headline and split regenerate rate by whether the prompt was edited, since edited means adherence and unedited means sampling variance. And read new users separately from existing ones: a new base model changes the style prior, so users whose saved prompts were tuned against the old model regress on contact. If new users prefer it and existing users do not, that is a migration problem, and the fix is a version pin, not a model rollback.

The assumption ledger

One table collects every assumption the chapter relies on, so you can state the design’s foundations and say what replaces the design when each one fails.

Each is sorted into one of three bins, the same three used in ch 01:

• State it — you are free to pick, and being wrong costs a re-derivation.
• Ask it — the answer changes the architecture, so it is worth an interviewer’s time.
• Load-bearing — if it is wrong, the design is not suboptimal, it is invalid.

The table is sorted with the load-bearing rows first. Those are the nine you should be able to recite.

The sentence that makes this visible to an interviewer: “This design rests on three things. One, that the text encoder is frozen — so anything its training objective threw away is gone before the denoiser sees the prompt, and that is why I spend parameters there rather than on the denoiser. Two, that the training loss contains no term for whether the image matches the caption — which is simultaneously why recaptioning is the best 9% of the budget I can spend and why classifier-free guidance has to exist at all. Three, that no single number measures quality here, so I gate on two human questions and a per-prompt adherence metric rather than on FID, which would actively push me toward the wrong guidance scale.”

Next: 10 — Personalized Headshot Generation — what changes when the subject is a specific person and the model has to be built per user.