11 — Text-to-Video

Generating a five-second clip the obvious way costs about two thousand times more than a single image: 1,483 PFLOP against 0.77 PFLOP. Done well, it comes down to about 100 times the compute and 113 times the delivered cost. Every one of these numbers is derived below.

Video is not images plus one axis. The compute arithmetic decides the product shape before any modelling choice is made: generation is asynchronous instead of interactive, metered instead of unlimited, tiered instead of single-quality, and evaluated by humans because every automated metric is blind to the failures that matter.

The goal. Design a system that turns a sentence of text into a five-second video clip, and work through the arithmetic that decides what the product can be: why video generation runs in a queue instead of answering instantly, why it sells credits instead of unlimited use, why it ships a cheap draft alongside an expensive final render, and why the only trusted evaluation is a person comparing two clips side by side.

Chapter 09 built the same machinery for still images and chapter 10 built the serving and cost discipline. Neither is a prerequisite here; each idea they supply is restated before it is used, with a link to the longer derivation.

1. Problem framing

Before any architecture, fix exactly what goes into the system, exactly what comes out, and the handful of terms the rest of the chapter uses without further explanation.

1.1 What goes in and what comes out

In goes a sentence. The user types a text prompt — say, “a woman in a navy blazer speaking to camera, static shot” — and may optionally attach one still image to be used as the clip’s first frame.

Out comes a video file. Five seconds long, at 24 frames per second (fps, the number of still images displayed each second), each frame 1,024 pixels wide by 576 tall, and the motion in it has to read as one continuous shot rather than a slideshow. Five seconds at 24 fps is 120 frames, and that one number drives everything below.

Three constraints bound the design. The clip must look like a single continuous shot. The cost of producing one clip has to fit inside what a consumer subscription can charge. And the wait has to be short enough that people come back tomorrow.

Why it is hard is that those 120 frames must each be individually plausible and mutually consistent, that you have to train that behaviour from a corpus whose captions were never written by anyone, and that you then have to measure it with an evaluation suite in which no automated number can see the failure that actually loses users.

1.2 The machinery this chapter assumes, in one page

A short vocabulary carries the whole chapter. Each term is defined here once and then used freely. If you already know them, skip to The opening move and the three reframes behind it.

How a diffusion model makes a picture

A diffusion model generates a picture by starting from pure random noise and removing a little of it at a time.

Training teaches one network a single narrow skill: shown a noisy picture and told how noisy it is, predict the noise that was added. That is the whole training objective — there is no “draw a cat” instruction anywhere in it.

Generation runs that one network repeatedly. Each pass is a sampling step, and after a few dozen steps the noise is gone and a picture is left.

The quantity the network outputs is written eps, the Greek letter epsilon, the conventional symbol for a noise term. So eps prediction means “the network’s guess at the noise currently sitting in this input.” When a diagram below ends in a box labelled eps prediction, that is what came out.

Latents, and why nobody runs diffusion on pixels

Running that denoising loop on full-resolution pixels is wasteful, so production systems run it on a latent — a compressed numeric stand-in for the picture, typically eight times smaller on each side.

The compressor and decompressor are a variational autoencoder (VAE): an encoder that maps pixels down to the latent, a decoder that maps the latent back to pixels, trained together so the round trip loses as little as possible.

Running diffusion on that latent instead of on pixels is latent diffusion. Doing it on a latent that is compressed in time as well as in space — several frames squeezed into one latent frame — is latent video diffusion. That is the architecture this entire chapter builds toward, and Spatiotemporal latent compression is where it gets priced.

The denoising network is a transformer

The denoiser itself is a diffusion transformer (DiT) — a transformer, the same architecture that powers language models, applied to image or video latents instead of words.

It chops the latent into a grid of small squares called patches and treats each patch as one token: one position in a sequence, exactly as a word is a token for a language model. Everything expensive in this chapter is counted in tokens.

Tokens exchange information through attention. Self-attention lets tokens read each other. Cross-attention lets each token read the encoded text prompt, and it is the only mechanism by which the prompt influences the picture at all.

Two numbers describe such a network: d, the width of the vector carried per token, and the number of blocks stacked on top of each other. This chapter uses d = 2048 with 40 blocks for the borrowed image model and d = 3072 with 40 blocks for the production video model.

Guidance, which doubles every bill

Classifier-free guidance (CFG) is the knob that makes the output actually follow the prompt. Each sampling step is run twice, once with the prompt and once with an empty prompt, and the difference between the two predictions is amplified and added back. That pushes the sample toward what this prompt implies and an empty prompt does not.

CFG doubles the work of every step. That is why every pass count below is the step count times two: 28 steps is 56 passes, 50 steps is 100 passes.

Temporal consistency

Temporal consistency means that the same thing in the world looks like the same thing from one frame to the next: the same face, the same shirt colour, the same car.

It is not a smoothing filter and it is not a post-processing step. The flipbook baseline shows it is a property of which distribution you modelled, which is why it cannot be bolted on afterwards.

Units and prices

Every cost claim here is arithmetic rather than assertion, so the units matter.

A FLOP is one floating-point operation, a single multiply or add. A TFLOP is a trillion of them and a PFLOP is a thousand TFLOP. “GPU-s” means GPU-seconds: one second of one chip.

This repo prices compute at two standing rates. An NVIDIA H100 accelerator sustains about 300 TFLOP/s and rents for $2.50 per GPU-hour. The older A100 sustains 150 TFLOP/s at $2.00. Every dollar figure below is one of those rates applied to a FLOP count printed next to it, so you can always check it.

1.3 The opening move, and the three reframes behind it

An interview on this problem is won or lost in the first two minutes, because the arithmetic forecloses most of the design space before anyone gets to state a preference.

First thing to say: “Let me do the arithmetic first, because it determines the architecture. A five-second clip is 120 frames. Generating them independently is 120x an image and produces a flipbook. Modelling them jointly with full 3D attention is 14,400x on the attention term, which is unaffordable. The design is forced: factorize the attention and compress the latent in time as well as space. Those two moves cut the attention term by about 470x and the clip by 19x end to end — the gap is because I spend some of the saving on a bigger model — and everything else follows from what is left.”

Two phrases in that script are the whole design and are derived in full below. Factorizing the attention means letting each token look at the other tokens in its own frame, and separately at the same spot in other frames, instead of at all tokens everywhere (Factorized attention). Compressing the latent in time means having the VAE represent four consecutive frames with one latent frame (Spatiotemporal latent compression).

Three reframes separate a candidate who has done this from one who has read about it. In each row, the naive view is not wrong so much as pointed at the wrong quantity.

Assumptions this framing rests on. Five seconds at 24 fps, so 120 frames, at 1,024 · 576. A consumer subscription around $9.99 a month rather than an enterprise contract. An H100 at 300 TFLOP/s and $2.50 per GPU-hour. Change the clip length or the frame rate and every number in Why video is not images plus one axis moves; change the accelerator and only the dollars move.

2. Why video is not “images plus one axis”

The obvious architecture is unaffordable, and it takes four steps to see why and to see what the affordable one looks like. Each step is one design and one cost, and the four together take a five-second clip from $3.43 to $0.178 while making the model larger.

2.1 The flipbook baseline

Start with the cheapest thing that could possibly work: run a still-image generator 120 times, once per frame. It is called the flipbook baseline because that is what it produces — a stack of individually fine pictures with no relationship to each other.

Take the image backbone from chapter 09 — a diffusion transformer with 2.6 billion parameters, token width d = 2048, and 40 stacked blocks — and run it once per frame at 1024 · 576.

Three pieces of setup, because the same three recur in every cost block in this chapter.

How many tokens is one frame. At 1024 · 576 the VAE’s eight-fold spatial compression produces a latent 128 by 72. Cutting that into 2 · 2 patches leaves a 64 by 36 grid. And 64 · 36 = 2,304 tokens for one frame.

The parameter term: 2 · params · tokens. This is the work of pushing every token through every weight. Each weight is used once per token, and using it costs two operations — one multiply and one add — hence the 2.

The attention term: 4 · tokens² · d · layers. This is the work of letting every token look at every other token. Per layer, attention does two passes over the full pair matrix: once to score every token against every other token, once to mix the values using those scores. Each pass is tokens² · d multiply-add pairs, which is 2 · tokens² · d operations, and two passes make 4. Multiply by the number of layers because every block does it again.

Now the numbers. Each line is the one above it carried one step further, and the last line is the price of one clip.

latent     128 · 72,  patch 2   ->  64 · 36  =  2,304 tokens per frame

per pass   parameter  2 · 2.6e9 · 2,304                  =  11.98 TFLOP
           attention  4 · 2,304^2 · 2,048 · 40 layers    =   1.74 TFLOP
                                                            -----
                                                            13.72 TFLOP
28 steps · 2 (CFG)  =  56 passes                         =    768 TFLOP per frame
120 frames                                               = 92,200 TFLOP  =  92.2 PFLOP
at 300 TFLOP/s, $2.50/GPU-h                              =    307 GPU-s  =  $0.21

Twenty-eight sampling steps, each run twice for guidance, is 56 passes per frame. Note the split in the per-pass total: attention is 1.74 of 13.72 TFLOP, about 13%. That share is the thing that changes in The blow up derived.

Exactly 120x an image, and the output is unusable. Each frame draws its own independent starting noise, so the model draws 120 independent samples from p(frame | prompt) — read that as “the probability of a frame, given the prompt,” the distribution over single frames that this model was trained to represent. Textures reshuffle, faces change, backgrounds shift. Nothing in the procedure couples the frames, because nothing in the procedure could.

Two repairs suggest themselves immediately. Neither works.

Sharing a seed across frames does not fix it. A seed is the number that determines which random noise you start from, so sharing one makes all 120 frames start from the same noise. But the starting noise is not the sample. Diffusion is chaotic in the initial condition: a tiny difference at the start is amplified into a large difference at the end. Two starting latents differing by 1% end up as visibly different images.

Warping frame t into frame t+1 does not fix it either. Optical flow is an estimate, for every pixel, of where that pixel moved to in the next frame, and warping means dragging the pixels along those estimates. It cannot invent content that enters the frame from outside. And it fails completely at occlusion boundaries — the edges where one object passes in front of another and pixels genuinely disappear and reappear — which is exactly where the eye looks.

The general statement is worth pinning down, because everything in the rest of the chapter is a consequence of it.

The joint distribution over all 120 frames is the probability of a whole clip. The marginal distribution is the probability of a single frame considered on its own, with the other 119 forgotten. A per-frame model only ever learned the marginal, and a marginal contains no information about what any other frame looked like.

Temporal consistency is not a polish step you add to a per-frame model. It is a property of the joint distribution, and you get it by modelling the joint distribution or you do not get it.

Assumptions this stage rests on. A 2.6B-parameter image backbone is available to reuse, the VAE compresses 8x on each spatial side, patches are 2 · 2, and 28 sampling steps with guidance is enough for an acceptable still. If your image model needs 50 steps rather than 28, every figure in this subsection scales by 50/28 and none of the conclusions change.

2.2 The blow-up, derived

If consistency requires the joint distribution, model the joint distribution and see what it costs. The honest version — every token in the clip attending to every other token — produces a number no consumer product can carry.

Model all 120 frames at once, with attention over the full spatiotemporal volume — the whole block of tokens, 120 frames deep and 2,304 tokens wide, treated as one flat sequence.

Nothing about the model changed. Same 2.6B parameters, same d = 2048, same 40 blocks, same two formulas from The flipbook baseline. The only thing that changed is the token count, from 2,304 to 276,480. The effect on each of the two terms:

tokens     120 · 2,304                       =  276,480

parameter term    2 · 2.6e9 · 276,480        =  1,438 TFLOP    (120x one frame)
attention term    4 · 276,480^2 · 2,048 · 40 = 25,050 TFLOP    (14,400x one frame)
                                                ------
per pass                                       26,488 TFLOP
56 passes                                    = 1,483 PFLOP
                                             = 4,944 GPU-s  =  82 minutes  =  $3.43

The parameter term grows linearly in frames; the attention term grows quadratically. The reason is mechanical. Every token still has to pass through every weight exactly once, so 120x the tokens is 120x the parameter work. But every token now has to compare itself against every other token, and the number of pairs among n things grows as n² — so 120x the tokens is 120² = 14,400x the attention work.

That inversion is the whole point. Attention is 13% of the cost for one image (1.74 of 13.72 TFLOP) and 95% of the cost for a 120-frame clip (25,050 of 26,488). Video architectures look different from image architectures because of exactly this: you are no longer optimizing a model, you are optimizing a sequence length.

And $3.43 a clip is not a product. A $9.99 subscription would buy fewer than three clips a month before paying for anything else. The next two subsections are the two ways out.

Assumptions this stage rests on. All 120 frames are generated in one pass rather than in overlapping windows, and attention is dense rather than sparse or windowed. Both are the honest starting point: they are what “model the joint distribution” means before you approximate it, and the next two subsections are the approximations.

2.3 Factorized attention

The first of the two saving moves splits one expensive attention operation into two cheap ones, and it turns out to cost almost nothing in quality. The saving is a factor of 114, and what you gave up for it can be stated precisely.

Instead of attending over the whole volume, attend twice.

Spatial attention: each token sees the other tokens in its own frame. That is 120 independent attention problems, each over 2,304 tokens.

Temporal attention: each token sees the token at the same spatial position in every other frame. That is 2,304 independent attention problems, each over 120 tokens.

Two smaller pair-counting problems have replaced one enormous one. In the block below, “120 groups of 2,304 tokens” means the tokens² in the attention formula is now 2,304², computed 120 times — not (120 · 2,304)² computed once.

spatial   120 groups of 2,304 tokens
          120 · 4 · 2,304^2 · 2,048 · 40   =  208.8 TFLOP

temporal  2,304 groups of 120 tokens
          2,304 · 4 · 120^2 · 2,048 · 40   =   10.9 TFLOP
                                              ------
                                              219.7 TFLOP   vs 25,050 full 3D

                                              -> 114x cheaper
per pass  1,438 (parameter) + 220            = 1,658 TFLOP
56 passes                                    =  92.8 PFLOP  =  309 GPU-s  =  $0.21

Compare the two lines to see where the saving comes from. Spatial attention now counts pairs within 2,304 tokens, 120 separate times, which is 120 · 2,304² pairs instead of (120 · 2,304)². Temporal attention counts pairs within 120 tokens, 2,304 separate times. Both are linear in the thing the full version squared.

Factorized attention makes a temporally-joint model cost the same as the flipbook. That is worth saying out loud in an interview: consistency is not something you pay for once you factorize, because the term you were afraid of was never the model, it was the T^2 — T being the number of frames, and T^2 the number of frame-to-frame pairs that full attention insists on computing.

What you give up is exact and worth stating precisely. In one layer, a token can only reach another token that shares a frame with it or shares a spatial position with it. Information travels diagonally through the volume, one hop per layer.

Work an example. To get from position p in frame 0 to position q in frame 119, information takes a temporal hop from (p, frame 0) to (p, frame 119), then a spatial hop from (p, frame 119) to (q, frame 119). Two hops, so two layers. The stack has 40 blocks alternating spatial and temporal attention, which is twenty times the routing depth any pair of tokens actually needs. That is why the approximation holds in practice rather than merely on paper.

Assumptions this stage rests on. Spatial and temporal blocks alternate through the stack rather than being segregated, and the network is deep enough (40 blocks) that two-hop routing has room to mix. A 12-block model factorized this way would genuinely lose long-range coherence, so the depth is doing load-bearing work here, not just capacity.

2.4 Spatiotemporal latent compression

The second saving move shrinks the sequence itself rather than the operation over it, and it is the only lever in the system that touches a quadratic term. It buys a four-fold token reduction — which has to be kept carefully separate from the model growth that happens at the same moment.

Compress the latent in time as well as in space. The compressor is a causal 3D VAE: 3D because it compresses across height, width and time rather than only height and width, and causal because it looks only backwards in time, encoding frame 0 by itself and then successive groups of four. The ratios are 8 · 8 spatially and 4x temporally. The spatial half is exactly the latent-diffusion construction of Latent diffusion derived, unchanged; only the time axis is new:

120 frames  ->  1 + (120 - 1)/4  ~=  30 latent frames
tokens      30 · 2,304            =  69,120      (4x fewer)

The first line is the causal encoding written out: one latent frame for frame 0 on its own, plus one for each group of four after it. That is 1 + 119/4 = 30.75, and every number below uses the round 30 — the same as 120/4. The 2.5% difference does not move any conclusion, and carrying 30.75 through the arithmetic would obscure the 4x.

Note what the second line does not say: the frame count did not drop. The clip is still 120 output frames. It is now represented by 30 latent frames, and the decoder expands it back to 120 at the end.

Production video models are larger than the image backbone borrowed above, so the compressed latent is priced here against a realistic one — 5 billion parameters, token width d = 3072, 40 blocks — and the sampler is run at 50 steps rather than 28, again doubled for guidance. Two things get worse in this block (more parameters, more steps) and one gets much better (4x fewer tokens):

parameter   2 · 5e9 · 69,120                     =  691 TFLOP
spatial     30 · 4 · 2,304^2 · 3,072 · 40        =   78 TFLOP
temporal    2,304 · 4 · 30^2 · 3,072 · 40        =    1 TFLOP
                                                    ----
per pass                                            770 TFLOP
50 steps · 2 (CFG) = 100 passes                  = 77.0 PFLOP
at 300 TFLOP/s                                   =  257 GPU-s  =  $0.178

At fixed model size the saving decomposes cleanly into three numbers you should be able to produce on demand:

• Parameter term: 4x. It is linear in tokens, and there are 4x fewer tokens.
• Spatial attention: 4x. Each group is still 2,304 tokens, but there are 30 groups instead of 120.
• Temporal attention: 16x. Each group went from 120 tokens to 30, and attention is quadratic: (120/30)² = 16.

Temporal compression is the only lever in the system that hits a quadratic term. That is why it is worth more than any parameter reduction.

Now the caveat, because the block above is not at fixed model size. Token width d is 3,072 rather than 2,048 and the parameter count is 5B rather than 2.6B. So the per-pass cost falls from 1,658 to 770 TFLOP — a factor of 2.15, not 4.

The compression bought roughly 4x and the model spent about half of it. Keep those two statements apart. The temptation to multiply the clean decomposition by anything else is where the arithmetic in this section usually goes wrong, and an interviewer who is paying attention will ask you to separate them.

Assumptions this stage rests on. A 4x temporal compression ratio, a 5B model at d = 3072, and 50 sampling steps. The 4x is the load-bearing one: it sets the token count, the aliasing limit in What temporal compression costs you, and the high-motion tier’s existence. The 5B and the 50 steps only move the dollars.

2.5 The escalation, in one table

All four designs, side by side, so the path from the unusable cheap one to the affordable good one is visible in a single view. Read the last column first: it is the only one that says whether the output is a video at all.

flowchart LR
    A["Per-frame<br/>$0.21<br/>NO consistency"] -->|"model the joint<br/>distribution"| B["Full 3D attention<br/>$3.43<br/>T^2 term is 95%"]
    B -->|"factorize:<br/>spatial + temporal<br/>114x on attention"| C["Factorized<br/>$0.21<br/>consistency is now free"]
    C -->|"4x temporal VAE<br/>4x params · 4x spatial<br/>16x temporal<br/>at fixed model size"| D["Compressed latent<br/>5B model<br/>$0.178"]

    style A fill:#9d0208,color:#fff
    style B fill:#bc6c25,color:#fff
    style C fill:#1d3557,color:#fff
    style D fill:#2d6a4f,color:#fff

The diagram traces the same four designs as a path. The per-frame design is cheap at $0.21 and delivers no frame consistency at all, which is why it is coloured as a failure rather than as a starting point. Modelling the joint distribution fixes consistency and costs $3.43, with the T^2 term accounting for 95% of it. Factorizing brings the price back to $0.21 with consistency intact — which is the sense in which consistency is now free: you are paying the flipbook’s price and getting a coherent clip. Compressing the latent then buys a larger model and a cheaper clip at $0.178.

Row 2 to row 4 is 19.3x on the clip — 1,483 PFLOP to 77.0 — and it is entirely sequence-length engineering. The model got bigger along the way and the clip got cheaper.

One trap to disarm before you quote any of this in an interview: the two savings do not multiply.

Factorization is 114x on the attention term (25,050 -> 220 TFLOP per pass). Temporal compression is 4x on the token count. Those are ratios of different things — one is a share of a pass, the other is a count of tokens — so 114 · 4 = 456 is not a number that describes anything.

Pick one quantity and chase it through both moves instead. There are two honest answers, and they differ by a factor of 25.

• Chase the attention term, holding d = 2048 so the model size does not contaminate the comparison. Spatial attention over 30 groups is 52.2 TFLOP, temporal attention over 30-token groups is 0.7, so 52.9 total against 25,050. That is 474x.
• Chase the whole clip, model growth and all. 1,483 PFLOP down to 77.0. That is 19.3x.

The gap between 474 and 19.3 is everything that is not attention. Factorization does not touch the parameter term at all, the 5B model makes that term larger, and the sampler runs 100 passes instead of 56.

2.6 What temporal compression costs you

Every saving in Spatiotemporal latent compression has a bill, and this one’s is exact: there is a speed of motion above which your system physically cannot represent what happened, and you can compute that speed from the compression ratio alone.

Compression in time is lossy in time, and the loss has a name. Two pieces of vocabulary make the block below readable.

Hz (hertz) means “times per second”.

The Nyquist limit is a result from signal processing. If you record something at a rate of f samples per second, the fastest repeating motion you can faithfully capture is f/2 repetitions per second, because you need at least two samples per cycle to know a cycle happened at all.

Motion faster than that does not vanish. It comes back wearing a disguise, appearing as some slower motion that was never there. That disguise is called aliasing.

Two divisions do all the work below. The latent carries 24 fps divided by 4, so 6 independent samples per second. Nyquist halves that, so 3 Hz.

output frame rate                              24 fps
temporal compression                            4x
independent temporal samples in the latent      6 Hz     (24 / 4)

Nyquist limit of the latent                     3 Hz     (6 / 2)
a wheel spinning at 3 revolutions/second        aliases

Motion faster than about 3 Hz cannot be represented in the latent and comes back as aliasing — the wagon-wheel effect, in which a spinning wheel appears to rotate slowly backwards; strobing on fast camera pans; and smeared limbs on running figures. This is not a model defect that more training fixes; it is a sampling-rate consequence of the compression ratio you chose, and it would still be there if the model were perfect. It is the honest answer to “why do fast motions look bad”, and it names its own fix: run a high-motion tier at 2x temporal compression instead of 4x, which doubles the token count and therefore costs roughly 2-4x more.

The choice of a causal VAE deserves its own justification, because it looks like a detail and is not. Encoding frame 0 by itself and then successive groups of four means a single still image is already a valid one-frame video, with no special casing anywhere in the code. That lets you train one model on images and video jointly — which matters enormously, because Video text pairs barely exist shows there is roughly a hundred times more usable image data than video data, and a video-only model throws all of it away.

Assumptions this stage rests on. A 24 fps output and a 4x temporal compression ratio, which together fix the 3 Hz ceiling. Nothing about the model changes this number — it is arithmetic on the sampling rate — so the only way to raise it is to compress less, which is why the high-motion tier exists as a product decision rather than a training one.

3. Architectures

One block of the production denoiser first, then the five architectures people reach for instead — so you can say not only what you would build but what each alternative gets wrong.

The diagram below is what Why video is not images plus one axis derived, drawn as data flowing through it. The input is a noisy video latent, 30 latent frames of 2,304 tokens each, and four stages act on it in order.

1. Spatial self-attention. Tokens in the same frame read each other. 30 groups of 2,304.
2. Cross-attention to the text. The same mechanism Cross attention is the mechanism uses for still images, and the only place in the entire network where the prompt enters.
3. Temporal self-attention. The token at a given spatial position reads that position in all the other frames. 2,304 groups of 30.
4. Feed-forward. A small two-layer network applied to each token independently. Most of the model’s parameters live here, which is why the parameter term of Spatiotemporal latent compression is 691 of the 770 TFLOP per pass.

That sequence repeats for 40 blocks. The last one emits the eps prediction, the network’s estimate of the noise to subtract at this sampling step. The arrow looping back to the input in the diagram is the 40-block repeat, not a recurrence in time.

flowchart TD
    IN["Noisy video latent<br/>30 latent frames · 2,304 tokens each"] --> SP["Spatial self-attention<br/>within each frame<br/>30 groups of 2,304"]
    SP --> XA["Cross-attention to text<br/>same mechanism as ch 09"]
    XA --> TP["Temporal self-attention<br/>across frames at each position<br/>2,304 groups of 30"]
    TP --> FF["Feed-forward"]
    FF -->|"· 40 blocks"| IN
    FF --> OUT["eps prediction"]

    style SP fill:#1d3557,color:#fff
    style TP fill:#bc6c25,color:#fff
    style XA fill:#40916c,color:#fff
    style OUT fill:#2d6a4f,color:#fff

Six approaches, priced against each other. Temporal receptive field means how many frames away a given output position is able to draw information from — one frame means it knows nothing about its neighbours; “full clip” means it can in principle see all 120. Relative cost is normalized so the production answer — the $0.178 clip of Spatiotemporal latent compression — is 1.0. Every row near 1.0 is in that same cost class. Only full 3D attention is not, and that row is the $3.43 clip.

Two rows need unpacking. A 3D convolution slides a small fixed-size box — the kernel — over height, width and time, combining whatever falls inside it; because the box is small, each output can only see three to five frames in either direction, which is what “kernel-limited” means. Inflation means taking a trained image model, copying its two-dimensional layers into a three-dimensional shape that accepts a stack of frames, and inserting new temporal layers between them.

Inflation is the tempting shortcut and the one to argue against carefully. Initializing from a trained image model and freezing the spatial layers gets you visual quality for free and trains in a fraction of the compute — genuinely attractive. The problem is structural: the spatial layers learned the marginal distribution of single frames, and a marginal has no notion of a before or an after. Long-range coherence, object permanence — the property that a thing which goes behind a pole is the same thing when it comes out — and physical plausibility all have to be carried by the thin temporal stack you bolted on. Object permanence is the failure that inflation cannot fix, and unfreezing everything for a joint training run is what buys it.

Most 3D convolutions in a production model live in the VAE rather than the denoiser, where their locality is a virtue rather than a limit: a decoder needs local smoothness in space and time, not long-range reasoning about what happened four seconds ago.

Assumptions this stage rests on. That you can afford to train the spatial layers rather than freeze them, which is what rules inflation out. If the budget genuinely only supports fine-tuning a frozen image backbone, inflation is the right answer and object permanence is the known defect you ship with — say that out loud rather than pretending the architecture has no cost.

4. Conditioning and motion control

Conditioning means every input other than the noise that steers what gets generated. Video adds its own signals — the first frame chief among them, worth more than any architecture change — and the same machinery explains why stitching clips end to end drifts and what to do instead.

Text conditioning is the cross-attention of Cross attention is the mechanism, unchanged: the prompt is encoded once into a sequence of vectors and every token in the latent reads from it at every block. The video-specific signals are where the quality actually comes from, and the table below is roughly in order of how much each is worth.

Four of those rows use vocabulary worth spelling out.

• I2V is image-to-video: the mode where the user supplies a picture and the system animates it.
• Micro-conditioning means feeding the model a single number alongside the prompt during training — motion strength, frame rate, an aesthetic score. At generation time you dial that number and get a predictable change without retraining anything.
• 6-DoF is six degrees of freedom: the six numbers that fully describe where a camera is and which way it is pointing, three for position and three for rotation.
• Control latents are per-frame encoded versions of a driving signal — a depth map, a stick-figure pose, an edge drawing — added into the early blocks so the output is forced to follow that motion exactly.

First-frame conditioning is worth more than any architecture change you can make. The reason is a decomposition. Text-to-video asks the model to invent what things look like and how they move at the same time, from a prompt that under-specifies both. Given the first frame, appearance is settled, the identity anchor is exact, and every unit of model capacity goes to motion instead. It also converts the hardest evaluation problem — “is this the same person throughout the clip” — into one that has a reference to compare against.

That is why the mature product shape is image-to-video with a text-to-image front end. Generate the first frame with the cheap, fast, well-controlled image model of chapter 09, let the user approve it, then animate. The user gets a $0.002 iteration loop on appearance instead of a $0.178 one.

Chaining, and why drift is linear

Five seconds is not a film, so the obvious question is how to get to sixty. The obvious method degrades predictably; a different method bounds the damage.

Chaining means generating five seconds, then generating the next five seconds conditioned on the last frame of the first, and so on. It accumulates error, and the measurements below show how fast.

Two quantities are tracked at each segment boundary.

• Identity cosine is the cosine similarity between face embeddings — two vectors describing a face, compared by the cosine of the angle between them. 1.00 is identical; lower is more different.
• Colour shift dE is the standard perceptual colour-difference unit, in which about 2 is the threshold a person can notice at all.

Read down the rows and watch both columns degrade together.

segment 1 -> 2   identity cos vs original 0.94, color shift dE 2.1
segment 2 -> 3                              0.88,                4.4
segment 3 -> 4                              0.81,                6.9
segment 4 -> 5                              0.74,                9.8

Each segment’s last frame is a generated frame, which is slightly off-distribution — it looks a little unlike anything in the training data, in ways too small to see — and it is the anchor for the next segment. Errors compound roughly linearly in segment count, which the four rows above show directly: identity falls by about 0.06 to 0.07 per boundary rather than collapsing all at once.

Keyframe-first-then-interpolate bounds the drift instead. Two definitions first. A keyframe is a frame at a fixed time that you commit to in advance. Frame interpolation means generating the frames between two known frames rather than after one: the model is told what the interval starts with and what it ends with, and has to fill in a plausible path between them.

The procedure is two passes.

1. Generate keyframes at 0 s, 5 s, 10 s and 15 s in a single pass. Sampling them jointly is what makes them mutually consistent with each other.
2. Interpolate each interval conditioned on both of its ends.

Every segment is now anchored twice instead of once, so error cannot accumulate past one interval — the far end of each interval is a frame the model was given, not one it drifted into. It costs one extra pass over the keyframes and is the correct architecture for anything over about 10 seconds.

Assumptions this stage rests on. That the product will eventually want clips longer than one model pass, and that a first-frame image is available or cheap to make. If the product is strictly five seconds and text-only, the whole chaining subsection is dead weight and first-frame conditioning loses most of its leverage — which is exactly the question to ask the interviewer before committing to it.

5. Data

Video training data is a fundamentally different problem from image training data. The filtering pipeline below turns 100 million raw videos into 105 million usable clips, and pricing it ends on a line item almost nobody budgets for.

5.1 Video-text pairs barely exist

A generative model needs pairs: a piece of media and a description of it. For video, the description side essentially does not exist — and the consequence is not “captions are noisy” but “you have to manufacture all of them.”

Images come with alt-text — the short written description that HTML lets a page author attach to an image so that screen readers can announce it. It is an accessibility standard, it sits in the page source, and there are billions of instances. The quality is poor but it is present, and the image chapter found that a carefully filtered 10% slice of it is worth keeping.

Video has no equivalent. What exists instead is a set of near-misses, each of which describes something other than the pictures:

Synthetic recaptioning — running a model over the media and having it write the description you wish the media had come with — is the fix. In images it was a 12%-of-budget improvement lever. In video it is not a lever at all; it is the only source of captions that exist. There is no “keep 10% of the originals” option, because there are no originals.

And the captions have to carry more. An image caption describes a configuration: what is in the picture and where. A video caption must describe a change: what moves, in which direction, and how the camera behaves.

The captioner is a vision-language model (VLM) — a model that takes images and text together and produces text. A VLM shown 8 frames sampled from a clip is not good at describing motion, because eight stills are a poor description of five seconds.

That failure is not neutral. Captions describing only static content actively teach the model that the prompt does not constrain motion — the same “the training loss rewards ignoring an uninformative caption” argument from Ml objective, with a sharper edge here, because there is no clean caption anywhere in the corpus to dilute it.

5.2 The pipeline, with yields

Walk a raw corpus through every filter and count what survives, because the survival rate is the answer to the most common wrong instinct in this problem — “just scrape more video.”

Three names in the block need defining before it makes sense.

• Shot-boundary detection finds the cuts — the points where the camera changed — and splits a four-minute upload into the individual continuous takes it is made of. It runs first because a training clip that contains a cut teaches the model that scenes teleport.
• Near-duplicate dedup removes clips that are visually almost the same as one already kept. On the open web that mostly means the same stock footage appearing in thousands of uploads.
• UI density, in the third filter, refers to on-screen user-interface elements: menus, cursors, game overlays. A generative model should not learn to draw those into a scene.

source                       100M videos, mean 4 min  =  2.4e10 seconds
shot-boundary detection      mean shot 4.5 s          ->  5.33e9 shots
keep shots of 5-10 s                          22%     ->  1.17e9 clips

filters (multiplicative)
  motion band: reject static screen-recordings AND
    excessive shake                            46%
  resolution >= 720p and aesthetic threshold   35%
  low burned-in text / subtitle / UI density   72%
  near-duplicate and stock-footage dedup       80%
  safety removal                               97%
                        combined  0.0900       ->  105M usable clips

Three arithmetic steps in that block, in order. 100M videos at 240 seconds each is 2.4e10 seconds of footage. Divided by a 4.5-second mean shot, that is 5.33e9 shots. Keeping only the 22% that run 5 to 10 seconds leaves 1.17e9 candidate clips.

Then the five filter rates multiply, because a clip has to pass all of them: 0.46 · 0.35 · 0.72 · 0.80 · 0.97 = 0.0900. And 1.17e9 · 0.0900 = 105M.

flowchart TD
    A["100M source videos<br/>2.4e10 seconds"] --> B["Shot-boundary split<br/>5.33e9 shots"]
    B --> C["Keep 5-10 s shots (22%)<br/>1.17e9 candidate clips"]
    C --> D["Five filters combined (9%)<br/>motion, resolution, text density,<br/>dedup, safety"]
    D --> E["105M usable clips"]

Roughly 2% of shots survive, and one source video yields about one usable clip. That number is the answer to “just scrape more video” — the corpus is not gated on collection, it is gated on the fact that most footage is static, shaky, subtitled, duplicated, or under-resolution, and none of those improve by scraping harder.

5.3 What the pipeline costs, and the surprise

Data preparation for video has a cost structure that does not resemble the image case, and pricing both halves makes the surprise a number rather than an anecdote. NVDEC below is the dedicated video-decoding circuit on an NVIDIA GPU, which turns compressed video files back into frames far faster than the general-purpose cores could.

decoding the corpus
  2.4e10 seconds of video at ~200x realtime on GPU NVDEC
  2.4e10 / 200                        =  1.2e8 GPU-s  =  33,300 GPU-h

captioning 105M clips with a 7B VLM
  8 frames · 256 image tokens + prompt =  2,300 in, 150 out
  (2,300 + 150) · 2 · 7e9              =  34.3 TFLOP per clip
  · 3 (decode memory-bound) / 300e12   =   0.34 GPU-s per clip
  105e6 · 0.34                         =  10,000 GPU-h

                                          ------
                                          43,300 GPU-h  =  ~$108,000

The captioning line is worth reading slowly, because it is the standard way to price any transformer inference. Three steps.

1. Count the tokens. Eight frames at 256 image tokens each is 2,048, plus the prompt, so about 2,300 tokens in and 150 out.
2. Price a pass. Two operations per parameter per token, the same 2 · params · tokens rule as The flipbook baseline: (2,300 + 150) · 2 · 7e9 = 34.3 TFLOP.
3. Apply the memory-bound penalty. The · 3 accounts for the caption-generation half — emitting those 150 output tokens one at a time — being memory-bound: the chip spends most of its time waiting for weights to arrive from memory rather than doing arithmetic, so you get roughly a third of the advertised throughput. (This is a different sense of “decode” from decoding the video corpus above. Same word, unrelated hardware path.)

Both halves are then converted to dollars at the standing $2.50 per GPU-hour: 43,300 · 2.50 = about $108,000.

Decoding the corpus costs three times what captioning it costs. That is the line item nobody budgets, and it is unique to video: in an image pipeline, decode is free and the captioning model dominates. Against a training run of roughly 370,000 GPU-hours (about $925,000 at $2.50 per GPU-hour), data preparation is about 12% of the budget — the same shape as the image chapter’s recaptioning ratio, arrived at by a completely different route.

Assumptions this stage rests on. A 100M-video source corpus averaging four minutes, a mean shot length of 4.5 seconds, and the five filter rates above. The load-bearing one is the 9% combined yield: it is what makes the usable video corpus roughly a hundred times smaller than the usable image corpus, which is what makes joint image-and-video training non-negotiable and therefore what forces the causal VAE of What temporal compression costs you.

6. Metrics

With the data pipeline in place, the next question is how to tell whether the trained model is any good — and the uncomfortable answer is that every automated video-quality number is either blind to the failures that matter or maximized by a model that does nothing. The one instrument that works has a price, derived below.

6.1 Why every automated metric is weak here

Three automated metrics are standard, and each one has something specific it cannot see. The last of the three has a defect severe enough to build the whole argument around.

FVD is Fréchet Video Distance, the video version of FID, Fréchet Inception Distance.

FID is the standard image-generation score, and it works like this: run a large batch of real images and a large batch of generated images through a fixed pretrained network, summarize each batch by the mean and covariance of the resulting feature vectors, and report the distance between those two summaries. Lower means the generated batch looks statistically more like the real batch.

FVD does exactly that, using I3D as the fixed feature extractor. I3D is the Inflated 3D ConvNet, a network trained to classify what action is happening in a video clip. Hold on to that phrase — trained to classify actions — because it is the source of the first problem below.

FVD inherits all of FID’s problems (Offline two axes never one number) and adds three more:

1. The feature extractor was trained to classify actions. A clip where the subject’s shirt changes colour at frame 60 is still, to I3D, “a person walking.” Injecting a deliberate identity swap mid-clip moves FVD by about 3% while human preference collapses.
2. It needs fixed-length, fixed-rate windows. Your clip has to be resampled to whatever length and frame rate I3D expects, so you are measuring a resampled proxy of your output rather than your output.
3. Its variance is brutal. Variance here just means how much the number bounces around when nothing about the model changed: the same model scored on 512 samples versus 2,048 samples can differ by about 15%, which is larger than most improvements you will ever ship.

Per-frame CLIPScore is the second metric.

CLIP (Contrastive Language-Image Pretraining) is a pair of encoders, one for images and one for text, trained so that a picture and its caption land close together in the same vector space. CLIPScore is the cosine similarity between an image’s vector and the prompt’s vector.

Applied per frame, it tells you the frames match the prompt. It says nothing whatsoever about whether they match each other, because it never compares two frames.

The third is temporal consistency, and it has a degenerate maximum — meaning its best possible score is achieved by an output that is worthless:

temporal consistency  =  mean over t of  cos( CLIP_img(f_t), CLIP_img(f_{t+1}) )

a completely static video:   every consecutive pair identical
                             ->  consistency = 1.000, the maximum

The metric that is supposed to measure temporal quality is maximized by the model failing to move.

The obvious counterweight has the mirror problem. Warp error takes frame t, drags its pixels along the estimated optical flow into where frame t+1 says they went, and measures how far the result is from the actual frame t+1. On a static video the optical flow is zero, so nothing gets dragged anywhere, so the warped frame is frame t+1 and the error is 0.000 — the best possible score again.

Either metric alone is gamed by a model that produces a still image. And a model that under-moves is a real and common failure, not a hypothetical one, which is why this matters.

The ratio people reach for next — warp error per unit of motion — does not rescue this, and it is worth being precise about why. Its denominator is motion, the number the line above calls gameable, and on a static clip the ratio is 0/0. So the scorecard has to gate on motion before it divides by it, and report undefined rather than a number, because inf and 0.0 are both lies about a clip that simply did not move.

The code below is that scorecard. It returns three raw numbers, a flag, and the ratio; the assertions underneath run it on a deliberately static clip and then on a moving one. On the static clip consistency comes out at a perfect 1.000 — the degenerate maximum — while under_motion is True and warp_error_per_motion is None rather than a number.

MOTION_FLOOR = 0.25          # px/frame. Below this the clip is not moving.


def _mean_abs(xs) -> float:
    xs = list(xs)
    return sum(abs(x) for x in xs) / len(xs)


def temporal_scorecard(frames, embed, flow):
    """Consistency is only meaningful jointly with motion. Never report one.

    frames  list of decoded frames, each a flat sequence of pixel values
    embed   f -> unit-norm image embedding, as a flat sequence
    flow    (f_a, f_b) -> (per-pixel flow MAGNITUDE, f_a warped into f_b)

    Magnitude, not per-component flow: mean(abs(u)) over components
    understates diagonal motion by up to sqrt(2), and motion is the
    denominator below.

    A static clip scores consistency 1.000 and motion 0.000 -- which is the
    point: the pair identifies it instantly, and either number alone hides it.
    """
    if len(frames) < 2:
        raise ValueError("a scorecard over fewer than two frames measures nothing")

    embs = [embed(f) for f in frames]
    n = len(frames) - 1
    consistency = sum(sum(x * y for x, y in zip(a, b))
                      for a, b in zip(embs, embs[1:])) / n

    motion = warp_err = 0.0
    for a, b in zip(frames, frames[1:]):
        magnitude, warped = flow(a, b)
        motion += _mean_abs(magnitude)
        warp_err += _mean_abs(x - y for x, y in zip(b, warped))
    motion, warp_err = motion / n, warp_err / n

    moving = motion >= MOTION_FLOOR
    return {
        "consistency": consistency,        # gameable alone: static -> 1.000
        "motion": motion,                  # gameable alone: chaos -> huge
        "warp_error": warp_err,            # gameable alone: static -> 0.000
        "under_motion": not moving,        # the finding a ratio would hide
        # Residual per unit of real motion -- undefined without real motion.
        # inf would be a lie: this clip has no warp residual, it has no
        # motion, and those are different findings about different models.
        "warp_error_per_motion": warp_err / motion if moving else None,
    }


# --- the static clip: the exact failure this section exists to catch --------

def unit(frame):
    norm = sum(x * x for x in frame) ** 0.5
    return [x / norm for x in frame]


STATIC = [[0.20, 0.51, 0.93, 0.10]] * 6
card = temporal_scorecard(STATIC, unit, lambda a, b: ([0.0] * len(a), list(a)))

assert abs(card["consistency"] - 1.0) < 1e-12      # the degenerate maximum
assert card["motion"] == 0.0 and card["warp_error"] == 0.0
assert card["under_motion"] is True
assert card["warp_error_per_motion"] is None       # 0/0: not inf, not zero

MOVING = [[0.2, 0.5], [0.5, 0.8], [0.8, 1.1]]
card = temporal_scorecard(
    MOVING, unit, lambda a, b: ([1.0] * len(a), [x + 0.2 for x in a]))

assert card["under_motion"] is False
assert round(card["warp_error_per_motion"], 3) == 0.1

try:                                               # and one frame is not a clip
    temporal_scorecard(STATIC[:1], unit, lambda a, b: ([0.0], list(a)))
    raise AssertionError("scored a one-frame clip")
except ValueError:
    pass

Report the pair. The ratio is a summary of the pair, and it only exists above the motion floor.

That is exactly the reason Offline two axes never one number refused to report a single quality number: two axes with different best-case directions cannot be collapsed into one without silently deciding which one you gave up.

There is a practical payoff to returning None. A pipeline that averages warp_error_per_motion over an evaluation set now has to decide what to do with them, and that is the correct place to be forced to think. The honest aggregate reads: “12% of clips were under-motion, and the remaining 88% averaged 0.31.”

Assumptions this stage rests on. A motion floor of 0.25 pixels per frame, which is a product judgement about how little movement still counts as a video, not a fact about optical flow. Raise it and more clips are reported as under-motion; lower it and the ratio starts dividing by noise. It is the one number in the scorecard worth arguing about with whoever owns the product.

6.2 Human evaluation, and why it dominates here

Every failure that actually loses users — flicker, morphing identity, an object that changes on the far side of an occlusion, a physically reversed trajectory — is invisible to every metric above. That makes human evaluation not a supplement here but the instrument — and the cost objection people raise against it does not survive being priced.

Three terms set the size of the study, and together they determine how many comparisons you need.

• Alpha is the false-positive rate you are willing to accept. 0.05 means a 5% chance of declaring a winner when the two models are actually identical.
• Power is the probability of detecting a real difference when one exists. 0.80 means you would catch it four times in five.
• “Detect 55/45” means being able to distinguish a model that wins 55% of head-to-head comparisons from a coin flip. The smaller the edge you want to detect, the more comparisons it takes.

Follow the block from the top: it turns those three choices into a rater count, then into hours, then into dollars.

per pairwise judgment    watch 2 clips (5 s each) + decide  ~=  25 s
                         (an image pair is ~4 s -- video is ~6x)

power calculation, same as ch 09 §7.1:
   detect 55/45 at alpha 0.05, power 0.80       ->  776 comparisons
   · 3 raters for agreement                      =  2,328 judgments
   · 25 s                                        =  16.2 rater-hours
   at $25/hour                                   =  $405 per model pair

That power calculation is the same one used for image models in Offline two axes never one number; only the seconds-per-judgment changed, because a rater has to watch ten seconds of video instead of glancing at two pictures.

$405 against a serving fleet that costs $342,000 a day (Serving derives that fleet number). Human evaluation is one tenth of one percent of a day of serving, and it is the only instrument that sees the failures. There is no cost argument for skipping it, and saying so with the ratio attached is a far stronger answer than saying “human eval is important.”

Structure the rating on separate axes, because they have different best directions and a rater cannot trade them off for you: prompt adherence, motion quality, temporal consistency, and aesthetic. A single “which is better” question collapses all four into one number and you can no longer tell which one regressed.

Assumptions this stage rests on. 25 seconds per pairwise judgment, three raters per comparison for agreement, and $25 per rater-hour. All three are estimates you should name as estimates; the conclusion survives any of them being off by a factor of two, because the gap to $342,000 a day is three orders of magnitude.

6.3 Online

Offline numbers gate a release; online numbers tell you whether the release was any good. These are the ones a video product watches, and the bottom three exist only because generation is asynchronous.

The regenerate-rate row carries a diagnostic worth stating in full, because it is free and most teams do not log it. If a user regenerates after editing the prompt, they are telling you the model did not do what the words said, which is a prompt-adherence problem. If they regenerate without touching the prompt, they are telling you this particular sample was bad, which is a sampling or quality problem. Same button, two completely different bug reports, separated by one boolean in the log.

7. Serving

The per-clip compute cost now becomes a product: a job system rather than a request-response service, with a cheap preview tier, a priority queue, and a fleet you commit to rather than autoscale. Every one of those is derived, not assumed.

At 257 GPU-seconds per clip, this is not a synchronous product — one where the user waits with the connection open for an answer — and pretending otherwise is the most common design error here. The architecture is a job system: the request returns a ticket, the work happens elsewhere, and the user is notified.

The diagram below is the whole pipeline, from a typed prompt to a file on a CDN. Two things to notice as you read it: everything cheap happens before the queue, and the only expensive box is the render pool.

flowchart TD
    R([Prompt + optional<br/>first frame]) --> MOD{Text + image<br/>moderation}
    MOD -->|block| REJ([Refuse])
    MOD -->|pass| CR{Credit check<br/>+ tier}
    CR -->|none| PAY([Upsell])
    CR -->|ok| DRAFT["Draft tier<br/>512 · 288 · 20 steps<br/>24 GPU-s · $0.016"]
    DRAFT --> PRE([Preview to user])
    PRE -->|discard| R
    PRE -->|approve| Q[[Priority job queue<br/>free · paid · pro]]
    Q --> REN["Render pool<br/>5B DiT · 50 steps · CFG<br/>8 GPUs sequence-parallel<br/>257 GPU-s"]
    REN --> DEC["Causal 3D VAE decode<br/>30 latent -> 120 frames"]
    DEC --> SCAN{"Safety scan<br/>EVERY frame<br/>72 ms total"}
    SCAN -->|flag| HOLD[Hold for review]
    SCAN -->|clear| WM[Per-frame watermark<br/>+ C2PA manifest]
    WM --> TR[Transcode ladder<br/>1080p / 720p / 480p]
    TR --> CDN([CDN + notify])

    style DRAFT fill:#40916c,color:#fff
    style REN fill:#1d3557,color:#fff
    style SCAN fill:#2d6a4f,color:#fff
    style REJ fill:#9d0208,color:#fff

Several boxes are named in shorthand, so here is each one in a line.

• Text + image moderation. Both inputs are checked against policy before any compute is spent. The image half exists because a user can smuggle in through a picture what the prompt filter would have caught.
• Credit check + tier. Looks up whether this user’s plan and remaining credits allow a render, and which service class they belong to. A user with no credits is shown an upsell, not a queue position.
• Draft tier. A cheap low-resolution preview for the user to approve or discard: about 24 GPU-seconds and $0.016, against the render’s 257 GPU-seconds. Its derivation is below.
• Priority job queue. One queue whose ordering respects three service classes — free, paid, pro — so a paying user does not wait behind a thousand free ones.
• Render pool. The fleet running the real 5B model at 50 steps with guidance, split sequence-parallel across 8 GPUs: the token sequence itself is divided among the chips, which then exchange the pieces they each need.
• Causal 3D VAE decode. Expands 30 latent frames back into 120 pixel frames.
• Safety scan. Classifies every frame. Anything flagged goes to a human for review rather than being silently deleted or published.
• Watermark + C2PA manifest. C2PA is the Coalition for Content Provenance and Authenticity, an industry standard that attaches signed metadata recording that this file was machine-generated and by what.
• Transcode ladder. Re-encodes the clip at 1080p, 720p and 480p so players on different connections each get a version they can stream.
• CDN + notify. The two happen together: the files reach the edge servers that hold copies close to viewers, and the user is told the job is done.

Dollars per second of video

Everything the product is allowed to be follows from one number, so derive it end to end rather than quoting the render cost alone. Utilization below means the fraction of the time your reserved GPUs are actually busy; nobody runs at 100%, so the honest delivered cost divides by a realistic figure.

render        257 GPU-s                                  $0.1784
3D VAE decode ~15% of render                             $0.0268
safety scan   120 frames · 0.6 ms                        $0.0001
watermark + transcode  (CPU)                             $0.0040
egress        5 s at 8 Mbps = 5 MB, $0.09/GB             $0.0005
                                                         -------
per clip at 100% utilization                             $0.210
at 60% fleet utilization                                 $0.349

per second of generated video                            $0.0699
per second at 100% utilization                           $0.0419

Two things to notice in that block. The render is $0.1784 of a $0.210 clip, so 85% of the cost is the denoiser and everything else is rounding. And dividing by 60% utilization is not a modelling choice — it is the difference between what you compute and what you pay for, and it is what moves the delivered clip from $0.210 to $0.349.

Two comparisons that put it in context. Both are priced on the same basis — one 1024 · 576 image by chapter 09’s method at this chapter’s per-frame arithmetic, one clip, both all-in:

one 1024 · 576 image, 56 passes · 13.72 TFLOP  =    768 TFLOP
  at 300 TFLOP/s, all-in                       =  $0.00185 at 100% util
one 5-second clip                              =  $0.210   at 100% util
                                                  -> a clip costs 113 images

$9.99/month buys, at 60% utilization:
   images   9.99 / 0.00308  =  3,200 images
   clips    9.99 / 0.349    =     29 clips

One line in that block is not obvious: the 0.00308 is the image’s $0.00185 divided by the same 60% utilization applied to the clip. Both sides of the comparison are delivered costs, not compute costs, which is the only way the ratio means anything.

The same subscription price buys 3,200 images or 29 clips. That ratio, not any modelling decision, is why video products meter credits and image products do not. It is the single most useful number to have ready in this interview.

Assumptions this stage rests on. 60% fleet utilization, $0.09 per gigabyte of egress, and a 5 MB delivered clip. The utilization figure is the load-bearing one, because it is what turns $0.210 into $0.349 and therefore what sets the 29-clips break-even. Ask the interviewer what utilization the platform actually runs at before defending the pricing model.

Fleet sizing

Unit cost becomes capacity by multiplying by demand, and capacity at this scale stops being an engineering decision and becomes a purchasing one.

1M clips/day  =  11.6 clips/s
each          =  257 render + ~38 decode  =  295 GPU-s

at 100% utilization   11.6 · 295           =  3,420 GPUs
at 60% utilization                         =  5,700 GPUs
at $60/GPU-day                             =  $342,000/day  =  $125M/year

Two of those lines are conversions rather than results. A million clips a day is 1e6 / 86,400 = 11.6 clips per second, since a day is 86,400 seconds. And $60 per GPU-day is just the standing $2.50 per GPU-hour times 24.

Nearly six thousand GPUs is a capacity commitment, not an autoscaling group. Hardware at that quantity is bought or reserved months ahead, not summoned when traffic arrives. Everything about the product follows from that one fact: you cannot burst, so you must queue; you cannot queue indefinitely without losing people, so you must tier; and you cannot tier without something cheap to put in the fast tier, so the draft tier is architectural rather than a nicety.

Why the draft tier is not optional

The draft is the same model at lower resolution and fewer steps, priced two ways below — against the render, and against the user behaviour it exists to absorb.

Where the 576 in the block comes from: at 512 · 288 the VAE’s eightfold spatial compression gives a 64 · 36 latent, and 2 · 2 patches make a 32 · 18 grid, so 576 tokens per latent frame. That is a quarter of the render’s 2,304, on the same 30 latent frames.

draft: 512 · 288, 20 steps, CFG
  tokens        30 · 576                    =  17,280   (4x fewer)
  parameter     2 · 5e9 · 17,280            =  173 TFLOP
  spatial       30 · 4 · 576^2 · 3,072 · 40 =    4.9 TFLOP
  temporal      576 · 4 · 30^2 · 3,072 · 40 =    0.25 TFLOP
  per pass                                  =  178 TFLOP
  40 passes                                 =  7.1 PFLOP  =  23.7 GPU-s  =  $0.0165

  770 / 178 = 4.3x per pass · 100 / 40 = 2.5x on passes
                                            -> 10.8x cheaper than the render

That 10.8x is the honest like-for-like number: render compute against draft compute. Comparing the draft’s render cost to the clip’s all-in cost — which also carries VAE decode, safety, transcode and egress — inflates it to 12.7x and is comparing two different things.

The behaviour that justifies the tier is measurable and worth measuring: users discard about four drafts before approving one, so a completed clip costs five drafts plus one render rather than one of each.

draft-then-render   5 · $0.0165 + $0.178  =  $0.261
render every time   5 · $0.178            =  $0.892
                                             -> 3.4x cheaper

And the latency story is better than the cost story: a draft is 23.7 GPU-seconds, which is about 4.6 seconds of wall-clock time when split across 8 GPUs, so the iteration loop feels interactive even though the product as a whole is asynchronous.

Assumptions this stage rests on. Four discarded drafts per keeper, and a draft at 512 · 288 with 20 sampling steps that is good enough to judge a clip by. If users discard one draft rather than four, the comparison becomes two drafts plus one render ($0.211) against two full renders ($0.357), which is 1.7x rather than 3.4x — at which point the tier is a latency feature rather than a cost feature, still worth having for a different reason.

Latency

Wall-clock time is what the user experiences, and it is not the same shape as cost: the cheapest stage of the pipeline is a third of the wait.

render     257 GPU-s, sequence-parallel over 8 GPUs at 65% scaling
           257 / (8 · 0.65)                     =  49 s
decode     3D VAE, poorly parallel              =  ~28 s
transcode + upload                              =  ~8 s
                                                   ----
                                                   ~85 s, plus queue

The 65% scaling figure on the render line is the efficiency loss from splitting one job across 8 GPUs: they have to exchange data, so 8 chips do about 5.2 chips’ worth of useful work, and 257 GPU-seconds becomes 49 seconds of waiting rather than 32.

The decode is a third of the wall clock and gets no attention in most designs. It is also the cheapest thing to fix.

Decoding is memory-bandwidth-bound: the chip is limited by how fast it can move data in and out of memory, not by arithmetic. And it splits cleanly across groups of frames, because each group is decoded independently of the others.

So shard it — hand each of 4 GPUs a different quarter of the frames — and you get close to a 4x speedup, taking about 28 seconds down to about 7 and the whole pipeline from 85 seconds to 64.

One safety consequence follows directly from these numbers, and it is worth volunteering: scan every frame. Putting 120 frames through a CLIP-class classifier at 0.6 ms each is 72 ms against an 85-second pipeline, which is 0.08% of it. Sampling every 8th frame saves nothing measurable and can miss a three-frame violation — and three frames is more than enough to be screenshotted and shared.

Assumptions this stage rests on. Eight-way sequence-parallel rendering at 65% scaling efficiency, and a decoder that is not yet sharded. Both are implementation states rather than laws: shard the decoder and the wall clock drops from about 85 seconds to about 64, which is the single largest latency win available in this design and costs no extra GPU-seconds, only engineering.

8. Failure modes

Four failures lose users, and each can be traced from a symptom you can see to the specific component that produced it; the rest get a detector and a guard apiece. None of them is fixed by “train longer” — every one has a named mechanism and a matching intervention.

Flicker

Flicker is a sudden one-frame change in something that should have stayed constant.

The trace below records the average colour of a jacket across nine consecutive frames as a hexadecimal colour code — the standard #RRGGBB notation in which each pair of characters is the red, green and blue intensity from 00 to FF. To compare two of them by brightness, use luma, the standard weighted brightness summary 0.2126·R + 0.7152·G + 0.0722·B.

Read down the frame numbers and watch frame 40, then watch frame 41 undo it.

PROMPT   "a woman in a navy blazer speaking to camera, static shot"

blazer mean colour, frames 36-44
  36 #24365E   37 #24375F   38 #233560   39 #253863
  40 #2E4576   <- luma 53 -> 68, a 27% jump in one frame
  41 #24365D   42 #24365E   43 #233460   44 #24365F

frame 40 is the first frame of a new 4-frame VAE group

The last line of the trace is the diagnosis. Frame 40 is 4 · 10, so it opens the eleventh decoder group.

Temporal attention operates on 30 latent frames; the decoder generates 120 output frames in groups of 4. High-frequency detail that the latent does not constrain — fabric texture, fine specular highlights, hair — is synthesized by the decoder per group. Group boundaries are therefore exactly where it can discontinuously change its mind, and the trace shows it doing so.

Three fixes, in increasing order of effort:

1. Overlap the decoder’s temporal windows and blend where they meet, so a group boundary is never a hard seam. Pure serving-side change.
2. Add a temporal-consistency term to the VAE’s own training loss, so the decoder is penalized for changing its mind. Requires retraining the VAE.
3. Lower the guidance scale, because CFG amplifies exactly this high-frequency component along with everything else. Free, but it costs prompt adherence.

Object permanence

The second failure is the one users describe as “it turned into a different car”, and it is the failure that most cleanly separates architectures. The trace below is one clip in three phases — before the occlusion, during it, and after it. The third phase is the bug.

PROMPT   "a red car drives past a lamppost, camera fixed"

frames 0-40    red sedan moving left to right, four visible wheels
frames 41-55   fully occluded by the lamppost
frames 56-119  a red HATCHBACK, different wheels, different roofline

Temporal attention gives frame 56 access to frame 40. That is not the same as memory. Attention is a soft read over a distributed representation, not a slot that stores “this specific car”, and nothing in the denoising objective rewards maintaining object identity across an occlusion. It rewards predicting plausible noise, and a hatchback is plausible.

This is the failure that inflation-based architectures cannot fix, because fixing it needs the spatial layers themselves to have learned that frames belong to sequences. Longer joint training and larger temporal receptive fields move it. Bolting a temporal stack onto a frozen image model does not.

Physics and the arrow of time

The third failure is the one that looks like a bug in reality itself, and its cause is in the data rather than in the architecture.

PROMPT   "a glass falls off a table and shatters on the floor"

3 of 8 samples: the glass falls, contacts, shatters -- and over the next
8 frames the shards travel BACK UP and reassemble

The denoising objective is symmetric in time. The training loss asks “is the noise prediction right for this frame”, and that question has the same answer whether the clip runs forwards or backwards. Nothing in it penalizes a trajectory that is locally plausible frame-to-frame and globally reversed.

The corpus makes it worse. Reversed and looped clips are a popular editing effect, so real footage of shards flying upward exists in quantity, and it actively teaches the model that reversal is a normal thing for video to do.

Both fixes are data-side and cheap. Detect and drop reversed and looped clips in the The pipeline with yields pipeline. And add an explicit time-direction conditioning signal, so the model has a handle it can be right or wrong about instead of no representation of direction at all.

Identity morphing

The fourth failure is a face slowly becoming a different face, and unlike the others it has a signature you can read straight off the measurement.

The numbers below are ArcFace cosine similarity: ArcFace is a face-recognition model that maps a face to a vector chosen so that two pictures of the same person point in nearly the same direction, and the number reported is the cosine of the angle between frame 0’s vector and frame t’s. Chapter 10 established roughly 0.65 as the threshold below which two images stop reading as the same person.

PROMPT   "a woman in a red coat walks toward camera, 5 seconds"

ArcFace cosine, frame 0 vs frame t
  t=12   0.91
  t=36   0.78
  t=72   0.61
  t=119  0.44      <- below the same-person anchor from ch 10 (~0.65)

Monotone decay is the signature of re-generation rather than tracking. A tracking failure would jump; this slides, a little further every frame.

The mechanism is that there is no identity state anywhere in the system. The face has to survive as a pattern in a distributed representation across 30 latent frames, while CFG at every step pulls each latent frame toward the text prior — and the prompt says “a woman,” not “this woman.”

First-frame conditioning is the strongest mitigation, because it converts an unanchored trajectory into one with a fixed endpoint. Reference-image conditioning and a per-frame identity loss during fine-tuning help further.

The rest

Nine further failures, each with the signal that detects it and the intervention that guards against it. OCR in the third row is optical character recognition — a model that reads text out of an image, used here to check whether a sign in the scene spells the same word in every frame.

Assumptions this stage rests on. That you have a decoder whose training loss you can modify, an ingest pipeline you can add filters to, and the ability to add conditioning signals and retrain. Every guard in this section is one of those three; none of them is a serving-time patch, which is why failure modes have to be designed for before the training run rather than after it.

9. Alternatives considered and rejected

Nine architectures, four operational shortcuts, and two measurement shortcuts that a reasonable person would propose, each with the honest reason it appeals and the specific number that rules it out. GAN in the table is a generative adversarial network — a generator and a critic trained against each other, which produces an image in one forward pass rather than dozens of denoising steps; mode collapse is its characteristic failure, where the generator finds a small set of outputs that fool the critic and stops producing anything else.

10. Interviewer pushback

Fourteen questions an interviewer actually asks on this problem, each with what it is testing and the answer as you would say it out loud. Two of them are traps — a good-looking number you are invited to accept, and a cost objection you are invited to concede.

“Why can’t you just generate 120 frames with the image model?” Testing: whether you understand that consistency is a distributional property. Because each frame would be an independent sample from p(frame | prompt), and nothing couples them. A shared seed does not help — diffusion is chaotic in the initial condition, so two latents differing by 1% end up visibly different images. It costs exactly 120x an image, $0.21 a clip, and produces a flipbook. Temporal consistency is a property of the joint distribution, so you either model the joint distribution or you do not have it.

“Then model them jointly. What does that cost?” Testing: the blow-up, derived live. With full 3D attention over 120 frames at 2,304 tokens each, that is 276,480 tokens. The parameter term goes up 120x, linearly, to 1,438 TFLOP a pass. The attention term goes up 120 squared — 14,400x — to 25,050 TFLOP a pass. Attention is 13% of an image’s cost and 95% of a clip’s. Fifty-six passes is 1,483 PFLOP, 82 minutes of an H100, $3.43 a clip. That is the number that forces the architecture.

“So how do you get it down?” Testing: whether you know which lever hits which term. Two moves. Factorize the attention into spatial-within-frame and temporal-across-frames: 220 TFLOP instead of 25,050, a factor of 114, and it makes a temporally-joint model cost the same as the flipbook. Then compress the latent 4x in time with a causal 3D VAE: at fixed model size that is 4x on the parameter term, 4x on spatial attention, and 16x on temporal attention because that one is quadratic. You land at 770 TFLOP a pass with a bigger 5B model, 257 GPU-seconds, $0.178 a clip. I would be careful about the headline ratio: the attention term alone falls about 474x, but the clip falls 19.3x, because the parameter term is untouched by factorization and the 5B model and 100 passes eat into the rest. Multiplying the 114x by the 4x gives 456 and means nothing — the two ratios have different denominators.

“What does temporal compression cost you?” Testing: whether you know your own tradeoffs. Bandwidth in time. Four-times compression on 24 fps leaves 6 Hz of independent temporal information, so a Nyquist limit of 3 Hz. Motion faster than that aliases — the wagon-wheel effect on spinning objects, strobing on fast pans, smeared limbs on runners. It is a sampling-rate consequence, not a training deficiency, and no amount of data fixes it. The fix is a 2x-compression high-motion tier at roughly 2-4x the cost.

“What is your dollars per second of video?” Testing: the number the product is built around. $0.042 per second at full utilization, $0.070 at a realistic 60%. A five-second clip is $0.210 raw, $0.35 delivered — about 113 images’ worth of compute at the same 1024 · 576. Which means a $9.99 subscription buys 3,200 images or 29 clips. That ratio is why video products meter credits and image products do not, and it decides the pricing model before anyone has picked a model architecture.

“Size the fleet for a million clips a day.” Testing: whether you can go from unit cost to capacity. 11.6 clips a second, 295 GPU-seconds each counting decode, so 3,420 GPUs saturated, about 5,700 at 60% utilization. At $60 per GPU-day that is $342,000 a day, $125M a year. That is a capacity commitment rather than an autoscaling group, which is exactly why the product has to queue rather than burst, has to tier rather than queue indefinitely, and needs a cheap draft tier to make the tiering work.

“Justify the draft tier.” Testing: whether tiering is a habit or a derivation. At 512 · 288 and 20 steps, the token count drops 4x and the pass count drops 2.5x, so a draft is 178 TFLOP a pass and 23.7 GPU-seconds — 10.8x cheaper, render compute against render compute. Users discard about four drafts per keeper, so draft-then-render is five drafts plus one render, $0.26, against five full renders at $0.89. That is 3.4x. And the draft is under five seconds of wall clock on eight GPUs, so the iteration loop feels interactive even though the product is asynchronous.

“Your temporal consistency score is 0.98. Good?” Testing: whether you take a good number at face value. It is a trap. Not by itself, because that metric is maximized by a static video — a still image repeated 120 times scores exactly 1.000. Warp error has the mirror problem: no motion means no flow means no residual. So I report consistency jointly with motion magnitude. The tempting single number is warp error per unit of real motion, and it works above a motion floor — but its denominator is motion, so on the static clip it is 0/0, and a scorecard that returns inf or zero there has hidden the exact failure it was built for. Below the floor the honest output is “under-motion”, not a number. A 0.98 alongside near-zero flow means the model failed to move, and under-motion is a common failure, not a hypothetical one.

“Why not gate releases on FVD?” Testing: whether you know what the metric’s backbone can see. Because I3D was trained for action classification and is close to blind to the things that ruin a clip. A deliberate identity swap at frame 60 moves FVD by about 3% while human preference collapses — the clip is still “a person walking.” It also needs fixed-length resampled windows, so you are scoring a proxy, and its variance across sample counts is around 15%, larger than most shippable improvements. I would use FVD as a smoke alarm and gate on human preference.

“Human eval on video sounds prohibitively expensive.” Testing: whether you price things or assume them. It is about 25 seconds a judgment against 4 for images, so six times an image study. To detect a 55/45 win at 80% power you need 776 comparisons, times three raters, which is 16 rater-hours, about $405. The serving fleet costs $342,000 a day. Human eval is one tenth of one percent of a day of serving and it is the only instrument that sees flicker, morphing, and permanence failures. There is no cost argument for skipping it.

“A car goes behind a pole and comes out as a different car. Why?” Testing: mechanism, and architecture consequences. There is no object state. Temporal attention lets frame 56 read frame 40, but that is a soft read over a distributed representation, not a slot holding “this specific car,” and nothing in the denoising objective rewards maintaining identity across an occlusion. This is also the failure that inflation-based architectures cannot fix: if the spatial layers are frozen from an image model, they learned the marginal distribution of single frames and have no notion of a sequence, so all the permanence has to ride on the thin temporal stack. Joint training with everything unfrozen is what buys it.

“How would you get to a 60-second video?” Testing: whether you know error accumulates. Not by chaining, or at least not naively. Chaining conditions each segment on the previous segment’s last generated frame, which is slightly off-distribution, so error compounds roughly linearly — measured, identity cosine drops from 0.94 to 0.74 over four chained segments. I would generate keyframes at fixed intervals in one pass so they are mutually consistent, then interpolate each interval conditioned on both ends. Every segment is anchored twice, so drift is bounded by one interval instead of accumulating. It costs one extra pass over the keyframes.

“Where does your caption data come from?” Testing: whether you know video data is a different problem. There is no alt-text for video. Titles describe the upload, descriptions are sponsorships, and ASR describes what is said rather than what is shown. So essentially 100% of usable captions are synthetic — in images, recaptioning was a lever; here it is the only source. And the captions have to describe change, not configuration, which VLMs are weak at from eight sampled frames. The pipeline is shot detection, then motion, resolution, text-density, dedup, and safety filters that keep about 9%, which is roughly one usable clip per source video. The surprise in the budget is that decoding the corpus costs three times what captioning it costs.

“You have a fixed GPU budget. Bigger model or longer clips?” Testing: whether you reason about where quality comes from. Neither first — I would spend it on first-frame conditioning and better captions. Image-to-video is the largest quality lever in the system because it removes appearance from the model’s job entirely and hands the whole capacity budget to motion, and it lets the user iterate on appearance in a $0.002 loop instead of a $0.178 one. After that, longer clips beat a bigger model, because the complaints are temporal — permanence, drift, chaining artifacts — and those improve with temporal context rather than with parameters. The last thing I would buy is resolution, which the upscaler handles at a fraction of the cost.

11. The assumption ledger

Every number in this chapter sits on top of an assumption, and an interview goes badly when one of them is treated as a fact. Here they all are in one place, each sorted into one of two handling rules — state it, meaning announce it unprompted before you build on it, or ask it, meaning you genuinely cannot proceed without the interviewer’s answer — together with what actually breaks if it turns out to be wrong.

The three to lead with are the frame count, the temporal compression ratio, and the utilization figure, because those are the ones that reach furthest: the first decides the compute, the second decides both the compute and the quality ceiling, and the third decides the pricing model. Everything else in this table can be wrong by a factor of two without changing a single architectural conclusion.

Next: the ML system design track.