17 — LLM Inference Performance

Most serving decisions — batch size, scheduler, parallelism layout, which GPU to buy — follow from one ratio: how much arithmetic the chip can do per byte it pulls from its own memory. The kv cache the most important mechanism in this chapter derived that decode is memory-bound, and Prefill vs decode the fact underneath every cost rule priced the consequence at the API level. This chapter works at the hardware level, where the same fact becomes a number you can compute, then covers the serving stack built around it: PagedAttention, continuous batching, chunked prefill, tensor and pipeline parallelism, disaggregated serving, and the profilers that tell you which one you need.

The reasoning is the same one you use when a service is capped by disk reads rather than CPU: the hardware differs, the discipline is identical. Every claim below is derived from arithmetic you can reproduce with a calculator.

By the end you should be able to:

• Place prefill and decode on a roofline plot and derive why batch-1 decode runs at ~0.3% of the GPU’s peak.
• Size a KV cache from four model dimensions, and convert free memory into a max-concurrency number.
• Explain what PagedAttention borrows from OS virtual memory and compute what it buys.
• Say when continuous batching is not enough and chunked prefill is needed, and its ITL/TTFT trade-off.
• Choose tensor vs pipeline parallelism from the communication arithmetic.
• Describe disaggregated serving and KV-aware routing as responses to the two phases having different bottlenecks.
• Read an Nsight Systems timeline and an Nsight Compute report, and explain why nvidia-smi’s “GPU util” is misleading.

Assumptions, fixed once. Unless a line says otherwise, every number below assumes an NVIDIA H100 SXM with 80 GB of HBM at 3.35 TB/s, 989 TFLOPS of dense BF16 compute, 900 GB/s of NVLink, and 64 GB/s of PCIe Gen5 x16 — representative spec-sheet values; real SKUs vary (an H200 has more and faster memory, a PCIe-card H100 less of both), and none of the reasoning changes when they do. BF16 means 2 bytes per number. Two reference model shapes recur: an 8B (8 × 10⁹ parameters: 32 layers, hidden width 4,096, 8 KV heads, head dimension 128) and a 70B (80 layers, hidden width 8,192, 8 KV heads, head dimension 128) — both Llama-3-shaped, chosen so the arithmetic checks against public configs.

1. The memory hierarchy, and why LLM serving is a bandwidth problem

A GPU is not a faster CPU. It is built around one assumption: that the problem is thousands of identical arithmetic operations wide, so it trades single-thread speed for tens of thousands of simple arithmetic units running in lockstep.

Two definitions first, because the chapter uses them throughout. A kernel is one function launched onto the GPU — “multiply these two matrices,” “apply softmax to these rows” — executed by thousands of threads at once; everything a GPU does is a sequence of kernel launches, and a serving engine’s decode step is a stream of them. An SM (streaming multiprocessor) is the unit those threads are scheduled onto: the H100 SXM has 132 of them, and each SM holds the arithmetic units, a slice of fast on-chip memory, and the scheduler for its resident threads. Threads execute in groups of 32 called a warp: one instruction, 32 data lanes.

The design has one more consequence worth knowing before the memory pyramid. A CPU hides memory latency with large caches and speculation; a GPU hides it with more parallelism. Each SM keeps many warps resident at once, and when one warp stalls waiting for a memory read, the scheduler issues instructions from another. This works only while there are enough independent operations in flight, and batch-1 decode is exactly the workload that fails to supply them.

The pyramid

Arithmetic units are useless if the data is not next to them. The memory system is a pyramid, and each level trades capacity for speed:

Three of those rows deserve one sentence each. SRAM is static RAM on the die itself: the only memory fast enough to keep the arithmetic units fed, and four orders of magnitude too small to hold a model. HBM is the stack of DRAM on the GPU package; its 80 GB is the capacity in every “does it fit” question, and its 3.35 TB/s is the bandwidth this chapter centers on. NVLink is NVIDIA’s GPU-to-GPU interconnect inside a server, 3.7× slower than HBM but 14× faster than PCIe, the general-purpose bus to the host. That 14× gap decides where multi-GPU schemes can live (§7).

flowchart TD
    R["Registers<br/>KBs per SM, ~1 ns"] -->|"within the SM"| S["SRAM: shared mem + L2<br/>~50 MB L2, tens of TB/s"]
    S -->|"on-die"| H["HBM<br/>80 GB @ 3.35 TB/s"]
    H -->|"NVLink ~900 GB/s"| G["Other GPUs in the node"]
    H -->|"PCIe ~64 GB/s"| D["Host DRAM / NVMe"]

    style H fill:#2d6a4f,color:#fff
    style G fill:#40916c,color:#fff
    style D fill:#bc6c25,color:#fff

The ratio that runs the chapter

Now put the compute next to the memory. A FLOP is one floating-point operation, one multiply or one add, and FLOP/s (often written FLOPS) is how many of them per second. The H100 does 989 × 10¹² per second in dense BF16. Divide the two headline numbers:

peak compute          = 989e12 FLOP/s
HBM bandwidth         = 3.35e12 bytes/s
ratio                 = 989e12 / 3.35e12  =  295 FLOPs per byte

For every byte the chip fetches from HBM, it has time to do ~295 floating-point operations before the memory system becomes the reason it waits. Feed it fewer than 295 FLOPs of work per byte and the arithmetic units idle; the workload is memory-bound — throughput set by bytes moved, not by math. Feed it more and it is compute-bound — the arithmetic units are the wall, and bandwidth is no longer the story.

This is the same shape as capacity-planning a service where every request scans data from disk: past a point, adding CPU does nothing, because requests per second equals disk bandwidth divided by bytes per request. It is the “we’re I/O-bound, more cores won’t help” argument. On a GPU serving LLM decode, the “disk” is HBM, and, as §2 shows, the bytes per request are large.

Two immediate consequences of the table, before the roofline formalizes them.

First, a floor you can compute in your head: touching everything in HBM once — which a decode step approximately does — takes

80 GB / 3.35 TB/s = 80e9 / 3.35e12 = 24 ms

so any workload that must sweep the whole card’s memory per step is capped at ~40 such steps per second, regardless of the arithmetic units. §4 fills the 80 GB with weights and KV cache and the cap becomes a token rate.

Second, one kernel makes the hierarchy concrete. FlashAttention’s main contribution is not writing the attention score matrix to HBM: it tiles the computation so scores are produced, used, and discarded inside SRAM, trading extra arithmetic for eliminated memory traffic. Spending extra FLOPs to save bytes is a good trade on a chip with 295 FLOPs to spend per byte, and it is the design principle behind most of the fast kernels profiled in §9.

Almost every term in an inference job description — batching, paged KV, quantization, speculative decoding, disaggregation — is a scheme for changing the FLOPs-per-byte of decode, or for spending the FLOPs decode leaves idle. Most of the chapter derives from the 295 ratio.

2. Roofline modeling

The roofline plot answers, for any kernel or workload, what throughput it is allowed to reach on a given chip, before you profile anything, from two numbers you can look up and one you can count.

Define the x-axis first. Arithmetic intensity is the FLOPs a workload performs per byte it moves from HBM:

intensity = FLOPs performed / bytes moved from HBM      units: FLOP per byte

The roofline is then a ceiling with two segments:

attainable FLOP/s = min( peak compute ,  intensity × bandwidth )

At low intensity the second term is smaller: you are paying for bytes, and doubling intensity doubles attainable throughput — a rising slope whose gradient is the memory bandwidth. At high intensity the first term caps you: the flat compute roof. The corner where they meet is the ridge point, and it is the 295 from §1, computed properly:

ridge = peak compute / bandwidth
      = 989e12 FLOP/s / 3.35e12 B/s
      = 295 FLOP per byte

One note on the numerator, because it is the most common way to get this wrong by 2×. NVIDIA’s headline figure for the H100 is 1,979 TFLOPS BF16, but that assumes 2:4 structured sparsity, a format where two weights in every group of four are zero and skipped by the hardware. Served LLM weights are dense; a standard checkpoint has no such structure. The dense peak is 989. Quoting the sparse figure halves your ridge point and doubles your apparent MFU (§9).

 attainable                SCHEMATIC - log-log axes, shape only
 FLOP/s
   989 T |. . . . . . . . . . . _______________●________   compute roof
         |                     /.               prefill,
         |                    / .               2,048-token prompt
         |        memory     /  .
         |        roof:     /   .
         |        slope =  /    .
         |        3.35TB/s/     .
  3.35 T |. . ●. . . . . /      .
         |  decode,     /       . ridge = 295
         |  batch 1    /        .
         +----1--------------- 295 ------- 2,048 ---------
                    arithmetic intensity, FLOP per byte

Placing decode on it

Generating one token means multiplying activations through essentially every parameter of the model. Two facts give the intensity directly.

First, the FLOP count. A matrix multiply against N parameters does one multiply and one add per parameter — a multiply-accumulate — so a forward pass costs ~2 FLOPs per parameter per token. (Attention over the KV cache adds more on long contexts; ignoring it makes decode look better than it is, so the conclusion below survives the simplification.)

Second, the byte count. Each BF16 parameter is 2 bytes, and at batch 1 every one of them must be streamed from HBM for every single token — 16 GB of weights cannot be cached in 50 MB of SRAM, so nothing carries over from one token to the next.

decode intensity (batch 1) = 2 FLOPs per parameter / 2 bytes per parameter
                           = 1 FLOP per byte

(Worth asking why the denominator is 2 and not 4. Models are served in 16-bit — BF16 keeps FP32’s exponent range in half the bytes — because on the memory roof, bytes are throughput: serving in FP32 would double every byte count in this chapter and halve every decode ceiling, for accuracy nothing downstream can measure. Quantization in §10 is the same logic continued past 16.)

One, against a ridge of 295. Batch-1 decode sits at the far left of the plot, and the roofline gives the cost:

attainable = min( 989e12 , 1 × 3.35e12 )  =  3.35e12 FLOP/s
fraction of peak = 3.35e12 / 989e12       =  0.34%

The GPU runs at a third of one percent of its rated compute. Not because of misconfiguration, but because the workload cannot feed the arithmetic units. Check it in the time domain on the 8B reference model, one line per step:

weights                = 8e9 params × 2 B          = 16 GB
time to stream them    = 16e9 / 3.35e12            = 4.78 ms per token
decode ceiling         = 1 / 0.00478               = 209 tokens/s
FLOPs per token        = 2 × 8e9                   = 16 GFLOP
time to compute them   = 16e9 / 989e12             = 16.2 µs
waiting-to-working     = 4,780 µs / 16.2 µs        = 295 : 1

The 295 reappearing is not a coincidence: it is the ridge point measured in time instead of intensity, 295 microseconds waiting on memory for every microsecond of arithmetic. This is also §1’s latency-hiding mechanism failing for lack of work: one token does not contain enough independent operations to keep 132 SMs busy while 16 GB streams past. (Chapter 00 §4 computed the same ceiling from the KV-cache side at 100K context; at short contexts the weights are the dominant stream, and the two calculations meet in §4.)

Placing prefill on it

Prefill processes the whole prompt in one pass, so each weight fetched from HBM is used against every prompt token before it is discarded:

prefill intensity ≈ 2 FLOPs × n_tokens per parameter / 2 bytes per parameter
                  = n_tokens FLOP per byte

2,048-token prompt: intensity ≈ 2,048  >>  ridge 295   ->  compute-bound

Same model, same weights, same GPU, opposite side of the ridge. The roofline gives this side too:

prefill FLOPs (8B, 2,048 tokens) = 2 × 8e9 × 2,048   = 32.8 TFLOP
at the full 989 TFLOPS roof      = 32.8e12 / 989e12  = 33 ms
at a realistic 50% of peak       = 32.8e12 / 494e12  = 66 ms

(No real workload sits on the roof — attention, normalization, and kernel-launch overhead drag it down; 40-60% of peak is a strong prefill, a claim §9 turns into the measurable MFU. The 50% here is an assumption, flagged and reused consistently.)

One boundary case falls out of the formula: prefill intensity is ≈ the prompt length in tokens, so a prompt shorter than the ridge point (under ~295 tokens) is still on the memory roof. A fleet serving short classification prompts never sees compute-bound prefill; “prefill is compute-bound” holds only for prompts long enough to amortize the weight-read. The formula tells you where a given workload sits.

Prefill saturates arithmetic; decode starves it. One model, two phases, two different regimes, which is §3’s subject.

Batching moves decode up the roofline

Batch-1 decode has one exit: make each weight-read serve more than one token. Decode B sequences together, one new token for each of B users in the same forward pass, and each streamed parameter contributes to B token generations:

decode intensity at batch B ≈ B FLOP per byte
break-even with the ridge:    B ≈ 295

So decode needs a batch in the hundreds before the compute roof is visible. Two caveats matter later.

First, the KV cache does not amortize. Weights are shared across the batch; each sequence’s KV cache is its own, and every step streams all of it. Write the intensity with both terms to see what long contexts do:

effective intensity = B × 2P FLOPs / ( 2P + B × kv_bytes ) bytes
                      where P = params, kv_bytes = per-sequence live cache

8B, B = 64, short chats (600 live tokens, 78.6 MB each — §4):
    bytes = 16e9 + 64 × 78.6e6 = 21.0e9
    intensity = 64 × 16e9 / 21.0e9              = 49    (not 64)

8B, B = 64, long contexts (8,000 tokens, 1.05 GB each):
    bytes = 16e9 + 64 × 1.05e9 = 83.2e9
    intensity = 64 × 16e9 / 83.2e9              = 12    (not 64)

Same batch, a quarter of the intensity, purely because the caches grew. Long-context serving is memory-bound twice over, in capacity and bandwidth, which is why §4 sizes the cache carefully and why KV-shrinking techniques compound.

Second, holding hundreds of sequences resident is a memory-capacity problem: every one needs its KV cache in HBM simultaneously. That is why §5’s memory manager is one of the largest throughput features shipped for LLM serving.

Batching is the primary lever; everything in §§4-6 exists to let you push it further.

For any performance question, draw the roofline first: two axes, two roofs, the ridge at ~295, decode at 1, prefill at prompt-length. Then place the question on it. “Should we quantize?” is “move decode right by shrinking bytes.” “Why is util high but throughput low?” is “you are on the memory roof, and util doesn’t measure that.”

3. Prefill and decode are two different machines

The user-visible metrics split along the phase boundary:

• TTFT — time to first token: queueing plus the entire prefill. The delay before anything appears.
• ITL — inter-token latency: the gap between one streamed token and the next, observed per token. Its spikes are what users perceive as stutter.
• TPOT — time per output token: the mean of the ITLs over a response. TPOT summarizes; ITL’s distribution diagnoses. A clean mean with a bad p99 has a specific cause, covered two paragraphs down.

(Latency budgets a whole request in these terms from the client side; here the concern is what one GPU does to them when it serves both phases at once.)

Put representative numbers on both, 8B model, using §2’s arithmetic. A 2,048-token prompt prefills in ~66 ms at the assumed 50% of peak, so TTFT lands well under typical targets before queueing. Decode at a healthy batch runs a step every ~6 ms (derived in §5), so a smooth stream shows ITL ≈ 6 ms. Representative chat SLOs — TTFT p90 under ~500 ms, ITL p99 under a few tens of ms — are met, in the no-contention case.

Budget a whole request from those two numbers and note where the time goes:

TTFT                        =  66 ms      (prefill, 2,048 tokens)
decode, 250 output tokens   = 250 × 6 ms  = 1,500 ms
total                       ≈ 1.57 s
decode share                = 1,500 / 1,566 = 96%

Ninety-six percent of the request’s wall clock is decode, the phase running at 0.3% of the chip’s arithmetic. That is Latency’s “decode is 89% of it” observation re-derived from the hardware side, and it is why the latency levers that matter (§10) target decode: shortening the prompt only attacks the 4%.

Head-of-line blocking

A serving GPU never runs one request. At any instant it holds a batch of in-flight decodes, each wanting a small, regular ~6 ms step, and a queue of new arrivals, each wanting a prefill that is a large block of compute. The two workloads share one set of SMs, and a naive scheduler runs whole requests at a time. Then a long prompt arrives:

in-flight decode step                               ≈ 6 ms    -> smooth ITL ≈ 6 ms
new arrival: 8,000-token prompt
prefill FLOPs      = 2 × 8e9 × 8,000               = 128e12 FLOP
prefill time       = 128e12 / (0.5 × 989e12)       = 259 ms

every in-flight decode stalls for ~260 ms          -> worst ITL ≈ 43× baseline

One long prompt turns every concurrent user’s smooth stream into a quarter-second freeze. That is head-of-line blocking, a long job in a shared queue delaying every short one behind it, and it is why the ITL distribution is the useful signal: the mean barely moves while p99 spikes in correlation with long-prompt arrivals.

Priorities alone cannot fix it. “Decode always preempts prefill” starves TTFT: arrivals queue behind a continuous decode step and never get a first token. “Prefill first” is the 260 ms freeze as policy. The phases want opposite things from the same hardware, and no priority ordering gives both:

The bottom row also says the two phases do not autoscale on the same signal: prefill demand grows with prompt traffic, decode demand with concurrent streams, and a workload shift (longer documents, chattier sessions) moves them independently.

This tension resurfaces twice. §6’s chunked prefill makes the two phases share one GPU without interference, a scheduling answer. §8’s disaggregation stops making them share at all, an architecture answer. Which one you need is a fleet-size question.

4. The KV cache, sized honestly

Chapter 00 §4 derived why the KV cache exists and what decode pays to re-read it; this section treats it as a capacity problem, because on a serving GPU the KV cache decides your batch size, and §2 made batch size the primary lever.

The size formula, term by term. Attention stores one K vector and one V vector per token, the leading 2. It does so independently in every layer, because each layer runs its own attention (Attention and why context costs what it does is the mechanism). Within a layer, one K/V pair per KV head, each head_dim numbers wide, each number dtype_bytes wide:

bytes per token = 2 × n_layers × n_kv_heads × head_dim × dtype_bytes

n_kv_heads is its own factor rather than “the number of heads” because in GQA (grouped-query attention) several query heads share one K/V pair, precisely so this number is smaller than the query-head count. MQA (multi-query attention) is the limit case: every query head shares a single K/V pair. Both are architecture decisions made at training time whose purpose is visible in this serving-time formula.

Substitute the 70B shape (80 layers, 8 KV heads via GQA, head_dim 128, FP16 = 2 bytes):

bytes per token = 2 × 80 × 8 × 128 × 2       = 327,680 B  ≈ 320 KB
8K sequence     = 327,680 × 8,000            = 2.62 GB     (8K taken as 8,000 tokens)

A third of a megabyte per token, two and a half gigabytes per long chat. The formula is linear in context:

The last row is the notable one: a single 128K-context conversation uses more than half an H100’s memory in cache alone. Long-context serving is a regime where a handful of requests exhaust the card and §2’s batching lever barely moves.

Why 70B needs tensor parallelism before anything else

Now try to place the model itself on one card:

weights = 70e9 × 2 B = 140 GB    vs    HBM = 80 GB

The weights alone are 1.75× the card. Before any KV cache, activation, or batch, a 70B model in 16-bit cannot fit on one H100. This is about fit, not speed, and it is the first reason 70B-class serving is tensor-parallel (§7). At TP = 4 the weights shard to 35 GB per GPU and the KV cache shards with the heads (8 KV heads / 4 GPUs = 2 per GPU):

per GPU:   80 − 35 (weights) − 4 (runtime reserve)  = 41 GB for KV
aggregate: 4 × 41                                    = 164 GB
concurrent 8K sequences ≈ 164 / 2.62                 ≈ 62

Max concurrency, worked on the model that does fit

The 8B model on a single H100, same formula, every substitution written out:

bytes per token   = 2 × 32 × 8 × 128 × 2      = 131,072 B  = 128 KB
per-seq KV at 8K  = 131,072 × 8,000           = 1.05 GB

weights           = 8e9 × 2 B                 = 16 GB
runtime + activations reserve                 ≈  4 GB
free for KV       = 80 − 16 − 4               = 60 GB

max concurrent    = 60 / 1.05                 ≈ 57 sequences

Compare against §2’s break-even: decode wants a batch near 295 to reach the ridge, and a full card of worst-case 8K sequences supports 57. Even with memory fully packed, decode stays memory-bound. The batching lever runs out of memory before it runs out of usefulness, so every byte shaved off the KV cache converts directly into batch, and batch into throughput.

This is why KV-shrinking choices are made at model-design time, before a serving engineer sees the model. Same 70B shape under the three attention designs:

GQA is an 8× cache reduction taken before any serving technique runs: a 70B with MHA would hold three 8K conversations per 80 GB of cache space; with GQA it holds twenty-four. The serving-time equivalent, quantizing the KV cache to FP8 and halving dtype_bytes, buys another 2× from the same formula, and the two stack.

Where this chapter’s ceiling meets chapter 00’s

Two decode ceilings are now in circulation and appear inconsistent: §2 derived 209 tok/s counting only the weights; chapter 00 §4 derived ~102 tok/s counting only the cache. Neither is wrong; each ignored the other’s stream, and a real decode step pays both:

step time = ( weight bytes  +  live KV bytes ) / bandwidth

Run it for the 8B at batch 1 as the context grows, one line per regime:

    1,000 tokens:  16 + 0.13 GB = 16.1 GB  ->  16.1 / 3,350 GB/ms = 4.8 ms  ->  208 tok/s
   10,000 tokens:  16 + 1.3  GB = 17.3 GB  ->  17.3 / 3,350       = 5.2 ms  ->  193 tok/s
  100,000 tokens:  16 + 13.1 GB = 29.1 GB  ->  29.1 / 3,350       = 8.7 ms  ->  115 tok/s

Short contexts: the weights dominate and §2’s ceiling holds. Long contexts: the cache takes over and chapter 00’s number is the truer one. The crossover — where one conversation’s cache costs as much bandwidth as the entire model — sits where the two streams are equal:

16e9 bytes / 131,072 bytes per token ≈ 122,000 tokens

One 122K-token context doubles the cost of every token this sequence generates. Per-sequence decode cost is not flat; it degrades with context length even after the memory to hold it is paid, which is the hardware-level reason long conversations get slower, and the second reason (after §2’s intensity drag) that compaction and context discipline are performance features, not only cost ones.

“Size the KV cache for model X” is a standard question, and the trap in it is the head count. Using 64 heads for a GQA model inflates the answer 8×. The safe answer names the formula, states whether the model uses GQA, and substitutes explicitly.

5. PagedAttention — virtual memory for the KV cache

The §4 arithmetic assumed every allocated KV byte holds a real token. In serving systems before 2023, most did not, and fixing that assumption produced the largest single throughput win in the modern stack.

The fragmentation problem

A sequence’s KV cache grows one token at a time toward a length nobody knows in advance: the model stops when it emits a stop token, and users cancel. The attention kernels of the day required each sequence’s cache to be contiguous in memory. The only safe contiguous allocation for an unpredictable length is the worst case: reserve max_len up front, per request, on arrival.

reservation per request (8B, max_len 8,192)  = 8,192 × 131,072 B = 1.07 GB
a chat turn that actually uses 300 tokens    =   300 × 131,072 B = 39 MB
internal fragmentation                        = 1 − 39 MB / 1.07 GB ≈ 96%

That reserved-but-unused space is internal fragmentation: memory allocated to a request and holding nothing, the same pathology as a fixed-size buffer pool sized for the largest message. It was not the only waste. Pre-paged systems wasted KV memory three ways:

• Internal fragmentation — the worst-case reservation above, by far the biggest term.
• External fragmentation — variable-sized contiguous slabs leave unusable gaps between allocations, like a heap without a compactor.
• Duplication — sequences sharing a prefix (same system prompt, same few-shot header) each stored their own identical copy of its K/V.

Across realistic traffic the vLLM paper measured 60-80% of KV memory wasted across the three. The GPU was not short of memory but short of usable memory, and batch size, the primary lever, was capped by the waste.

The fix, borrowed from operating systems

PagedAttention is the vLLM project’s application of virtual memory to the KV cache:

• Carve KV memory into fixed-size blocks of, say, 16 tokens each (the page).
• Give each sequence a block table: a small array mapping its logical token positions to whichever physical blocks it owns, no contiguity required (the page table).
• Allocate a new block only when the previous one fills: allocation on demand, not on arrival.

The attention kernel walks the block table the way a CPU’s memory unit walks page tables:

sequence A, 40 tokens live, block size 16:

logical tokens   0-15    16-31    32-39 and growing
physical block     7       12        3
block 3 is 8/16 full -> the ONLY waste this sequence has

Fragmentation collapses to at most one partially-filled block per sequence:

worst-case waste per sequence = 16 tokens × 131,072 B ≈ 2 MB    (vs ~1 GB reserved)

The block size is itself a trade-off, the same one page size is in an OS. Smaller blocks bound waste tighter but make block tables longer and KV reads more scattered, more indirection per attention step; bigger blocks read more contiguously but let internal fragmentation return (a 1,024-token block wastes up to 1,023 tokens per sequence, halfway back to the reservation model). Implementations landed around 16 tokens.

Two more OS ideas come for free, and they eliminate the duplication term. Sequences sharing a prefix point their block tables at the same physical blocks: prefix sharing, one copy of the system prompt regardless of how many users share it. And a shared block is copied only when someone writes into it: copy-on-write, the same semantics as fork. One 2,000-token system prompt at 100 concurrent users:

naive:  100 × 2,000 × 131,072 B = 26.2 GB of identical bytes
paged:    1 × 2,000 × 131,072 B = 0.26 GB

(When memory still runs out, from a burst of long generations, the engine preempts: it evicts a sequence’s blocks and later either recomputes them or restores them from a copy swapped to host DRAM over PCIe. vLLM exports preemption counters; a nonzero rate signals that the KV budget, not the scheduler, is the binding constraint.)

What it buys, in tokens per second

The point was never memory hygiene; it was batch. Same 8B card, same 60 GB of KV space, a mixed workload averaging 600 live tokens per sequence:

live KV per sequence = 600 × 131,072 B                    = 78.6 MB

naive batch  = 60 GB / 1.05 GB reservation                ≈ 57 sequences
paged batch  = 60 GB / 78.6 MB actual                     ≈ 760 sequences

The next step is subtler than “13× the batch, 13× the throughput.” Wasted space is not wasted traffic: the naive system never reads its empty reservations, so both systems stream only live bytes. What changes is how many sequences share each pass over the weights:

per-step HBM traffic = weights + live KV:
naive:  16 + 57 × 0.0786   = 20.5 GB  -> 20.5 / 3,350 GB/ms = 6.1 ms  ->  57 / 0.0061  ≈  9,300 tok/s
paged:  16 + 760 × 0.0786  = 75.7 GB  -> 75.7 / 3,350 GB/ms = 22.6 ms -> 760 / 0.0226  ≈ 33,600 tok/s

3.6× the throughput from the same hardware, inside the 2-4× the vLLM paper reported over the systems it compared against, and the mechanism is §2’s: more sequences amortizing each weight-read, intensity climbing the memory roof. Note the trade the arithmetic exposes: ITL rose from 6.1 ms to 22.6 ms, because a bigger batch makes every step heavier. Throughput and per-token latency trade against each other, and production schedulers cap batch to hold an ITL target rather than maximizing tokens per second, which is why a serving config has a knob there.

vLLM is the reference implementation and the default open-source serving stack; PagedAttention has since been adopted almost everywhere (TensorRT-LLM, SGLang, and others) to the point where “the KV cache is paged” is now an assumption, not a feature.

To measure it: serve an 8B model with vLLM, sweep --max-num-seqs, and plot tokens/s and p99 ITL against batch while watching the KV-cache-usage and preemption counters it exports. You will see throughput climbing near-linearly with batch, then flattening as KV memory saturates, ITL rising throughout, and preemptions appearing where the curve bends. That is §2’s roofline and §4’s capacity wall traced empirically.

6. Batching strategies

Batching is the lever; the strategy question is when requests are allowed to join and leave the batch. Three generations of answer, each fixing the previous one’s idle time.

Static batching: wait, pad, drain

Static batching is inherited from the translation-model era: collect B requests, pad them to equal length, run the whole batch to completion, admit nobody until it drains. It assumes what was true for translation, outputs of similar, predictable length, and LLM traffic violates that: generation lengths vary by two orders of magnitude and are unknown in advance. The batch runs at the pace of its longest member while finished slots idle:

batch of 8, output lengths 100, 200, ..., 800 tokens
slot-steps run    = 8 × 800   = 6,400
slot-steps useful = 100 + 200 + ... + 800 = 3,600
utilization       = 3,600 / 6,400 = 56%

Nearly half the decode capacity is spent computing padding, since the tail of every batch is mostly empty, plus a second cost: arrivals queue until the entire batch drains, so one 800-token response blocks admission for seconds. Head-of-line blocking again, this time at the batch boundary.

Continuous batching: the iteration is the scheduling unit

Continuous batching is the fix, introduced by the Orca paper as iteration-level scheduling. Decode has a natural preemption point every few milliseconds, the token boundary, because every sequence in the batch does the same thing: one forward pass, one token. So schedule iterations instead of requests: after every decode step, finished sequences exit the batch and waiting requests join it. A slot that frees at token 100 serves a new user at token 101.

flowchart LR
    subgraph ST["Static - slots wait for the longest request"]
        S1["step 1<br/>A B C D"] --> S2["step 150<br/>A · C ·"] --> S3["step 700<br/>A · · ·"] --> S4["drain, then<br/>admit E F G H"]
    end
    subgraph CB["Continuous - slots refill at token boundaries"]
        C1["step 1<br/>A B C D"] --> C2["step 150<br/>A E C F"] --> C3["step 700<br/>G E H F"] --> C4["never drains"]
    end

    style S2 fill:#9d0208,color:#fff
    style S3 fill:#9d0208,color:#fff
    style C2 fill:#2d6a4f,color:#fff
    style C3 fill:#2d6a4f,color:#fff

Utilization no longer depends on length variance: the batch stays full as long as the queue is nonempty, and the padding waste disappears with it. The same move removes the batch-boundary queueing: a new arrival waits at most one iteration (milliseconds) for admission instead of one batch-drain (seconds), a TTFT win the utilization number does not show. This is also why this and §5 ship together: join-anytime scheduling needs allocate-anytime memory. A scheduler admitting a sequence mid-flight cannot pre-reserve a contiguous gigabyte; paged KV is what makes iteration-level admission affordable. Every modern engine (vLLM, TensorRT-LLM, SGLang) runs this pair by default.

Chunked prefill: the knob and its trade

Continuous batching still gets one thing wrong, computed in §3: joining the batch requires a prefill, and a long prefill freezes every in-flight decode for ~260 ms. The scheduler has two bad options: run the prefill now and spike everyone’s ITL, or defer it and let TTFT degrade in the queue.

Chunked prefill (the Sarathi lineage, now a standard engine option) resolves this by splitting the prompt into fixed-size chunks, say 512 tokens, and co-scheduling one chunk per iteration alongside the normal decode step instead of letting prefill monopolize the GPU:

8,000-token prompt, 512-token chunks               = 16 chunks
chunk compute = 2 × 8e9 × 512 / (0.5 × 989e12)     = 16.6 ms
iteration     = decode step 6.1 + chunk 16.6       ≈ 23 ms

worst ITL during a long admission:  260 ms  ->  ~23 ms    (11× better)
TTFT for the long prompt: 16 chunks × 23 ms ≈ 370 ms  vs  ~260 ms dedicated  (+40%)

That is the knob and its trade: chunk size buys ITL smoothness for the batch at a TTFT cost to the long-prompt request. Small chunks favor streaming chat; big chunks favor TTFT-sensitive, long-document workloads. In engine configs the same idea usually appears as a token budget per iteration, one number capping decode tokens plus prefill-chunk tokens per step, which is the actual knob.

The underlying point: chunked prefill deliberately lowers prefill efficiency (each chunk re-reads weights a monolithic prefill would read once) to protect a latency SLO. It is the standard soft-real-time trade, smooth over fast, and when no chunk size satisfies both sides, that is the case for §8.

The three strategies, as one decision:

7. Tensor and pipeline parallelism

§4 established that 140 GB of 70B weights cannot fit on an 80 GB card. Multi-GPU inference is not an optimization for large models; it is the entry requirement. The two ways to split a model differ in what they cut, and the communication arithmetic decides which to use where.

One distributed-systems reflex to set aside first: “just add replicas” does not apply. Replication, the same full model on more GPUs behind a load balancer, scales request throughput and is correct once the model fits; it does nothing for a model that does not fit and nothing for per-token latency, because each replica still faces §2’s memory wall alone. TP and PP are model parallelism: they split one copy of the model itself, which is why they exist and why they cost communication.

Tensor parallelism: cut every matrix

Tensor parallelism (TP) splits each weight matrix across p GPUs (attention heads distributed, so 8 KV heads at TP = 4 means 2 per GPU with the KV cache sharding along with them; MLP columns likewise) so every GPU computes a slice of every layer, in lockstep, for every token.

Partial results must be recombined, exactly twice per layer. The standard Megatron-style layout splits the first matrix of a pair column-wise and the second row-wise, so the intermediate never needs communicating, but the output of each pair is a partial sum every GPU needs in full. There are two such pairs per transformer layer (the attention block ending in its output projection; the MLP ending in its down-projection), hence two all-reduce operations (the collective where every GPU ends up holding the sum of all GPUs’ contributions) per layer, per token. The library performing these collectives is NCCL (NVIDIA’s collective-communication library), whose kernels appear by name on the §9 timeline.

Per token, the payload of each all-reduce is one activation vector, so communication scales with the hidden dimension. The 70B shape at TP = 4, decoding, one line per step:

activation per token = 8,192 × 2 B                 = 16 KB
ring all-reduce traffic per GPU ≈ 2(p−1)/p × 16 KB = 24 KB
collectives per token = 2 × 80 layers              = 160
bytes per token per GPU = 160 × 24 KB              = 3.8 MB
wire time on NVLink     = 3.8e6 / 900e9            ≈ 4 µs

The bytes are trivial. What is not trivial is that a decode step issues 160 synchronizing collectives, each carrying a fixed launch-plus-latency floor of a few microseconds regardless of payload. Count latency, not bytes:

latency floor ≈ 160 × 5 µs                         = 0.8 ms per token
decode step at TP = 4: weights 35 GB / 3.35 TB/s   = 10.4 ms
overhead on NVLink                                 ≈ 8%

same 160 collectives over PCIe (~25 µs each)       ≈ 4 ms  ->  ~40% overhead

That last line is the placement rule: TP communicates heavily per token, so it stays on NVLink, inside one node. Across a slower link the collectives dominate the step. In exchange, TP is the only scheme that reduces per-token latency: each GPU streams 1/p of the weights, so §2’s memory wall divides by p:

70B, batch 1, TP = 4:  35 GB / 3.35 TB/s = 10.4 ms  + 0.8 ms comm  ≈ 11.2 ms
decode ceiling                                                     ≈ 90 tok/s

for a model that could not fit on one card at all.

Pipeline parallelism: cut the layer stack

Pipeline parallelism (PP) assigns each GPU a contiguous run of layers, a stage, and tokens flow through the stages in sequence: 80 layers at p = 4 is 20 layers per stage. Communication per stage boundary is one 16 KB activation handoff, three boundaries total, once per token; the bytes and message counts are small enough that the interconnect barely matters. PP is what crosses nodes.

The price is the pipeline bubble. While stage 1 works on the first unit of work, stages 2-4 sit idle; symmetrically, stage 1 goes idle while the last unit drains through. The fix is to keep the pipe full: split the batch into m microbatches fed in back-to-back so all stages overlap on different waves. The idle fraction is start-up plus drain over total slots:

time to fill the pipe       = p − 1 stage-steps
useful work                 = m stage-steps per stage
total                       = m + p − 1

bubble fraction = (p − 1) / (m + p − 1)

p = 4, m = 16:   3 / 19 = 15.8% of stage-time idle
p = 4, m = 64:   3 / 67 =  4.5%

The bubble shrinks only when microbatches greatly outnumber stages, which requires traffic. PP is therefore a throughput regime, not a latency one: a token still visits all 80 layers sequentially, so single-stream latency does not improve, and a deep pipeline under low traffic is mostly bubble.

Choosing, and combining

The standard layout follows directly: TP as far as the NVLink domain reaches (typically the 8 GPUs of one node), and PP across nodes on top only when a whole node is still too small. The large-model case:

405B model, BF16:  405e9 × 2 B = 810 GB of weights
one node, TP = 8:  810 / 8     = 101 GB per GPU   -> does not fit in 80
TP = 8 × PP = 2 (16 GPUs):  810 / 16 ≈ 51 GB per GPU  -> fits, with room for KV

Two nodes, TP inside each, PP between them: every placement decision came from the arithmetic above.

flowchart LR
    subgraph N1["Node 1 — PP stage 1 (layers 1-40)"]
        A["GPU 0"] --- B["GPU 1"] --- C["... GPU 7"]
    end
    subgraph N2["Node 2 — PP stage 2 (layers 41-80)"]
        D["GPU 8"] --- E["GPU 9"] --- F["... GPU 15"]
    end
    N1 -->|"activation handoff (PP, inter-node)"| N2

    style N1 fill:#2d6a4f,color:#fff
    style N2 fill:#40916c,color:#fff

Within each node the 8 GPUs run TP over NVLink (all-reduces per layer); between nodes a single activation crosses per token via PP.

(To measure it, extend §5’s project: vLLM takes --tensor-parallel-size on a multi-GPU box. Serve the same model at TP=1 and TP=2 and watch batch-1 latency drop by nearly half while tokens-per-GPU throughput barely moves or dips, which is the collectives’ overhead made visible.)

Training uses the same two words plus more (data parallelism, gradient all-reduces, optimizer sharding) but optimizes a different objective: step time over a fixed corpus with gradients to synchronize, where inference parallelism only has to fit weights plus KV and hit a latency target. Scale covers that side, and conflating the two regimes is a common interview trap.

8. Disaggregated serving and KV-aware routing

Chunked prefill made the two phases share a GPU without interference. The stronger move, once the fleet is large enough, is to stop making them share.

Disaggregated serving runs prefill and decode on separate GPU pools. A request lands on the prefill pool, which computes the TTFT-critical work as one uninterrupted compute-bound block, no chunking; the KV cache it produces ships to a decode-pool worker, which streams tokens from it with a step nothing interrupts. The idea comes from the DistServe and Splitwise papers; Dynamo is NVIDIA’s productized version of the same lineage, and rack-level designs pair bandwidth-heavy SKUs with the decode pool for the reasons §2 derived.

The network hop is worth it because §3’s table showed the phases scale on different axes, and a shared GPU forces one machine to be provisioned for both. Separate them and each pool is sized, scheduled, and scaled against its own bottleneck:

• The prefill pool is compute-bound: size it in prompt-tokens-per-second, scale it when TTFT degrades.
• The decode pool is bandwidth- and memory-bound: size it in concurrent-sequences-at-target-ITL, scale it when ITL degrades.

Interference is gone by construction: there is no long prompt to stall a decode, because they are on different machines, and the pools autoscale independently. It is the same argument as splitting a monolith’s read and write paths onto separately-scaled replicas. DistServe frames the win as goodput: requests per second that meet both the TTFT and ITL SLOs, rather than raw tokens per second, and shows shared GPUs sacrificing goodput even at high utilization, because utilization does not track which SLO was missed.

Size the pools with arithmetic. Representative traffic (100 req/s, mean prompt 2,000 tokens, mean output 250 tokens) against representative per-pool capacities (measure your own):

prompt demand = 100 × 2,000 = 200,000 prompt tok/s
prefill capacity ≈ 10,000 tok/s per GPU group     -> 20 prefill groups

output demand = 100 × 250   = 25,000 output tok/s
decode capacity ≈ 2,500 tok/s per GPU group       -> 10 decode groups

A 2:1 pool ratio for this mix. Writing it as two independent divisions makes clear that the ratio moves when the workload does. Longer documents: prefill pool grows, decode untouched. Chattier sessions: the reverse. A shared fleet cannot express that; two pools can.

The hop has a cost. One 8K-token 70B request’s KV, from §4:

KV to ship = 2.62 GB
over NVLink        900 GB/s:  2.62 / 900   ≈  3 ms
over 400 Gb RDMA    50 GB/s:  2.62 / 50    ≈ 52 ms

(RDMA, remote direct memory access, is the NIC writing straight into the peer machine’s memory with no CPU copies on either side; it is how KV moves between nodes.) Against a TTFT already in the hundreds of milliseconds, 3-52 ms of transfer is affordable, and engines hide most of it by shipping layer by layer, overlapped with the prefill still producing later layers. The same arithmetic says when disaggregation is wrong: small models, short prompts, or a fleet too small to keep two pools independently busy, where the transfer plus operational complexity buys nothing chunked prefill was not already delivering. Below roughly node scale, share; at fleet scale, split.

flowchart LR
    U([Requests]) --> RT["Router<br/>KV-aware"]
    RT --> P1["Prefill pool<br/>compute-bound<br/>scales on prompt tok/s"]
    P1 -->|"KV over NVLink / RDMA"| D1["Decode pool<br/>bandwidth-bound<br/>scales on concurrent seqs"]
    RT -->|"prefix already warm here"| D1
    D1 --> O([Streamed tokens])

    style P1 fill:#2d6a4f,color:#fff
    style D1 fill:#bc6c25,color:#fff
    style RT fill:#40916c,color:#fff

KV-cache-aware routing

Once KV caches are first-class objects that live somewhere, where a request lands matters. KV-cache-aware routing sends a request to the worker that already holds the KV blocks for its prefix, instead of load-balancing round-robin and re-prefilling from scratch.

The workload where this matters most is multi-turn chat, and the arithmetic is Deriving the numbers’ statelessness fact seen from the server side: turn t’s prompt is turn t−1’s entire conversation plus one message, so the prefix-reuse rate approaches 100%. Route turn t to the worker that served turn t−1 and the prefill is one message long; route it elsewhere and the whole conversation re-prefills. This is the serving-side mechanism behind the prompt caching priced at the API level: Prompt caching derived derived the billing rules from causal attention; here, a cache hit means the K/V blocks are resident in that worker’s HBM and §5’s prefix sharing points the new sequence at them without copying.

Size the win on the repo’s reference agent (chapter 09 runs it at 6,000 prefix tokens plus 1,200 per turn). At turn 20 the accumulated context is 6,000 + 19 × 1,200 = 28,800 tokens. Serve it on the 8B and compare a cold worker against a warm one:

cold worker: prefill 28,800 tokens = 2 × 8e9 × 28,800 = 461 TFLOP
             at 50% of peak        = 461e12 / 494e12  = 932 ms of TTFT
warm worker: prefill 1,200 new     = 2 × 8e9 × 1,200  = 19.2 TFLOP
             at 50% of peak        = 19.2e12 / 494e12 =  39 ms of TTFT

routing hit vs miss ≈ 24× on TTFT, and the gap widens every turn

A router that lands 90% of turns warm versus one that lands 30% is worth more TTFT than any kernel optimization in this chapter, which is why the routing layer, the least GPU-specific component in the stack, is a performance component.

The open example is the llm-d project, a Kubernetes-native inference scheduler whose router scores workers by prefix overlap before dispatching; Dynamo ships a router doing the same job in the NVIDIA stack.

The failure mode is a distributed-systems problem in GPU form: prefix affinity fights load balancing. Pin every conversation to its warm worker and hot conversations create a hotspot on one GPU while its neighbors idle; balance perfectly and every hop costs a full re-prefill. Real routers score both, overlap × current load, and accept that the KV cache is now distributed state with the usual placement, eviction, and consistency problems. It is the same tension as a sticky-session tier or a sharded cache.

9. Measuring it — utilization, MFU, and the Nsight pair

Every claim so far was derived. On the job you have to verify them on a live system, and the first tool most people reach for, nvidia-smi, measures almost nothing useful here.

Why “GPU util: 95%” is nearly meaningless

nvidia-smi’s utilization is kernel residency: the fraction of the sample window during which at least one kernel was executing. Not how many SMs that kernel used. Not whether it computed or waited on memory. One kernel, one SM, running continuously:

a kernel occupying 1 of 132 SMs, always resident   ->  "GPU util: 100%"
actual compute in use                              ≤ 1/132 ≈ 0.8%

This is not hypothetical; it is this chapter’s main case. Batch-1 decode keeps a kernel resident nearly always, so nvidia-smi reads ~100% while §2 showed the chip is doing 0.34% of its possible arithmetic. “Util is high, we need more GPUs” is the wrong conclusion drawn from this number; the fix is batch size, which is free.

The memory gauge misleads the same way for a different reason: serving engines preallocate the KV space (vLLM grabs ~90% of HBM at startup and manages it internally), so “memory used” reads pinned near full whether the cache holds one sequence or seven hundred. Both headline numbers are useless for the two questions you actually have. Use the engine’s own counters for KV-pool occupancy, and the two ratios below for efficiency.

The two ratios that mean something

• MFU — model FLOPs utilization: the FLOPs implied by your token throughput, divided by peak. tokens/s × 2 × params / peak FLOP/s.
• MBU — model bandwidth utilization: the bytes your decode must stream per second (weights plus live KV), divided by peak bandwidth.

Which one matters is phase-dependent, and that is the key point of the measurement story. Decode first, 8B, batch 1, a healthy deployment measuring 146 tok/s:

bytes per token   = 16e9 (weights; short context, KV negligible)
bytes/s           = 146 × 16e9        = 2.34e12
MBU               = 2.34e12 / 3.35e12 = 70%

FLOP/s            = 146 × 2 × 8e9     = 2.34e12
MFU               = 2.34e12 / 989e12  = 0.24%

Same measurement, two denominators, opposite verdicts: this decode is healthy, at 70% of the possible bandwidth and closing on the 209 tok/s ceiling, while its MFU looks catastrophic and is irrelevant. MBU also gives a distance-to-ceiling that no counter does: 70% achieved means the remaining headroom on this workload is at most 1/0.7 ≈ 1.4×, so a promised 5× kernel win on this decode is not possible; the 5× has to come from a lever that changes the workload (batch, quantization), not the kernel. Now prefill, using Latency’s loaded-endpoint observation of ~20,000 prompt tok/s on the same class of model:

FLOP/s = 20,000 × 2 × 8e9 = 3.2e14
MFU    = 3.2e14 / 989e12  = 32%

A third of peak, decent, with headroom worth profiling for. Judge decode by MBU, judge prefill by MFU (40-60% is strong; the 50% assumed in §3 and §6 sits in that band). Quoting the wrong ratio for the phase is the measurement version of quoting sparse TFLOPS.

What goes on the dashboard

Before the profilers, which are for incidents and tuning sessions rather than routine monitoring, the standing metrics that make either ratio computable when you need it:

• Tokens per second, prefill and decode separately. One combined number hides the phase split that every diagnosis starts from.
• TTFT p50/p90 and ITL p50/p99. The tails carry the diagnoses: p99 ITL spiking with long-prompt arrivals is §3’s head-of-line signature.
• MBU and MFU, computed from the throughput numbers above and the fixed model constants — two multiplications, no profiler needed.
• KV-pool occupancy and preemptions per minute, from the engine. Occupancy pinned at 100% with preemptions climbing means the batch is capacity-capped: the §4 wall, not the scheduler.
• Batch occupancy per iteration. If it never approaches its configured cap while the queue is nonempty, something upstream (admission, KV, token budget) is the binding constraint.

Nothing in that list requires Nsight; all of it decides whether to reach for Nsight and for which of the two.

The Nsight pair

Two profilers at two levels, and the division of labor matters more than any individual feature, because using them in the wrong order wastes time.

Nsight Systems (nsys profile -o trace <your serve command>) is the system-wide timeline: every kernel launch, memcpy, NCCL collective, and CPU thread on one time axis, viewed in a GUI afterward. It answers “where did the wall-clock go?”, the questions between kernels:

• Gaps. Whitespace on the GPU row while the CPU row is busy means launch overhead is pacing the GPU: decode steps are thousands of microsecond-scale kernels, and the CPU can fail to keep up. The standard fix is CUDA Graphs: record the step’s whole launch sequence once, replay it as a single unit, and the gaps close. (Engines enable this by default; a config change can silently disable it, and the timeline is where you catch that.)
• Serialization. At TP, do NCCL collectives overlap compute, or does the timeline alternate compute-silence-compute? The §7 overhead arithmetic assumed overlap; this is where you check it.
• Foreign work on the critical path. Host-to-device copies before each step, tokenization on the serving thread, a Python scheduler stall, all visible as CPU-row activity the GPU row waits on.

Nsight Compute (ncu) is the per-kernel view: it replays one kernel (much slower, so you sample rather than wrap the whole server) and reports why that kernel is slow. It answers “is this kernel at its roofline?”:

• The Speed of Light section: this kernel’s achieved percentage of peak compute and of peak memory throughput, side by side, the roofline verdict per kernel. A decode matrix-vector kernel showing memory ~80%, compute ~5% is §2 confirmed empirically; the same shape on a prefill GEMM is a bug.
• Occupancy — resident warps per SM as a fraction of the maximum the SM can host, i.e. how much latency-hiding parallelism (§1) the kernel offers the scheduler. Low occupancy plus low memory throughput means the kernel cannot keep the memory system busy: too few threads in flight to cover latency.
• Achieved bandwidth in GB/s, to compare directly against the 3.35 TB/s spec.

The workflow is always the same order: nsys to find which kernel or gap owns the time, then ncu on that kernel to find why. Profiling kernel-by-kernel before looking at the timeline is how people spend a week optimizing a kernel that was 4% of the step.

What a healthy decode profile looks like

The checklist form, for reading someone else’s trace or your own:

• Timeline dense: kernel gaps under a few percent of step time (CUDA Graphs on and actually engaged).
• The step dominated by a few fused kernels — attention and the big matrix-vector multiplies — not thousands of slivers.
• Those kernels’ Speed of Light: memory high (60-80%+), compute low. That is correct for decode; do not “fix” it.
• Aggregate MBU at 60-80%; if it reads 20%, suspect batch too small, kernels unfused, or KV badly scattered.
• TP only: NCCL time overlapped with compute or under ~10% of the step.
• Batch pinned at the KV-memory or ITL limit, not at an unraised default.

This section covers the reading skill: what the timeline, the SOL panel, and the two ratios mean, and which conclusion follows from which. Fluency with the tools comes from running them on a real serve, the second half of §5’s project: wrap the vLLM sweep in nsys profile, find the decode step in the GUI, and check the checklist against it once.

10. The levers, ranked

Everything above, folded into a table indexed by symptom, because that is how the problem arrives:

Two rows deserve their arithmetic, because both are direct applications of §2.

Quantization stores weights in fewer bytes (INT8 or FP8, one byte per parameter) while computing in higher precision. Halve the bytes and decode intensity doubles, which on the memory roof means the ceiling doubles:

8B in INT8:  weights = 8e9 × 1 B                = 8 GB
             time per token = 8e9 / 3.35e12     = 2.39 ms
             decode ceiling = 1 / 0.00239       = 419 tok/s     (was 209)
             intensity = 2 FLOPs / 1 byte       = 2 FLOP per byte

Weights-only quantization is the safe first step (the KV cache and activations are separate, harder decisions). The caveat is that it is the first lever in this table that can affect output quality; measure on your evals before assuming the 2× is free.

Speculative decoding spends the FLOPs decode leaves idle. A small draft model proposes k tokens cheaply; the target model verifies all k in one forward pass, and because that pass is memory-bound, verifying k+1 positions streams the same 16 GB of weights as generating one. With acceptance rate α per token, the expected tokens produced per target pass:

expected tokens = 1 + α + α² + ... + α^k

k = 4, α = 0.7:  1 + 0.7 + 0.49 + 0.343 + 0.240  = 2.77 tokens per pass
                 -> up to ~2.8× lower single-stream latency

The trade is explicit: total compute goes up (a whole draft model runs, and rejected tokens are wasted work) to reduce latency, which is the right trade on a machine that is 99.7% idle arithmetic (§2) and the wrong one on a compute-saturated prefill pool.

Order of operations follows chapter 09’s discipline: measure first (MBU/MFU, the ITL distribution, KV-pool usage and preemptions); then the free structural wins (paged KV and continuous batching, which a modern engine gives you by default: confirm, don’t assume); then scheduling policy (chunked prefill, token budgets); and only then the levers that cost quality, hardware, or complexity (quantization, parallelism changes, disaggregation). Each step changes the numbers feeding the next, so the order is forced.

The mechanism → lever map

The chapter in one table, in chapter 00’s closing format: cover the right column and regenerate it from the left.

Cheat sheet

Numbers are representative (H100 SXM, BF16, spec-sheet peaks); re-derive on your SKU before quoting any of them.

What interviewers probe

“Why is decode memory-bound? Derive it.” Say: generating one token multiplies through every parameter — about 2 FLOPs per parameter — and at batch 1 every 2-byte parameter must stream from HBM, because 16-plus gigabytes of weights can’t live in 50 MB of SRAM. That’s 1 FLOP per byte, against a ridge point of 989 TFLOPS over 3.35 TB/s ≈ 295 FLOP per byte. So attainable throughput is 1 × 3.35 TB/s = 3.35 TFLOP/s — a third of a percent of peak. The fix is intensity: batch B sequences and each weight-read serves B tokens — with the caveat that KV reads don’t amortize, so long contexts drag effective intensity back down.

“Size the KV cache for a 70B model at 8K context.” Say: 2 for K and V, times 80 layers, times 8 KV heads — it’s GQA; that 8 would be 64 under vanilla multi-head, an 8× difference, so always check — times head dim 128, times 2 bytes for fp16: 320 KB per token, 2.62 GB per 8K sequence. Then the capacity point: the weights alone are 140 GB against an 80 GB card, so this model is tensor-parallel before the cache stores its first token, and at TP=4 the aggregate free memory holds about sixty 8K conversations.

“nvidia-smi shows 95% GPU util but throughput is bad. What’s wrong?” Say: that metric is kernel residency — any kernel running during the sample window counts the whole window — so a batch-1 decode reads ~100% while doing 0.3% of possible FLOPs. Measure MBU for decode and MFU for prefill instead. If MBU is high, the GPU is honestly at its memory roof and the lever is batch or quantization; if MBU is also low, profile — nsys for launch gaps and serialization on the timeline, then ncu Speed-of-Light on the dominant kernel. Most likely root cause: batch too small, usually capped by KV memory.

“When tensor parallelism vs pipeline parallelism?” Say: TP splits every matrix and pays two all-reduces per layer — 160 latency-bound collectives per token on an 80-layer model — so it needs NVLink and stays inside a node; in exchange it divides per-token latency by p and is the only way a 140 GB model exists at all. PP splits the layer stack, ships one small activation per stage boundary, crosses nodes happily, but pays the bubble — (p−1)/(m+p−1) — so it needs microbatch traffic and never helps a single stream. Standard layout: TP to the edge of the NVLink domain, PP across nodes on top; a 405B in BF16 is 810 GB, which is TP8 × PP2 across two nodes at ~51 GB per GPU.

“Your P99 ITL is terrible but the mean is fine.” Say: classic head-of-line blocking — long prefills freezing in-flight decodes; an 8K prompt is ~260 ms of solid compute walking in front of a 6 ms heartbeat, so p99 spikes exactly when long prompts arrive while the mean barely moves. Confirm by correlating ITL spikes with prompt-arrival sizes, then enable chunked prefill and tune the per-iteration token budget, accepting a bounded TTFT cost. If the fleet is large and the two SLOs keep fighting, that’s the case for disaggregating prefill and decode pools and optimizing goodput rather than tokens per second.

“How would you raise throughput without buying GPUs?” Say: in order — confirm the engine pages its KV and batches continuously, because that’s the 2-4× class of win and a misconfiguration can silently forfeit it; raise concurrency until KV memory or the ITL target caps it; quantize weights to FP8/INT8, which halves bytes per parameter and doubles the decode ceiling; and if traffic is multi-turn, add prefix caching with KV-aware routing so conversations stop re-prefilling. Every step is the same roofline argument: more FLOPs per HBM byte, or fewer bytes per token.

“What batch size should we run?” Say: it’s not a free parameter — it’s pinned between two constraints you can compute. The ceiling is the ITL SLO, because every added sequence adds its KV stream to the step (§5’s arithmetic showed batch 57 → 760 taking the step from 6 to 23 ms). The cap is KV memory: free HBM over per-sequence cache, so for the 8B at worst-case 8K contexts, 60 GB over 1.05 GB ≈ 57 — which is why paged KV, by shrinking the effective per-sequence footprint to actual usage, is what raises the cap. Then sweep between the two on real traffic and pick the largest batch that holds p99 ITL; the knee in that curve is the KV pool saturating.

“Prefill is supposed to be compute-bound — why is your prefill MFU only 30%?” Say: MFU counts only the useful model FLOPs, and real prefill spends time on things that aren’t the big matmuls — the quadratic attention kernels, normalizations, kernel-launch overhead between them — plus scheduling effects like chunked prefill deliberately splitting the work and short prompts that never amortize the weight-read (a prompt under ~295 tokens is still memory-bound by the intensity formula). 30-50% is the normal band; the way to find the specific gap is ncu on the dominant kernels — if the big GEMMs individually show high compute Speed-of-Light, the loss is between kernels, not in them.

“You keep saying roofline — what’s on the axes?” Say: x is arithmetic intensity — FLOPs performed per byte moved from HBM; y is attainable FLOP/s — the min of peak compute and intensity times bandwidth. The ridge sits at ~295 FLOP/byte on an H100 with dense BF16 peaks. Decode at batch 1 sits at x ≈ 1, prefill at x ≈ prompt length; batching, quantization, and speculative decoding are all moves on that one plot. Then offer to draw it.