Book Notes / AI Inference at Scale

Christopher Nuland, Michael Goin, and Carlos Costa / Reading notes

Selected notes from the early-release preview, checked against serving implementations.

The service promise determines what counts as capacity

A serving system converts a particular workload into completed results under constraints on quality, delay, and cost. Token throughput describes part of that process. It does not establish how many useful requests finish within the service's promised latency.
Prefill processes the known prompt; autoregressive decode generates a response sequentially, using earlier tokens to generate later ones. Time to first token and time to complete the response therefore expose different costs. A faster first token can coexist with an unchanged wait for a complete structured result.
Capacity depends on the workload shape. Longer prompts increase prefill work and retained attention state; longer outputs occupy generation capacity for more steps. A request-rate figure without those lengths can describe demand too loosely to predict whether a service will meet its promise.

A standard transformer key-value cache retains earlier keys and values so subsequent decoding steps do not have to recompute them. For ordinary full attention, its storage grows with the tokens retained across active sequences. Model weights and runtime buffers consume memory alongside that state.
Consider an illustrative cache with 32 layers, eight key-value heads per layer, head dimension 128, and two bytes per stored value. Each retained token needs 2 × 32 × 8 × 128 × 2 = 131,072 bytes: 128 KiB. An 8,192-token sequence therefore needs 1 GiB for this cache alone. Thirty-two such active sequences need 32 GiB before weights, working memory, and allocation overhead.
The two at the front counts keys and values. The head count is the number of key-value heads, which can differ from the number of query heads. Cache quantization, sliding-window attention, sharing, and other architectures change the accounting. The calculation is a capacity model for stated conditions, not a universal memory-per-token constant.
This creates a scheduling constraint. More simultaneous requests can share model-weight access and improve throughput, while their growing caches compete for finite capacity. When an engine preempts a sequence and later recomputes discarded state, additional GPU activity can represent repeated work rather than additional completed requests. Admission and cache management affect how much of the execution survives.

Prefix caching can reuse attention state for a matching token prefix under a compatible model execution. Semantic similarity alone is insufficient. A repeated leading document may save substantial prefill work; a different prefix that happens to mean the same thing cannot simply reuse those keys and values.
The direct saving is in prompt processing. It does not eliminate the sequential work of generating the requested output. A high cache-hit rate can therefore coexist with little improvement in completion latency when decoding dominates the request.
Routing adds another dependency. In an illustrative comparison, a warm replica with 80 ms of queueing and 10 ms of prefill reaches decode after 90 ms. A cold replica with 5 ms of queueing and 50 ms of prefill reaches it after 55 ms. Assuming equal subsequent decoding, the cache hit finishes later. Reuse is valuable only in relation to the work and waiting it actually removes.

If memory pressure is causing recomputation, reducing simultaneous retained state should reduce preemption under a comparable workload. That may also reduce opportunities to batch work. Track both changes: freeing memory does not establish that overall service capacity increased.
Compare completed requests within the relevant latency budget, while holding the quality requirement and workload distribution explicit. Include failed requests and timeouts. Otherwise, a configuration can look better by doing less useful work or leaving the hardest requests outside the measured population.
Keep the conclusion as narrow as the experiment. The arithmetic above establishes a memory requirement; the routing example establishes a possible failure of a cache-only policy. Neither measures an actual deployment. Performance claims still need traces or benchmarks that exercise the intended prompt lengths, output lengths, arrivals, and reuse pattern.