Prefill, Decode and the Cost of a Token
Why reading a prompt is compute-bound and writing an answer is memory-bound: FLOPs per byte, the roofline and the ridge point, time to first token and time per output token, batching, what long contexts do to it, fp8 and int4 weights, when a model needs more GPUs, and how training differs.
An interactive AI Infrastructure lesson: 18 steps, about 30 minutes, on a live simulation in your browser.
An assistant on one H100 runs Llama 3.1 8B in bf16. A user sends 2,000 tokens of context and gets 500 tokens back. The stage times it with the same model the serving lessons use.
Prefill reads all 2,000 prompt tokens in one pass: 55.9 ms until the first token appears, the time to first token (TTFT). Then decode writes 500 tokens one pass at a time, 6.1 ms each, the time per output token (TPOT). The whole answer takes 3.11 s, and 98% of it is decode.
What you will learn
Two phases
- One request, two very different halves: Every request has a prefill (the whole prompt in one pass, which sets TTFT) and a decode (one pass per output token, which sets TPOT); for most answers decode is most of the time.
FLOPs per byte
- FLOPs per byte: Arithmetic intensity I = FLOPs / (bytes moved); above the GPU's ridge point, (peak FLOP/s) / bandwidth (295 on an H100), work is compute-bound, below it memory-bound.
- Drill: the decode speed limit: At batch 1, TPOT ≥ (weight bytes) / (memory bandwidth); real kernels reach about 80% of the bandwidth, so 4.8 ms becomes about 6 ms.
- A 20-token prompt: Every forward pass costs at least one full read of the weights; a pass only becomes compute-bound when it carries enough tokens to cross the ridge (about 300 on an H100 in bf16).
Batching
- Eight conversations, one read: Batching shares one read of the weights across many sequences: throughput rises almost linearly while decode stays memory-bound, at nearly the same latency per token.
- Sixty-four at once: Up to the ridge point, adding sequences to a decode batch costs little latency and multiplies throughput; it is the cheapest performance in LLM serving.
- Break it: batch the long requests: Each sequence in a batch adds its own KV cache to every decode step's reads and to memory; long contexts cap the batch size and keep decode memory-bound.
Fewer bytes per weight
- fp8: half the bytes: Fewer bytes per weight speeds up memory-bound decode in proportion; prefill speeds up only if the maths itself runs in the lower precision.
- int4 weights, bf16 maths: Weight-only quantisation (int4 weights, bf16 maths) speeds up memory-bound decode and does nothing for compute-bound prefill.
More than one GPU
- Break it: 70B on one GPU: Weights must fit before anything else: parameters × bytes per parameter ÷ GPUs must be well under each GPU's memory.
- Fitting is not enough: A serving plan must fit weights + KV cache + runtime overhead; loading successfully is not the same as being able to serve.
- Four GPUs, then eight: Tensor parallelism splits every layer across GPUs, so each decode step reads 1/N of the weights per GPU; communication after each layer keeps the speed-up below N.
- Drill: weights per GPU: Weights per GPU = parameters × bytes ÷ tensor-parallel size; the rest of the GPU is the KV cache.
What users feel
- A 16,000-token prompt: Prompt length drives TTFT (linearly for the matmuls, quadratically for attention); output length drives total time through TPOT.
- Turning milliseconds into an SLO: Interactive SLOs are TTFT and TPOT at p99; offline work is tokens per second per GPU; batching trades the second for the first.
- Training is the other workload: Inference: 2 × params FLOPs per token, mostly memory-bound, 2 bytes per param. Training: 6 × params FLOPs per token, compute-bound, ~16 bytes per param plus activations.
Recap & playground
- Cheat sheet
- Playground