Autoregressive generation produces one token at a time. Each token depends on the tokens before it. Prefill processes all prompt tokens together. Decode then produces one new token per pass through the model.
Model
Hardware
Prompt
Controls
Phase
Idle
Token Sequence
KV Cache
Timing
Click a preset to change the settings.
Watch the prefill phase process all prompt tokens at once. Decode then proceeds one token at a time. Why the difference?
Watch the KV cache grow. Each new token adds one column across all layers, and every decode step re-reads the cache. The KV read/step number tracks this growth.
Try a longer prompt. Prefill FLOPs grow, but the time barely moves: one pass through the weights is the floor until the prompt reaches hundreds of tokens.
Each decode step loads all model weights from memory. At batch=1, what fraction of the GPU's arithmetic capacity is used?
Why can the model not generate all output tokens in parallel as it processes input tokens?
Each generation step passes through every model layer. Follow its shapes in Tensor Shapes.