Model Overview

The stack contains every transformer layer. Prefill processes the prompt tokens together. Decode processes one new token at a time. Start the animation, then select a layer to inspect its operations.

Model

Hardware

Quantization

Request

Batch Size 1 Sequence Length 1 (decode)

Inference Phase

Speed 5

Summary

Data Movement During Inference

Transformer Layer Stack

Model Totals (per decode token, batch=1)

Click a preset to change the settings.

Load GPT-2, which has only 12 layers. Now switch to 405B to compare their scale.
Select any transformer layer to inspect its operations.
Set 70B to INT4 and watch weight memory drop by 4×.
Run prefill, then switch to decode. The same stack moves data differently.
Compare the share of FLOPs used by attention and the feed-forward network (FFN) across model sizes.

How does the attention and FFN share of FLOPs change between small and large models?
Why does weight memory scale linearly with parameters but FLOPs scale differently?
Why does decode keep reading the model weights for one token while the KV cache grows one step at a time?

Layer Detail shows the cost of each operation.