A tensor is a multidimensional array of numbers. Follow one tensor through a transformer layer. The feed-forward network (FFN) uses the widest matrices and usually accounts for most FLOPs.
Model
Tensor Flow Through One Layer
FLOPs Summary
Click a preset to change the settings.
Increase the sequence length. Which operations grow fastest? (QKT and Score×V scale as S².)
Compare the FFN Up dimensions [d, 4d] with the Q Projection dimensions [d, d]. The wider FFN matrix explains its share of FLOPs.
Switch between GPT-2 and LLaMA 70B. How do the relative proportions change?
Why does FFN typically account for ~65% of FLOPs? Look at the weight matrix dimensions.
At what sequence length do attention FLOPs, which scale as S², overtake FFN FLOPs, which scale as S?