Layer Overview

One transformer layer contains attention, a feed-forward network (FFN), and supporting operations. The diagram follows token data through each operation. Select a block to inspect its matrix size, arithmetic work, and memory traffic.

Model

Hardware

GPU

Request

Batch Size 1 Sequence Length 1024

Performance Options

FlashAttention Quantization

Selected Operation

Name

—

Transformer Layer Operation Flow

FLOPs Breakdown (per layer)

Memory Traffic Breakdown (per layer)

All Operations

Operation	Type	M	K	N	FLOPs	Traffic	FLOPs/Byte	Limit	Time (ms)	% FLOPs

Roofline Model

Click a preset to change the settings.

Set sequence to 8192 and watch the attention share of FLOPs increase.
Enable FlashAttention at 4K. What changes on the roofline chart? (Same FLOPs, fewer bytes.)
Select operations in the table. Find which are memory-bound (orange) and compute-bound (green).
Set batch=64 and track the M dimension's effect on arithmetic intensity.

At what sequence length does attention overtake the FFN in FLOPs? What does this imply for long prompts?
Why are attention matrix multiplications memory-bound while projection matrix multiplications are compute-bound? What differs in their dimensions?

GEMM & Tiling maps these matrix multiplications onto GPU tiles.