GEMM is the matrix multiplication used in most transformer operations. Tiling divides a large multiplication into blocks that fit in fast on-chip memory (SRAM). Reusing each block reduces transfers to high-bandwidth memory (HBM). This can increase throughput, which is the number of tokens processed per second.
Hardware
Model
Layer Operation
Request
Tiling
Performance Options
Selected Operation
FLOPs
—
Bytes (tiled)
—
FLOPs/Byte
—
Limit
—
Matrix Dimensions:
Tiling Grid & Wave Mapping
Single Tile Accumulation
Memory Traffic: Untiled vs Tiled
Roofline Model
Matrix Multiplications in One Layer
Operation
M
K
N
FLOPs
Bytes (tiled)
FLOPs/Byte
Limit
Time (ms)
Click a preset to change the settings.
Try 64×64 tiles and watch wave efficiency drop. Now switch to 128×128.
Switch between GPUs (T4 → A100 → H100) with the same configuration. How does the number of waves change?
Look at the memory-traffic bars. What is the reduction factor from naive to tiled?
Increase batch or sequence length until the tiling grid shows several waves.
Does increasing batch size change the arithmetic intensity of an operation? Why or why not?
Why does the best tile size depend on which GPU you are using?