During decode, a batch of one loads all model weights for one token. This uses little of the GPU's arithmetic capacity. A larger batch shares each weight load across more tokens. The FLOPs-per-byte ratio, called arithmetic intensity, increases.
Model
Hardware
Request
Quantization
GPU Arithmetic Use
Throughput & Latency
Arithmetic Intensity vs Batch Size
Static vs Continuous Batching
Memory Budget
Click a preset to change the settings.
Start at batch=1. The GPU-use gauge shows about 1%. Drag the batch slider up to 128.
Switch to INT4 and watch the arithmetic-intensity curve shift up. The FLOPs stay fixed while the bytes fall to one quarter.
Find the batch size that crosses the ridge point. The color changes from orange to green.
Watch throughput and latency as you increase the batch. When does the tradeoff stop being worthwhile?
Switching from FP16 to INT4 shifts the arithmetic-intensity curve up. Why does shrinking the weights make it easier to become compute-bound? (Hint: the FLOPs do not change.)
At batch=1, the A100 GPU-use gauge shows about 1%. What does this mean for the cost of serving one request at a time?