KV Cache & Memory

Model

Batch Size 1 Sequence Length 1024

Weight Quantization

KV Cache Precision

Weights

—

KV Cache

—

Total

—

GPU Memory

—

GPU	VRAM	Weights	KV Cache	Total	Fits?	GPUs Needed

Click a preset to load an interesting configuration.

Load 70B FP16 — check which GPUs can fit it. Now switch to INT4
Toggle between MHA → GQA → MQA and watch the KV cache size in the formula box change
Set batch to 64 with LLaMA 7B — watch the memory bar overflow. What's the max batch that fits?
Toggle KV cache precision from FP16 to INT8 — is it more or less impactful than weight quantization?

For 100 concurrent users with 4K context, is the bottleneck weight memory or KV cache memory? Does the answer change at 8K?
Which factors in the KV cache formula are set by the model architect vs controllable at inference time?

Batching reuses weights across requests to fill the GPU → Batching Simulator