KV Cache & Memory

During generation, attention reuses key and value data from earlier tokens. The KV cache stores this data. Its size grows with the sequence length and can limit the batch size.

Model

Hardware

Request

Batch Size 1 Sequence Length 1024

Attention Variant

Precision

Weight Quantization

KV Cache Precision

Summary

Weights

—

KV Cache

—

Total

—

GPU Memory

—

GPU Memory Budget

KV Cache Formula

Attention Variant Comparison

Does It Fit?

GPU	VRAM	Weights	KV Cache	Total	Fits?	GPUs Needed

KV Cache vs Sequence Length

Click a preset to change the settings.

Load 70B FP16 and check which GPUs can fit it. Now switch to INT4.
Toggle between MHA → GQA → MQA and watch the KV cache size in the formula box change
Set the batch to 64 with LLaMA 7B and watch the memory bar overflow. What is the largest batch that fits?
Toggle KV-cache precision from FP16 to INT8. Does this save more or less memory than weight quantization?

For 100 requests with 4K-token context, which uses more memory: weights or KV cache? Does the answer change at 8K?
Which formula values come from the model? Which can the operator change during inference?

Batching reuses weights across requests to fill the GPU → Batching Simulator