LLM Inference Costs
Follow language-model inference from tokens to tensors, then connect the operations to runtime and cost.
From tokens to tensors
A prompt becomes token IDs, vectors, attention scores, and finally a stream of generated tokens.
Overview 1
Model Overview
All transformer layers appear in one stack.
Click any layer to inspect its operations.
Overview 2
Layer Detail
Inspect every operation in a transformer layer:
which are GEMMs, where the FLOPs go, and which are compute-bound vs.
memory-bound.
Topic 1
Tokenizer
Type text and watch it split into the tokens an LLM processes.
Topic 2
Embeddings
Token IDs become vectors through table lookup. Cosine similarity compares their directions.
Topic 3
Positional Encoding
Add position information to embeddings. Without it, word order is invisible.
Topic 4
Attention
Step through Q/K/V projections, attention scores, and softmax.
Topic 5
Attention Patterns
Causal, sliding-window, prefix, and document masks appear as query-key grids linked to KV-cache reuse.
Topic 6
Decoding
Temperature, top-k, and top-p reshape the distribution that selects the next token.
Topic 7
Generation
Tokens appear one at a time while the KV cache grows with each step.
Topic 8
Tensor Shapes
Follow a tensor through one transformer layer and compare the operation widths that make FFNs dominate FLOPs.
Inference performance
Memory traffic and GPU use determine response time and cost.
Scale
Powers of Ten
Zoom one forward pass from the whole model down to a
single GPU tile. Each level's FLOPs and bytes are built from the level
below, so you can see exactly where the sums stop adding up — and why
that gap is the memory traffic.
Method 1
GEMM & Tiling
Watch how tiling divides matrices into blocks,
maps them to GPU SMs, and reduces memory traffic by reusing data in
fast memory.
Method 2
Flash Attention
Compare standard and Flash Attention side by side. The math is the same, but Flash uses far less peak memory.
Method 3
KV Cache & Memory
Explore how KV cache memory scales with sequence
length, batch size, and attention variants (MHA, GQA, MQA).
A fit table compares models across GPUs.
Method 4
Batching Simulator
Watch the estimated use of GPU arithmetic increase
as the batch grows. Compare static and continuous batching.
Method 5
Speculative Decoding
A small draft model proposes tokens, a large model
verifies them in one pass. See when this speeds up generation and
when it does not.
Method 6
Inference Cost Estimator
Estimate prefill speed, decode speed, memory
requirements, and cost per million tokens for a selected model and
GPU.
A prompt has two inference phases
A prompt passes through the same sequence of events in any autoregressive LLM:
- Tokenize your input into a sequence of integer
token IDs.
- Prefill: feed all input tokens through the model's
transformer layers in parallel. This produces the internal
representation and the first output token.
- Decode: generate output tokens one at a time, each
requiring a full pass through every layer.
- Repeat step 3 until the model produces a stop token or hits the
length limit.
Prefill phase
Processes all input tokens at once.
Compute-bound. Arithmetic speed limits this phase
because all prompt tokens are available at once.
Decode phase
Generates one token per step.
Memory-bound. Memory bandwidth limits this phase.
The GPU loads all model weights for each new token.
A GPU can perform arithmetic faster than memory can supply data. Many
inference methods reduce memory transfers or reuse transferred data.
Matrix multiplication accounts for most work
Inside each transformer layer, almost all the work is matrix
multiplications (GEMMs). There are two main groups:
Attention
- Q, K, V projections: three weight matrices multiply
the input to produce queries, keys, and values.
- QKT: queries times keys (transposed) to
compute attention scores. This is the one that scales with sequence
length squared.
- Score × V: attention-weighted sum of values.
- Output projection: one more weight matrix.
Feed-Forward Network (FFN)
- Up projection: expands from hidden size to ~4×
hidden size.
- Gate projection: in models like LLaMA, a parallel
expansion for gated activation.
- Down projection: contracts back to hidden size.
For an M×K matrix times a K×N matrix, the compute cost is:
FLOPs = 2 × M × K × N
A model like LLaMA 7B has hidden=4096 and
FFN dim=11008. That single FFN Up projection is a
4096×11008 matrix. It performs about 45 million multiply-accumulates per token
and per layer, repeated across 32 layers.
The FFN layers typically account for ~65% of all FLOPs. The attention
projections add ~30%. The actual attention matmuls (QKT,
Score×V) are only ~5% until sequence length gets very long.
Compute work and memory traffic
Every operation has two costs:
- FLOPs: how many floating-point operations it
requires (compute).
- Bytes: how much data must be moved to/from memory
(traffic).
The ratio between them is arithmetic intensity:
Arithmetic Intensity = FLOPs / Bytes (FLOP/byte)
Every GPU has a ridge point, the arithmetic intensity
where compute speed and memory bandwidth are in balance. Below the ridge,
the operation is memory-bound
(waiting for data). Above it, the operation is
compute-bound (GPU is fully
utilized).
For example, the A100 has 312 TFLOP/s compute and ~1.7 TB/s effective
bandwidth, giving a ridge point of about 180 FLOP/byte.
Below 180 FLOP/byte, memory bandwidth limits the operation before compute
speed does.
Decode at batch=1 has an arithmetic intensity of about 1 FLOP/byte. It
uses only a small fraction of peak arithmetic speed. Batching increases
this ratio, while quantization reduces the bytes transferred.
Useful inference estimates
- FLOPs per token ≈
2 × num_parameters
(forward pass).
- Weight memory ≈
num_parameters ×
bytes_per_param (FP16 = 2 bytes, INT8 = 1, INT4 = 0.5).
- Decode throughput ≈
memory_bandwidth /
weight_memory tokens/sec (memory-bound).
- Prefill throughput ≈
GPU_TFLOPS / (2 ×
num_params) tokens/sec (compute-bound).
- Arithmetic intensity of decode ≈
batch_size FLOP/byte for FP16 (in general 2 ×
batch_size / bytes_per_param). This is why batching
matters.
The Cost Estimator compares these estimates with
a more detailed performance model.