The model turns each token into a query, key, and value vector (Q, K, and V). Query-key scores set the attention weight for each other token.
This example uses random projections. It shows the calculation but not weights learned by a model.
Input Sequence
Head Dimension (d_head)
Hover over a row in any matrix to see which tokens it attends to.
Q, K, V Projections
Click a preset to change the settings.
Hover over a query row and follow it across all four matrices. How do its scores become softmax weights?
Compare a Scores row with its Softmax row. Softmax increases the differences between the resulting weights.
Try a longer sequence. The score matrix grows as S×S, one row and one column per token.
Change the head dimension. How does it affect the score magnitudes?
Why divide by sqrt(d_head) before softmax? What would happen without the scaling?
In causal (decoder) attention, the upper triangle of the score matrix is masked. Why? (See it live on Attention Patterns.)
Multi-head attention runs several heads in parallel. Why might different heads learn different patterns?
Attention Patterns restricts the query-key grid with causal, windowed, and document masks.