Without position information, attention treats "the cat sat" and "sat the cat" identically. Position encodings tell the model where each token is.
Model
Encoding Type
Sequence Length
Position A
Position B
Dot Product Similarity
—
Nearby positions → higher similarity
Sine/Cosine Wave Construction
Positional Encoding Heatmap
Position Vectors Comparison
Position A
Position B
Click a preset to change the settings.
Toggle sinusoidal and rotary encoding. The dot product does not change. In this pairwise view, both reduce to cos((a−b)·ω); they act in different places (added to embeddings vs. rotating Q/K inside attention).
Drag positions A and B apart. Track how the dot product decays with distance.
Notice the different frequency bands: low dimensions change slowly, high dimensions oscillate fast.
Increase the maximum positions. The high-frequency stripes become denser.
Why use different frequencies for different dimensions? What would happen if all dimensions used the same frequency?
How do sinusoidal and rotary encodings behave when positions exceed the training length?
The dot product of two position vectors encodes relative distance. Why is this useful for attention?
Attention uses these position-aware vectors in query-key scores.
Embeddings constructs the token representations before position is added.