Each token ID maps to a learned vector called an embedding.
The model retrieves this vector from an embedding table.
These example vectors use hand-picked word groups. They imitate broad clustering patterns but do not come from a trained model.
Model
Input Tokens
Selected Token
Click a token row to inspect it.
Embedding Table Size
Embedding Heatmap (rows = tokens, columns = first 64 dimensions of d_model)
← embedding dimensions →
Cosine Similarity Matrix
Click a preset to change the settings.
Hover over the heatmap to inspect individual dimensions. Related words have similar patterns.
Try "cat dog bird fish." The animals cluster with high cosine similarity.
Compare "the red cat and blue dog." Function words (the, and) cluster separately from content words.
Hover the similarity matrix to see exact similarity values between token pairs.
The embedding table has vocab_size × d_model parameters. For GPT-2 (50K × 768), how many MB is that in FP16?
Why is embedding a lookup and not a matrix multiplication? (Hint: the input is a one-hot vector)