How it works

Making it fast

Embeddings

Each token ID maps to a learned vector — the embedding. This is a table lookup, not a computation.

Model

Input Tokens

Selected Token

Click a token row to inspect it.

Embedding Table Size

Embedding Heatmap (rows = tokens, cols = first 64 dims of d_model)

← embedding dimensions →

Cosine Similarity Matrix

Click a preset to load an interesting configuration.

Click different tokens and compare their heatmap rows — do similar words have similar patterns?
Look at the cosine similarity matrix — which token pairs are most similar?
Change the model to see how d_model affects the embedding width.

The embedding table has vocab_size × d_model parameters. For GPT-2 (50K × 768), how many MB is that in FP16?
Why is embedding a lookup and not a matrix multiplication? (Hint: the input is a one-hot vector)

Embeddings need position information — see how it's added → Positional Encoding