Text is split into tokens, the units that an LLM processes.
Each token maps to an integer ID in the model's vocabulary.
The split approximates GPT-2 byte-pair encoding (BPE). The displayed IDs are examples, not entries from the GPT-2 vocabulary.
Input Text
Token Counts
Selected Token
Click a token to inspect it.
Legend
Word start
Continuation
Punctuation
Whitespace
Tokenized Text
Click a preset to change the settings.
Type a long English sentence and count the tokens. Now type the same idea in Chinese. How does the token count compare?
Try Python code and natural language for the same algorithm. Which uses more tokens?
Spaces often attach to the start of the next word (·The and The). Why would BPE do this?
Why do rare words get split into more tokens than common words?
If a model has vocab_size=50,257 and d_model=768, how large is the embedding table in bytes (FP16)?