How it works

Making it fast

Tokenizer

See how text is broken into tokens — the atomic units that LLMs process. Each token maps to an integer ID in the model's vocabulary.

Input Text

Token Stats

Legend

Word start

Continuation

Punctuation

Whitespace

Tokenized Text

Click a preset to load an interesting configuration.

Type a long English sentence and count tokens. Now type the same idea in Chinese — how does token count compare?
Try Python code vs natural language for the same algorithm. Which uses more tokens?
Notice that spaces often attach to the start of the next word (·The vs The). Why would BPE do this?

Why do rare words get split into more tokens than common words?
If a model has vocab_size=50,257 and d_model=768, how large is the embedding table in bytes (FP16)?

Now that you see tokens, see how they become vectors → Embeddings