BPE Tokenizer Visualization | Byte-Pair Encoding Step by Step

The idea: glue the most common pair together, again and again

A language model reads tokens, not letters or whole words. Letters make texts very long. Whole words need a huge vocabulary and still miss new words. Byte-pair encoding (BPE) sits in between. It starts from single characters and learns which pieces are common enough to deserve a token of their own.

The BPE training loop: split words into characters plus an end marker, count adjacent pairs weighted by word count, merge the most frequent pair into a new token, record the rule and add the token to the vocabulary, then repeat until the vocabulary is big enough. — Training is one loop: count pairs, merge the winner, write down the rule, repeat.

The canvas uses the classic example corpus: low ×5, lower ×2, newest ×6 and widest ×3. Every word ends with the marker _ (written </w> in the original paper), so the tokenizer can tell est at the end of a word from est in the middle.

Training: one merge at a time

Count every pair of neighbouring tokens. A pair in newest counts 6 times, because the word occurs 6 times.
Pick the pair with the highest count.
Merge: replace that pair everywhere in the corpus by one new token.
Record the merge rule, for example e + s → es, and add the new token to the vocabulary.

On this corpus the first merges are e + s (9), es + t (9) and est + _ (9): the ending est_ is shared by newest and widest. Each merge makes the corpus shorter: 95 → 86 → 77 → 68 tokens. (Demo 1: the first three merges)

Ties need a fixed rule

Two pairs often have the same count. In merge 4, l + o and o + w both have 7. Any fixed rule works, but it must be the same every time, or two training runs on the same data give different tokenizers. This page takes the pair it saw first when reading the corpus from the top. Libraries use their own rules, such as the smallest pair in sort order. (Demo 2: a tie)

The vocabulary grows by one per merge

The vocabulary starts with the 11 symbols that appear in the corpus. Each merge adds exactly one token. This corpus allows 15 merges. After that every training word is a single token and no pair is left. The vocabulary then has 26 tokens. (Demo 3: the vocabulary grows to 26)

Merges	Vocabulary	Corpus length	Last rule
0	11	95 tokens	(characters only)
3	14	68 tokens	`est + _ → est_`
8	19	36 tokens	`new + est_ → newest_`
15	26	16 tokens	`lower + _ → lower_`

Encoding a word: replay the rules in order

To tokenize new text, split each word into characters plus _. Then go through the merge rules in the order they were learned and apply each one wherever it fits. The order matters: rule 2 (es + t) can only fire after rule 1 has made es. The resulting tokens are looked up in the vocabulary, and the model gets their ids.

A training word such as newest comes out as one token, newest_ (Demo 4: encode a seen word). A word the tokenizer never saw is cut into pieces it knows: lowest becomes low + est_, and newer becomes new + e + r + _. (Demo 5: encode an unseen word)

Encoding lowest: l o w e s t _ becomes l o w es t _ after rule 1, l o w est _ after rule 2, l o w est_ after rule 3, lo w est_ after rule 4 and low est_ after rule 5. Rules 6 to 15 do not apply. The result is two tokens, low and est_. — An unseen word still gets a short encoding from pieces learned on other words.

This is why BPE never has an "unknown word": in the worst case a word falls back to single characters. The model on the How a Large Language Model Works page uses a tokenizer like this.

Vocabulary size: the trade-off

More merges mean longer tokens, so the same text needs fewer of them. newest costs 7 tokens with no merges, 4 after 4 merges (n e w est_) and 1 after 8. (Demo 6: more merges = fewer tokens)

	Small vocabulary	Large vocabulary
Tokens per text	many (slower, uses more of the context window)	few
Embedding table	small	large (one row per token)
Rare tokens	each token is seen often in training	many tokens are rare and poorly learned

Real tokenizers pick a target size and stop there: GPT-2 learned 50,000 merges on top of 256 byte values (50,257 tokens with one special token). Newer models use 100,000 to 200,000 tokens.

Real tokenizers

Byte-level BPE (GPT-2 and later) starts from the 256 byte values instead of letters, so any text, emoji or language can be encoded.
Pre-tokenization first splits text into words, numbers and punctuation with a regular expression, and keeps the leading space with the word (·the) instead of an end marker.
WordPiece (BERT) picks merges by a likelihood score instead of raw counts. Unigram (SentencePiece) starts big and removes tokens. Both still split rare words into pieces.

See also How a Large Language Model Works for what happens to the token ids next, Attention for how the model mixes tokens, and Huffman coding for another way to give frequent things short codes.

See also Word2Vec: once text is split into tokens, each token gets a learned embedding vector.

What the page leaves out

Bytes instead of letters, the pre-tokenization regex, special tokens such as <|endoftext|>, fast training (real trainers update pair counts incrementally instead of recounting), encoding by merge rank with a priority queue, and dropout variants of BPE. The corpus has 4 words; real tokenizers train on gigabytes of text.