Tokens: merging the commonest pairs
Why it matters Real models do not read characters or whole words. They read tokens, pieces in between, made by a simple counting rule. It is why a model "sees" a common word as one piece and a rare one as several.
Byte pair encoding (BPE) starts with single characters and repeats one step: find the pair of neighbouring symbols that appears most often in the text, and glue it into one new symbol everywhere.
Tiny example
Words "low" (5 times) and "lot" (2 times). Pairs: l o appears 7 times, o w 5, o t 2. Merge l o: now "lo w" and "lo t". Next, lo w appears 5 times: merge it, and "low" is a single token.
Real tokenisers add two things this lesson leaves out. They start from bytes rather than characters, so any text in any language, emoji included, can be written with a 256-symbol start. And they mark where words begin (GPT-2 glues the space onto the front of the next word), so a piece at the start of a word and the same letters inside a word become different tokens. Characters per word is the cost of reading character by character. Tokens per word falls as merges are added: fewer, bigger steps for the model, at the price of a bigger vocabulary.
Words and symbols in this lesson
- token, BPE
- a piece of text the model reads; byte pair encoding glues the commonest neighbouring pair, again and again
Write pair_counts(vocab) (how often each neighbouring pair appears, weighted by word count) and merge(vocab, pair) (glue that pair everywhere). Guess my_guess, the tokens per word after 200 merges (it starts at about 4.6). The loop is written for you.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.