Syllabus Lesson 135 of 246 · Module 12: Words as numbers (embeddings)
Module 12: Words as numbers (embeddings)

What a model learned about letters

Why it matters Nobody tells a language model that a and e are both vowels. If its embeddings put them close together anyway, the model discovered it from text alone. This is the first glimpse of "meaning" learned from data.

To compare two embeddings, use cosine similarity: the dot product divided by both lengths. It is 1 when two vectors point the same way, 0 when they are unrelated (at right angles), and minus 1 when they point opposite ways. It ignores length, which mostly reflects how often a symbol appears.

You will look inside the course's recorded tiny GPT (Module 13 builds it). It read only this course's lesson text, one character at a time, with each character as a row of 64 numbers.

Predict first: which group of character pairs will be the most similar on average: two vowels, a vowel and a consonant, or two digits?

Words and symbols in this lesson
embedding
the row of learned numbers that stands for one symbol
cosine similarity
dot product over both lengths: 1 same direction, 0 unrelated, -1 opposite
Your turn

Set my_prediction to "vowels", "mixed" or "digits". Write cosine(a, b) and nearest(ch, k) (the k other characters with the highest cosine to ch, best first). Then fill group with the average cosine over all pairs of different characters within each group.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.