What a model learned about letters
Why it matters Nobody tells a language model that a and e are both vowels. If its embeddings put them close together anyway, the model discovered it from text alone. This is the first glimpse of "meaning" learned from data.
To compare two embeddings, use cosine similarity: the dot product divided by both lengths. It is 1 when two vectors point the same way, 0 when they are unrelated (at right angles), and minus 1 when they point opposite ways. It ignores length, which mostly reflects how often a symbol appears.
You will look inside the course's recorded tiny GPT (Module 13 builds it). It read only this course's lesson text, one character at a time, with each character as a row of 64 numbers.
Predict first: which group of character pairs will be the most similar on average: two vowels, a vowel and a consonant, or two digits?
Words and symbols in this lesson
- embedding
- the row of learned numbers that stands for one symbol
- cosine similarity
- dot product over both lengths: 1 same direction, 0 unrelated, -1 opposite
Set my_prediction to "vowels", "mixed" or "digits". Write cosine(a, b) and nearest(ch, k) (the k other characters with the highest cosine to ch, best first). Then fill group with the average cosine over all pairs of different characters within each group.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.