Syllabus Lesson 137 of 246 · Module 12: Words as numbers (embeddings)
Module 12: Words as numbers (embeddings)

Meaning from the company a word keeps

Why it matters "You shall know a word by the company it keeps." Count which words appear near which, and the counts alone place king near queen and let you do arithmetic like king minus man plus woman.

To test this honestly we need to know the right answers, so this lesson uses a made-up little world with planted structure: every person is royal or common, he or she, grown-up or young, and each of those shows up as other words in the same sentence ("the queen rules in the palace and she reads"). Real text is far messier, but the mechanism is the same.

From counts to vectors

Count, for every pair of words, how often they appear within a window of w words of each other. Each word's row of counts is its vector. A helper turns raw counts into "how much more often than chance" (PPMI), so that "the" does not dominate everything.

An analogy a : b = c : ? is answered by the word closest to b minus a plus c. There are 72 such questions in this world.

Predict: with a window of 2 words, and with a window covering the whole 9-word sentence, what share will be right?

Words and symbols in this lesson
window
how many words either side count as a word's company
PPMI
how much more often two words meet than chance would give, with negatives set to 0
Your turn

Set my_guess to your two predictions. Write cooccurrence(sents, win): a V by V array of counts, adding 1 to [word, other] for every other word within win positions in the same sentence. Then fill acc for windows 2 and 8 with score(ppmi(...)).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.