Learning word vectors (skip-gram)
Why it matters Counting needs a V by V table, which is impossible for a million words. Word2vec learns short vectors directly by gradient descent instead, and the same idea, learning vectors by predicting context, is inside every language model.
Skip-gram with negative sampling. Each word has two vectors: v (as itself) and u (as someone's context). For a real pair (a word and a word from its sentence), push the score v dot u up; for k random "negative" words, push it down:
loss = -log sigmoid(v . u_pos) - sum over negatives of log sigmoid(-v . u_neg)
The gradients, by the chain rule
With p = sigmoid(v . u_pos) and q_k = sigmoid(v . u_neg_k):
- dL/dv = (p - 1) u_pos + sum over k of q_k u_neg_k
- dL/du_pos = (p - 1) v
- dL/du_neg_k = q_k v
Read them: if the real pair already scores high, p is near 1 and almost nothing changes. Mistakes produce big pushes, exactly like cross-entropy.
Batched, with None
For a batch, p has shape (B,) and u_pos has shape (B, d). To scale each row of u_pos by its own (p - 1), add a direction: (p - 1)[:, None] * u_pos, shape (B, 1) times (B, d). For the negatives, q has shape (B, k) and v has shape (B, d); q[:, :, None] * v[:, None, :] lines them up as (B, k, 1) times (B, 1, d), giving (B, k, d): one scaled copy of v per negative. (The Python module's shapes lesson introduced None.)
Words and symbols in this lesson
- PPMI
- how much more often two words meet than chance would give, with negatives set to 0
- skip-gram, negative sampling
- learn vectors by scoring real (word, context) pairs high and random pairs low
Write sgns_grads(v, u_pos, u_neg) for a batch: v and u_pos have shape (B, d), u_neg has shape (B, k, d). Return (g_v, g_pos, g_neg) with the same shapes. The training loop uses it; then the 72 analogies must be at least 90% right.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.