Syllabus Lesson 138 of 246 · Module 12: Words as numbers (embeddings)
Module 12: Words as numbers (embeddings)

Learning word vectors (skip-gram)

Why it matters Counting needs a V by V table, which is impossible for a million words. Word2vec learns short vectors directly by gradient descent instead, and the same idea, learning vectors by predicting context, is inside every language model.

Skip-gram with negative sampling. Each word has two vectors: v (as itself) and u (as someone's context). For a real pair (a word and a word from its sentence), push the score v dot u up; for k random "negative" words, push it down:

loss = -log sigmoid(v . u_pos) - sum over negatives of log sigmoid(-v . u_neg)

The gradients, by the chain rule

With p = sigmoid(v . u_pos) and q_k = sigmoid(v . u_neg_k):

  • dL/dv = (p - 1) u_pos + sum over k of q_k u_neg_k
  • dL/du_pos = (p - 1) v
  • dL/du_neg_k = q_k v

Read them: if the real pair already scores high, p is near 1 and almost nothing changes. Mistakes produce big pushes, exactly like cross-entropy.

Batched, with None

For a batch, p has shape (B,) and u_pos has shape (B, d). To scale each row of u_pos by its own (p - 1), add a direction: (p - 1)[:, None] * u_pos, shape (B, 1) times (B, d). For the negatives, q has shape (B, k) and v has shape (B, d); q[:, :, None] * v[:, None, :] lines them up as (B, k, 1) times (B, 1, d), giving (B, k, d): one scaled copy of v per negative. (The Python module's shapes lesson introduced None.)

Words and symbols in this lesson
PPMI
how much more often two words meet than chance would give, with negatives set to 0
skip-gram, negative sampling
learn vectors by scoring real (word, context) pairs high and random pairs low
Your turn

Write sgns_grads(v, u_pos, u_neg) for a batch: v and u_pos have shape (B, d), u_neg has shape (B, k, d). Return (g_v, g_pos, g_neg) with the same shapes. The training loop uses it; then the 72 analogies must be at least 90% right.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.