Syllabus Lesson 134 of 246 · Module 12: Words as numbers (embeddings)
Module 12: Words as numbers (embeddings)

An embedding is a lookup, and a lookup is a multiply

Why it matters A network only does arithmetic, but text is made of symbols. Every language model starts by turning each symbol into a short list of numbers, its embedding.

Give every symbol a number from 0 to V minus 1 (its id). The embedding table E has one row per symbol, d numbers each. Turning an id into numbers is simply reading its row: E[id].

Why that is still a layer

Write the id as a one-hot vector: V zeros with a single 1 at position id. Then one-hot times E picks out exactly that row. So an embedding is a linear layer whose input happens to be one-hot. We do the lookup instead because it skips V times d multiplications by zero.

It also tells you how it trains: the gradient only reaches the rows that were used. A symbol that never appears in a batch keeps its row unchanged.

Tiny numbers

V = 3, d = 2, E = [[1, 2], [3, 4], [5, 6]]. id 1 as one-hot is [0, 1, 0], and [0, 1, 0] times E = [3, 4], the same as E[1].

Words and symbols in this lesson
embedding
the row of learned numbers that stands for one symbol
one-hot
V zeros with a single 1; times the table, it picks one row
Your turn

Write onehot(ids, V) returning an array of shape (len(ids), V). Then set rows_with_gradient to the number of rows of the embedding table that get a non-zero gradient after the backward pass shown.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.