An embedding is a lookup, and a lookup is a multiply
Why it matters A network only does arithmetic, but text is made of symbols. Every language model starts by turning each symbol into a short list of numbers, its embedding.
Give every symbol a number from 0 to V minus 1 (its id). The embedding table E has one row per symbol, d numbers each. Turning an id into numbers is simply reading its row: E[id].
Why that is still a layer
Write the id as a one-hot vector: V zeros with a single 1 at position id. Then one-hot times E picks out exactly that row. So an embedding is a linear layer whose input happens to be one-hot. We do the lookup instead because it skips V times d multiplications by zero.
It also tells you how it trains: the gradient only reaches the rows that were used. A symbol that never appears in a batch keeps its row unchanged.
Tiny numbers
V = 3, d = 2, E = [[1, 2], [3, 4], [5, 6]]. id 1 as one-hot is [0, 1, 0], and [0, 1, 0] times E = [3, 4], the same as E[1].
Words and symbols in this lesson
- embedding
- the row of learned numbers that stands for one symbol
- one-hot
- V zeros with a single 1; times the table, it picks one row
Write onehot(ids, V) returning an array of shape (len(ids), V). Then set rows_with_gradient to the number of rows of the embedding table that get a non-zero gradient after the backward pass shown.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.