Attention is a soft lookup
Why it matters Attention is the one new idea in a transformer. Everything else you already know. It is a dictionary lookup that, instead of picking one entry, blends all of them.
See it
Play the animation. A Python dictionary matches a key exactly and returns one value. Attention has a query q, a list of keys and a list of values. It scores the query against every key with a dot product, turns the scores into weights with softmax (positive, adding to 1), and returns the weighted average of the values.
weights = softmax(K @ q)
output = weights @ V
Tiny numbers
q = [1, 0]. Keys [2, 0], [0, 2], [0, 0] give scores 2, 0, 0. Softmax: e^2 = 7.39 and 1 and 1, so the weights are 0.79, 0.11, 0.11. With values 10, 20, 30 the output is 0.79 times 10 + 0.11 times 20 + 0.11 times 30 = 13.2: mostly the first value, a little of the others.
Query, key, value: what I am looking for; what each item offers to be matched on; what each item hands over if matched.
Words and symbols in this lesson
- query, key, value
- what a position looks for; what each position offers to match on; what it hands over
- attention
- softmax of query-key scores, used to average the values
Write attend(q, K, V) for one query: return (weights, output).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.