Syllabus Lesson 141 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

Attention is a soft lookup

Why it matters Attention is the one new idea in a transformer. Everything else you already know. It is a dictionary lookup that, instead of picking one entry, blends all of them.

See it

Play the animation. A Python dictionary matches a key exactly and returns one value. Attention has a query q, a list of keys and a list of values. It scores the query against every key with a dot product, turns the scores into weights with softmax (positive, adding to 1), and returns the weighted average of the values.

weights = softmax(K @ q)
output  = weights @ V

Tiny numbers

q = [1, 0]. Keys [2, 0], [0, 2], [0, 0] give scores 2, 0, 0. Softmax: e^2 = 7.39 and 1 and 1, so the weights are 0.79, 0.11, 0.11. With values 10, 20, 30 the output is 0.79 times 10 + 0.11 times 20 + 0.11 times 30 = 13.2: mostly the first value, a little of the others.

Query, key, value: what I am looking for; what each item offers to be matched on; what each item hands over if matched.

Words and symbols in this lesson
query, key, value
what a position looks for; what each position offers to match on; what it hands over
attention
softmax of query-key scores, used to average the values
Your turn

Write attend(q, K, V) for one query: return (weights, output).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.