Many heads, same cost
Why it matters One attention pattern per layer is limiting: a position might want to look at the previous character and, at the same time, at the start of the word. Multi-head attention runs several smaller attentions side by side.
See it
Play the animation. Split the d columns of Q, K and V into H groups of d / H. Each group (a head) does its own attention with its own pattern. Glue the H outputs back side by side into d columns, then mix them with one more matrix Wo.
The cost: the same matrices as one big head, just cut into pieces, so the parameter count does not change. Each head is narrower, so it divides by the square root of d / H.
Words and symbols in this lesson
- attention
- softmax of query-key scores, used to average the values
- head
- one of H narrower attentions run side by side, each with its own pattern
Write multi_head(X, Wq, Wk, Wv, Wo, H) (no mask), returning the output and a list of H attention matrices.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.