Syllabus Lesson 147 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

Many heads, same cost

Why it matters One attention pattern per layer is limiting: a position might want to look at the previous character and, at the same time, at the start of the word. Multi-head attention runs several smaller attentions side by side.

See it

Play the animation. Split the d columns of Q, K and V into H groups of d / H. Each group (a head) does its own attention with its own pattern. Glue the H outputs back side by side into d columns, then mix them with one more matrix Wo.

The cost: the same matrices as one big head, just cut into pieces, so the parameter count does not change. Each head is narrower, so it divides by the square root of d / H.

Words and symbols in this lesson
attention
softmax of query-key scores, used to average the values
head
one of H narrower attentions run side by side, each with its own pattern
Your turn

Write multi_head(X, Wq, Wk, Wv, Wo, H) (no mask), returning the output and a list of H attention matrices.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.