Syllabus Lesson 144 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

Self-attention: every position asks every other

Why it matters In a transformer there is no separate encoder to look back at. Every position of the text attends to the others, so each character's vector gets updated with information from the rest of the sentence.

Start with X, one row per position (t rows, d columns). Three learned matrices make every position's query, key and value at once:

Q = X @ Wq     K = X @ Wk     V = X @ Wv
A = softmax(Q @ K.T / sqrt(dk))      # t by t: row i is how much position i attends to each position
out = A @ V

This is the soft lookup of lesson 1, done for all t queries at once with two matrix multiplies, which is why it runs so well on fast maths routines.

Words and symbols in this lesson
query, key, value
what a position looks for; what each position offers to match on; what it hands over
attention
softmax of query-key scores, used to average the values
self-attention
every position attends to the positions of the same text
Your turn

Write self_attention(X, Wq, Wk, Wv) returning (out, A).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.