Self-attention: every position asks every other
Why it matters In a transformer there is no separate encoder to look back at. Every position of the text attends to the others, so each character's vector gets updated with information from the rest of the sentence.
Start with X, one row per position (t rows, d columns). Three learned matrices make every position's query, key and value at once:
Q = X @ Wq K = X @ Wk V = X @ Wv
A = softmax(Q @ K.T / sqrt(dk)) # t by t: row i is how much position i attends to each position
out = A @ V
This is the soft lookup of lesson 1, done for all t queries at once with two matrix multiplies, which is why it runs so well on fast maths routines.
Words and symbols in this lesson
- query, key, value
- what a position looks for; what each position offers to match on; what it hands over
- attention
- softmax of query-key scores, used to average the values
- self-attention
- every position attends to the positions of the same text
Write self_attention(X, Wq, Wk, Wv) returning (out, A).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.