Backward through attention
Why it matters Every library computes this for you, but it is only four steps you already know. Deriving it once is what makes attention a layer you understand, not a box you call.
Forward, for one sequence X (t positions by d): Q = X Wq, K = X Wk, V = X Wv, S = Q KT / sqrt(d), A = softmax(S) row by row, out = A V. Backward goes through the same steps in reverse, given G = dL/d(out):
- out = A V is a matrix multiply (Module 2's shape rule): dA = G VT and dV = AT G.
- A = softmax(S), row by row (Module 2's softmax gradient): dS = A * (dA - (dA * A).sum(-1, keepdims=True)).
- S = Q KT / sqrt(d): dQ = dS K / sqrt(d) and dK = dST Q / sqrt(d).
- Q = X Wq (and the same for K and V): dWq = XT dQ, dWk = XT dK, dWv = XT dV.
Here d is the width of the queries and keys (Wq's number of columns). No new rule appears anywhere: attention's backward pass is two matrix multiplies and a softmax, run backward.
Recap Undo the steps in reverse order, each with the rule you already have.
Words and symbols in this lesson
- query, key, value
- what a position looks for; what each position offers to match on; what it hands over
- attention
- softmax of query-key scores, used to average the values
Write attention_backward(X, Wq, Wk, Wv, G) returning (dWq, dWk, dWv) for out = softmax(Q KT / sqrt(d)) V with no mask. The hidden test checks every entry against nudging.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.