Syllabus Lesson 146 of 246 · Module 13: Attention and transformers
Module 13: Attention and transformers

Backward through attention

Why it matters Every library computes this for you, but it is only four steps you already know. Deriving it once is what makes attention a layer you understand, not a box you call.

Forward, for one sequence X (t positions by d): Q = X Wq, K = X Wk, V = X Wv, S = Q KT / sqrt(d), A = softmax(S) row by row, out = A V. Backward goes through the same steps in reverse, given G = dL/d(out):

  1. out = A V is a matrix multiply (Module 2's shape rule): dA = G VT and dV = AT G.
  2. A = softmax(S), row by row (Module 2's softmax gradient): dS = A * (dA - (dA * A).sum(-1, keepdims=True)).
  3. S = Q KT / sqrt(d): dQ = dS K / sqrt(d) and dK = dST Q / sqrt(d).
  4. Q = X Wq (and the same for K and V): dWq = XT dQ, dWk = XT dK, dWv = XT dV.

Here d is the width of the queries and keys (Wq's number of columns). No new rule appears anywhere: attention's backward pass is two matrix multiplies and a softmax, run backward.

Recap Undo the steps in reverse order, each with the rule you already have.

Words and symbols in this lesson
query, key, value
what a position looks for; what each position offers to match on; what it hands over
attention
softmax of query-key scores, used to average the values
Your turn

Write attention_backward(X, Wq, Wk, Wv, G) returning (dWq, dWk, dWv) for out = softmax(Q KT / sqrt(d)) V with no mask. The hidden test checks every entry against nudging.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.