Backward for matrix multiply
Why it matters Matrix multiply is the workhorse of every layer. Its backward rule is two lines, and you can derive it from shapes alone.
Tiny numbers
Take out = A @ B with A of shape (2, 3) and B of shape (3, 4), so out is (2, 4). The gradient arriving, G, has out's shape (2, 4). A's gradient must have A's shape (2, 3). The only way to make (2, 3) from G (2, 4) and B (3, 4) is G @ B.T. Likewise B's gradient (3, 4) must be A.T @ G.
Check one entry in your head: out[0, 0] = A[0, 0] B[0, 0] + A[0, 1] B[1, 0] + A[0, 2] B[2, 0]. Nudge A[0, 1] and out[0, 0] moves by B[1, 0] times the nudge. That is exactly entry (0, 1) of G @ B.T when G is 1 at (0, 0) and 0 elsewhere.
Recap dA = G @ B.T, dB = A.T @ G. If you forget, match the shapes.
Words and symbols in this lesson
- Tensor
- here: an array plus its gradient plus the Tensors it came from
- .grad
- the gradient of the final loss with respect to this Tensor
- unbroadcast
- summing a gradient back down to the shape it was stretched from
Write __matmul__ with its _backward. The hidden test gradient-checks (A @ B) * C for random shapes.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.