Gradients of whole matrices (the shape rule)
Why it matters Layers work on whole matrices, so their gradients are matrices too. One rule and one trick derive almost every layer's backward pass without memorising anything.
The shape rule
The loss L is one number. The gradient of L with respect to any array has exactly the shape of that array: one slope per entry. So if W is (3, 4), dL/dW is (3, 4).
Tiny numbers: Y = X @ W
Say X is (2, 3), W is (3, 4), Y = X @ W is (2, 4), and some later steps turn Y into L. Backward hands you G = dL/dY, shape (2, 4). Each entry Y[i, j] is a sum over k of X[i, k] W[k, j]. Nudge W[k, j]: it moves every Y[i, j] in column j by X[i, k] times the nudge. So dL/dW[k, j] = sum over i of X[i, k] G[i, j]. That is one entry of X.T @ G, which is (3, 4): the shape of W, as the rule says.
The same reasoning for X gives dL/dX = G @ W.T, shape (2, 3). A quick way to remember both: put G with the other matrix, transposed, in the only order whose shapes fit.
The Jacobian, and why nobody builds it
The full table of how every output entry moves with every input entry is the Jacobian. For Y = X @ W with W of size 1000 by 1000 and 64 rows, that table would have 64 thousand outputs times a million weights: 64 billion entries. Backward never builds it: it only ever needs G times the Jacobian, which is the small product X.T @ G. That product is the vector-Jacobian product from the last lesson.
Jacobian: every output's slope with respect to every input, as one big table. Vector-Jacobian product: the gradient pushed back through one step, computed directly without the table.
Recap Same shape as the thing. For Y = X @ W: dW = X.T @ G and dX = G @ W.T.
Words and symbols in this lesson
- vector
- a list of numbers, one per feature
- derivative (df/dx)
- how fast the output changes per unit nudge of the input
Write matmul_grads(X, W, G) returning (dX, dW) for Y = X @ W, and elementwise_grads(A, B, G) returning (dA, dB) for Y = A * B (same shapes, item by item). The hidden test checks both against nudging, on shapes you have not seen.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.