Backward when shapes stretch (broadcasting)
Why it matters This is the most common autograd bug in the world. A bias of shape (4,) added to a batch of shape (32, 4) is used 32 times, so its gradient must add up all 32 contributions.
Tiny numbers
out = X + b with X (3, 2) and b (2,). The gradient arriving is (3, 2). b was copied to all 3 rows, so each copy received a share: b's gradient is the sum over the rows, shape (2,). If you hand back the (3, 2) array, shapes no longer match, or worse, broadcasting hides it and the numbers are 3 times wrong.
The rule: wherever broadcasting added a direction, sum it away; wherever it stretched a size-1 direction, sum it but keep it as size 1.
Recap Copied forward means summed backward.
Words and symbols in this lesson
- autograd
- automatic gradients: you write the forward pass, the engine does the backward
- Tensor
- here: an array plus its gradient plus the Tensors it came from
- .grad
- the gradient of the final loss with respect to this Tensor
- unbroadcast
- summing a gradient back down to the shape it was stretched from
Write unbroadcast(grad, shape): sum grad down to shape. Your add and multiply already call it.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.