Syllabus Lesson 57 of 246 · Module 4: Your own autograd engine
Module 4: Your own autograd engine

Backward when shapes stretch (broadcasting)

Why it matters This is the most common autograd bug in the world. A bias of shape (4,) added to a batch of shape (32, 4) is used 32 times, so its gradient must add up all 32 contributions.

Tiny numbers

out = X + b with X (3, 2) and b (2,). The gradient arriving is (3, 2). b was copied to all 3 rows, so each copy received a share: b's gradient is the sum over the rows, shape (2,). If you hand back the (3, 2) array, shapes no longer match, or worse, broadcasting hides it and the numbers are 3 times wrong.

The rule: wherever broadcasting added a direction, sum it away; wherever it stretched a size-1 direction, sum it but keep it as size 1.

Recap Copied forward means summed backward.

Words and symbols in this lesson
autograd
automatic gradients: you write the forward pass, the engine does the backward
Tensor
here: an array plus its gradient plus the Tensors it came from
.grad
the gradient of the final loss with respect to this Tensor
unbroadcast
summing a gradient back down to the shape it was stretched from
Your turn

Write unbroadcast(grad, shape): sum grad down to shape. Your add and multiply already call it.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.