Syllabus Lesson 58 of 246 · Module 4: Your own autograd engine
Module 4: Your own autograd engine

Sums, means, reshapes and transposes

Why it matters A loss is a sum or a mean, and layers reshape and transpose all the time. Each needs a backward rule, and each rule is a one-liner.

  • sum (to one number): every entry contributed with slope 1, so each gets the arriving gradient. Gradient: ones of the input's shape, times out.grad.
  • mean: a sum times 1/size. Build it from sum and multiply, so it needs no rule of its own.
  • reshape: values are only rearranged, so reshape the gradient back to the input's shape.
  • transpose: transpose the gradient back.

Recap Moving numbers around moves the gradient back the same way.

Words and symbols in this lesson
Tensor
here: an array plus its gradient plus the Tensors it came from
.grad
the gradient of the final loss with respect to this Tensor
unbroadcast
summing a gradient back down to the shape it was stretched from
Your turn

Write sum, mean, reshape and the T property with their backward rules.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.