Syllabus Lesson 55 of 246 · Module 4: Your own autograd engine
Module 4: Your own autograd engine

Backward for add and multiply

Why it matters With these two rules and the ordering, your engine can already differentiate any formula made of sums and products.

Each operation attaches a small function, _backward, to the Tensor it makes. When called, it reads out.grad and adds its share to each parent's grad:

  • For out = a + b: a gets out.grad, b gets out.grad.
  • For out = a * b: a gets out.grad * b.data, b gets out.grad * a.data.

Always add to the parent's grad (+= style), never overwrite: a node used twice must collect both contributions. Then backward() sets the final node's grad to 1 and calls every _backward in reverse topological order.

(The provided unbroadcast helper, imported at the top, handles shapes that were stretched by broadcasting. You will write your own version in two lessons; for now, wrap each contribution in it.)

Words and symbols in this lesson
Tensor
here: an array plus its gradient plus the Tensors it came from
topological order
every node listed after the nodes it depends on
.grad
the gradient of the final loss with respect to this Tensor
unbroadcast
summing a gradient back down to the shape it was stretched from
Your turn

Fill in the two _backward functions and the backward method. The hidden test gradient-checks formulas mixing + and *, including a node used twice.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.