Backward for add and multiply
Why it matters With these two rules and the ordering, your engine can already differentiate any formula made of sums and products.
Each operation attaches a small function, _backward, to the Tensor it makes. When called, it reads out.grad and adds its share to each parent's grad:
- For
out = a + b: a getsout.grad, b getsout.grad. - For
out = a * b: a getsout.grad * b.data, b getsout.grad * a.data.
Always add to the parent's grad (+= style), never overwrite: a node used twice must collect both contributions. Then backward() sets the final node's grad to 1 and calls every _backward in reverse topological order.
(The provided unbroadcast helper, imported at the top, handles shapes that were stretched by broadcasting. You will write your own version in two lessons; for now, wrap each contribution in it.)
Words and symbols in this lesson
- Tensor
- here: an array plus its gradient plus the Tensors it came from
- topological order
- every node listed after the nodes it depends on
- .grad
- the gradient of the final loss with respect to this Tensor
- unbroadcast
- summing a gradient back down to the shape it was stretched from
Fill in the two _backward functions and the backward method. The hidden test gradient-checks formulas mixing + and *, including a node used twice.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.