Recording the graph and ordering it
Why it matters Backward must visit each box only after every box that depends on it. Getting this order wrong gives silently wrong gradients.
See it
Picture x used twice: a = x * 2, b = x * 3, L = a + b. This is a diamond. Going backward from L, we must finish with both a and b before handling x, because x's gradient is the sum of what comes back through both.
The fix is to list the nodes in topological order: every node after all of its parents. Backward then walks that list in reverse. A depth-first search makes the list: visit a node's parents first, then add the node, and skip nodes already seen.
Topological order: an ordering of a graph where every node comes after the nodes it depends on.
Words and symbols in this lesson
- Tensor
- here: an array plus its gradient plus the Tensors it came from
- topological order
- every node listed after the nodes it depends on
- .grad
- the gradient of the final loss with respect to this Tensor
- unbroadcast
- summing a gradient back down to the shape it was stretched from
Write the topo method: return every node self depends on (including itself), each one after all of its parents, with no duplicates.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.