Syllabus Lesson 54 of 246 · Module 4: Your own autograd engine
Module 4: Your own autograd engine

Recording the graph and ordering it

Why it matters Backward must visit each box only after every box that depends on it. Getting this order wrong gives silently wrong gradients.

See it

Picture x used twice: a = x * 2, b = x * 3, L = a + b. This is a diamond. Going backward from L, we must finish with both a and b before handling x, because x's gradient is the sum of what comes back through both.

The fix is to list the nodes in topological order: every node after all of its parents. Backward then walks that list in reverse. A depth-first search makes the list: visit a node's parents first, then add the node, and skip nodes already seen.

Topological order: an ordering of a graph where every node comes after the nodes it depends on.

Words and symbols in this lesson
Tensor
here: an array plus its gradient plus the Tensors it came from
topological order
every node listed after the nodes it depends on
.grad
the gradient of the final loss with respect to this Tensor
unbroadcast
summing a gradient back down to the shape it was stretched from
Your turn

Write the topo method: return every node self depends on (including itself), each one after all of its parents, with no duplicates.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.