Syllabus Lesson 28 of 246 · Module 2: Maths and probability, only as needed
Module 2: Maths and probability, only as needed

Why backward is cheap (reverse mode)

Why it matters A network has millions of weights. If each gradient needed its own pass, training would be impossible. Going backward gets all of them for about the price of one more pass.

Tiny numbers

Take L = f3(f2(f1(x))) where x has 3 numbers and L is one number. Going forward with nudges, you nudge one input, push it through all three steps, and learn one gradient. Three inputs means three full trips.

Going backward, you start from L, which is one number, and push "how much does L care about this?" back through f3, then f2, then f1. One trip gives the gradient for every input at once.

With 3 inputs you make 1 trip instead of 3. With a million weights, 1 trip instead of a million. A backward trip is not free, though: it costs about 2 to 3 forward passes, because each step must multiply by its local slopes. So in real work the saving is about n / 3, which for a million weights is still around 300,000 times. That is why every deep learning library works backward.

Reverse mode: computing gradients by sweeping from the output back to the inputs. Its building block is called a vector-Jacobian product; the next lesson takes it apart.

Recap One output, many inputs: go backward.

Words and symbols in this lesson
vector
a list of numbers, one per feature
Your turn

Fill in the operation counts for a network with n weights and one loss. forward_trips(n): how many full passes the nudge-each-input method needs (one nudged pass per weight; leave out the single un-nudged pass you would also run). backward_trips(n): how many backward sweeps reverse mode needs. Then set saving_for_a_million to the ratio of trips (not of work) for n = 1,000,000.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.