Why a stretched bowl is slow (conditioning)
Why it matters Real loss landscapes are steep in some directions and nearly flat in others. That mismatch, not the number of weights, is what makes plain gradient descent slow.
See it
Take f = (k x squared + y squared) / 2. With k = 1 it is a round bowl. With k = 100 it is a long thin valley: steep across, gentle along. The learning rate has to suit both directions at once. The best single choice, lr = 2 / (k + 1), makes each step keep (k - 1) / (k + 1) of the distance in both directions: 0.98 of it when k = 100. Any bigger and the steep direction overshoots and settles more slowly (above 2 / k it grows without limit); any smaller and the gentle direction crawls even slower. So the walk zigzags across the valley while it creeps along it.
Condition number: the ratio of the steepest to the gentlest curvature, here k. Plain gradient descent needs roughly k steps to make good progress.
Call the Curve: before running, predict how many steps it takes to get within 0.001 of the bottom for k = 1, 10 and 100, using the best fixed learning rate, 2 / (k + 1).
Recap Stretch slows plain descent in proportion to the stretch.
Words and symbols in this lesson
- condition number
- how much steeper the steepest direction is than the gentlest
Fill in my_guess (three step counts), then write steps_to_converge(k): start at (1, 1), use lr = 2 / (k + 1), and count steps until the distance to (0, 0) is below 0.001 (stop at 100,000).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.