Loss surfaces you can walk
Why it matters Every optimiser is a different way of walking downhill on a landscape you cannot see. Walk a simple one first.
Picture the loss as a landscape: two weights are your position on a map, and the loss is the height. Training is walking downhill. The gradient points uphill, so each step goes the opposite way: position = position - lr * gradient.
Tiny numbers
On the bowl f(x, y) = x squared + y squared, the gradient at (3, 4) is (6, 8). With lr 0.1 the step goes to (3 - 0.6, 4 - 0.8) = (2.4, 3.2). Every step keeps 80% of the distance to the bottom, so after 20 steps you are at about 1% of where you started.
Learning rate: how far to step relative to the gradient. Too small crawls; too big overshoots.
Recap Step against the gradient; the learning rate sets the stride.
Write descend(grad, start, lr, steps) that returns the list of positions visited (start included) as NumPy arrays. Then choose my_lr so that, on the bowl from the lesson, starting at (3, 4), you are within 0.01 of the bottom in at most 15 steps.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.