Noisy steps (SGD and minibatches)
Why it matters Real datasets have millions of examples. Computing the exact gradient on all of them for every step is far too slow, so we estimate it from a small batch.
A minibatch gradient uses, say, 32 random examples instead of all 512. It points roughly downhill, with some noise. You get 16 cheap noisy steps for the price of one exact step, and in practice that wins easily.
What the noise does
The loss curve becomes jagged instead of smooth, and near the bottom it never settles completely: it jiggles around the minimum. Lowering the learning rate late in training calms the jiggle. Some noise is even helpful in big networks, shaking the walk out of poor spots.
SGD: stochastic gradient descent, gradient descent with minibatch estimates of the gradient.
Recap Many cheap noisy steps beat few exact ones.
Words and symbols in this lesson
- SGD
- gradient descent using small random batches
- minibatch
- a small random slice of the data used for one step
Write sgd(X, y, lr, batch, epochs, seed) for linear regression (loss: mean squared error), shuffling each epoch, returning (w, losses) where losses is the full-data loss after each epoch. With batch 32, lr 0.05 and 10 epochs it must end below 0.02.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.