Syllabus Lesson 66 of 246 · Module 5: Optimisation
Module 5: Optimisation

Noisy steps (SGD and minibatches)

Why it matters Real datasets have millions of examples. Computing the exact gradient on all of them for every step is far too slow, so we estimate it from a small batch.

A minibatch gradient uses, say, 32 random examples instead of all 512. It points roughly downhill, with some noise. You get 16 cheap noisy steps for the price of one exact step, and in practice that wins easily.

What the noise does

The loss curve becomes jagged instead of smooth, and near the bottom it never settles completely: it jiggles around the minimum. Lowering the learning rate late in training calms the jiggle. Some noise is even helpful in big networks, shaking the walk out of poor spots.

SGD: stochastic gradient descent, gradient descent with minibatch estimates of the gradient.

Recap Many cheap noisy steps beat few exact ones.

Words and symbols in this lesson
SGD
gradient descent using small random batches
minibatch
a small random slice of the data used for one step
Your turn

Write sgd(X, y, lr, batch, epochs, seed) for linear regression (loss: mean squared error), shuffling each epoch, returning (w, losses) where losses is the full-data loss after each epoch. With batch 32, lr 0.05 and 10 epochs it must end below 0.02.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.