Batch size and learning rate together
Why it matters Batch size and learning rate are not separate knobs. Change one and the best value of the other moves.
A bigger batch gives a less noisy gradient, so it can take a bigger step safely. A rough rule many people start from: if you double the batch size, you can roughly double the learning rate, up to a point where the step becomes unstable.
Results also wobble from seed to seed, so compare settings over several seeds, never one run.
One more trap: the labels here carry noise with spread 0.1, so no setting can push the loss below about 0.1 squared = 0.01. Near that floor several pairs tie (for example 0.0113 against 0.0114). A "winner" by less than the seed-to-seed wobble is not a winner, so the run also lists every pair within 5% of the best.
Recap Tune them together, and average over seeds.
Words and symbols in this lesson
- SGD
- gradient descent using small random batches
Predict first: in my_pick, write the (batch, lr) pair from the grid you think ends with the lowest loss after 2 epochs. Then write sweep(X, y, batches, lrs, seeds) returning a dictionary from (batch, lr) to the mean final loss over the seeds (use the given sgd), and best(results) returning the best pair.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.