Wider models want smaller steps
Why it matters The learning rate that works for a small model can wreck a bigger one. Teams that train large models spend real money finding this out; you can read it from recorded runs.
The recorded gpt-width-lr experiment trained the tiny GPT at widths 32, 64 and 128, each with learning rates 0.001, 0.003 and 0.01, for 2,000 steps, with three seeds each, and measured the loss on held-out text at the end.
Predict before you look: which learning rate is best for the widest model, 128?
Words and symbols in this lesson
- query, key, value
- what a position looks for; what each position offers to match on; what it hands over
Set my_guess to one of 0.001, 0.003 or 0.01, then fill mean_loss: (width, lr) to the average final held-out loss over the three seeds (exp.seeds(d=..., lr=...) gives the three runs; each run's final held-out loss is run.val_loss[-1][1]). Then fill best_lr: width to the learning rate with the lowest average.
Then use Module 9's noise rule to say which settings really differ. With three seeds each, two averages differ for real only if the gap is more than about 2 × s × sqrt(2/3), where s is the larger of the two settings' seed spreads (np.std of the three values). Anything closer is a tie.
Set ties: width to the list of learning rates that tie with that width's best (the best included).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.