A better average, the same skill
Why it matters "Our model has lower loss" is the most common claim in language modelling. The loss is an average over every prediction, and an average can improve without the part you care about improving at all.
The course recorded two GPTs trained the same way, 32 and 64 wide. The wider one has the lower held-out loss. But predictions are not all alike: finishing a word you have started ("gradi" then "e") is easy; choosing the first letter of the next word is choosing what to say next, and it is hard.
judge.per_position(model, X, Y) gives each prediction's loss. Average it over all positions, over first letters of words, and over the rest of each word.
Predict first: does the wider model get better at choosing the next word?
Words and symbols in this lesson
- seed
- the number that fixes a run's random choices; another seed gives another result
Set my_guess to "yes" or "no". Fill loss[width] with a dict of 'all', 'first letter' and 'inside word' for widths 32 and 64. Set gain_from to the kind of position ("first letter" or "inside word") where the wider model's gain comes from.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.