Syllabus Lesson 230 of 246 · Module 22: Judging claims
Module 22: Judging claims

A better average, the same skill

Why it matters "Our model has lower loss" is the most common claim in language modelling. The loss is an average over every prediction, and an average can improve without the part you care about improving at all.

The course recorded two GPTs trained the same way, 32 and 64 wide. The wider one has the lower held-out loss. But predictions are not all alike: finishing a word you have started ("gradi" then "e") is easy; choosing the first letter of the next word is choosing what to say next, and it is hard.

judge.per_position(model, X, Y) gives each prediction's loss. Average it over all positions, over first letters of words, and over the rest of each word.

Predict first: does the wider model get better at choosing the next word?

Words and symbols in this lesson
seed
the number that fixes a run's random choices; another seed gives another result
Your turn

Set my_guess to "yes" or "no". Fill loss[width] with a dict of 'all', 'first letter' and 'inside word' for widths 32 and 64. Set gain_from to the kind of position ("first letter" or "inside word") where the wider model's gain comes from.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.