Syllabus Lesson 200 of 246 · Module 19: Bigger, and how much better (scaling laws)
Module 19: Bigger, and how much better (scaling laws)

Guess a bigger model

Why it matters Training a large model costs months and millions. Before anyone spends that, they predict how good it will be from small runs. The surprising fact behind the AI boom is that the prediction usually works. First, see how good your own intuition is.

The models in this module are small character models: they read 8 characters and predict the next. One number sets their size, the width of the hidden layer. The course trained 150 of them (10 widths, 5 amounts of data, 3 seeds each) on 1.5 million characters of its own lesson text, and measured each one's loss on 100,000 characters it never saw.

A width-128 model (24,401 weights) trained on 1.5 million tokens reaches a held-out loss of about 1.86. A token here is one prediction: one character.

Before you look: a width-1024 model has 7.7 times the weights. Trained on the same data, what loss will it reach? For scale: predicting every character equally likely scores 4.42, and the best of these models get under 2.

Words and symbols in this lesson
token
here, one prediction: one character
Your turn

Set my_guess to your predicted held-out loss for width 1024 at 1.5 million tokens. Then set actual from the recorded runs (the mean over the 3 seeds), and error to how far off you were (actual minus guess).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.