Syllabus Lesson 205 of 246 · Module 19: Bigger, and how much better (scaling laws)
Module 19: Bigger, and how much better (scaling laws)

Predict a bigger run, then see it

Why it matters A fit that only describes the runs you have is a summary. A fit that predicts a run you did not have is a law, at least for now. Here is the test.

The course trained one more size after the grid was done: width 2048, about 375,000 weights, twice the largest in the grid, on the full 1.5 million tokens, 3 seeds. You have not seen its loss.

Use the power law from lesson 3 (one width at a time, 1.5 million tokens) to predict it. Commit the number in code, then open the recorded result.

Words and symbols in this lesson
power law
y = A x^-alpha: a straight line on log-log axes
token
here, one prediction: one character
Your turn

Fit the 1.5-million-token grid with scaling.fit_power_law, set prediction for width 2048 (scaling.params_of(2048) weights), then set actual (the mean of scaling.recorded_bigger()) and error. Compare it with your gut error in lesson 1 by setting better_than_my_gut.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.