Predict a bigger run, then see it
Why it matters A fit that only describes the runs you have is a summary. A fit that predicts a run you did not have is a law, at least for now. Here is the test.
The course trained one more size after the grid was done: width 2048, about 375,000 weights, twice the largest in the grid, on the full 1.5 million tokens, 3 seeds. You have not seen its loss.
Use the power law from lesson 3 (one width at a time, 1.5 million tokens) to predict it. Commit the number in code, then open the recorded result.
Words and symbols in this lesson
- power law
- y = A x^-alpha: a straight line on log-log axes
- token
- here, one prediction: one character
Fit the 1.5-million-token grid with scaling.fit_power_law, set prediction for width 2048 (scaling.params_of(2048) weights), then set actual (the mean of scaling.recorded_bigger()) and error. Compare it with your gut error in lesson 1 by setting better_than_my_gut.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.