Syllabus Lesson 207 of 246 · Module 19: Bigger, and how much better (scaling laws)
Module 19: Bigger, and how much better (scaling laws)

Gate: plan a training run (AI off)

Why it matters This is the decision scaling laws exist for: you have a budget and one shot. Plan it from evidence.

Gate rules: AI tutor off. You have 3 x 10^10 FLOPs for one run of the course's character model, on the same text as the grid. Choose a width (a power of 2) and the number of tokens the budget then allows. Use the recorded grid any way you like: fits, tables, both.

The check predicts the loss of your choice and of the best possible choice with the course's two-variable fit, and passes when yours is within 0.02 of the best and you spent the budget (within 1%).

Words and symbols in this lesson
scaling law
how loss falls as weights, data or compute grow, usually a power law towards a floor
token
here, one prediction: one character
FLOPs (6 N D)
training cost: about 6 operations per weight per token
Your turn

Set width and tokens.

AI off: no tutor, hints or worked solution on this one. Your own thinking only.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.