Syllabus Lesson 204 of 246 · Module 19: Bigger, and how much better (scaling laws)
Module 19: Bigger, and how much better (scaling laws)

The best use of a fixed budget

Why it matters With a fixed compute budget C, a bigger model sees fewer tokens (C = 6 N D). Too big and it is starved; too small and it is full before the budget runs out. In 2022 this calculation (the "Chinchilla" paper) showed that most large models of the day were far too big for the data they were trained on.

Fit both effects at once:

loss = E + A / N^alpha + B / D^beta

scaling.fit_2d fits it to all 150 runs. Then for a budget C, every width N fixes D = C / (6 N), and the formula predicts the loss: pick the width with the lowest prediction. scaling.params_of(width) gives a width's weights.

Where the field is now Hoffmann et al. (2022, "Chinchilla") fitted this same two-variable form to large language models and found the compute-optimal split at about 20 training tokens per weight. Labs now often train well past it (far more tokens for a smaller model), because a smaller model is cheaper every time it answers: compute-optimal for training is not optimal for use. And when they make a model wider, they rescale its starting weights and per-layer learning rates (muP, Yang et al. 2022) so that the best learning rate found on a small model still works on the big one: the problem the "wider models want smaller steps" lesson of Module 13 measured.

Words and symbols in this lesson
exponent (alpha, beta)
how fast loss falls with more weights (alpha) or more data (beta)
token
here, one prediction: one character
FLOPs (6 N D)
training cost: about 6 operations per weight per token
compute-optimal
the split of a fixed budget between model size and data that gives the lowest loss
Your turn

Fit the grid into f. Write plan(f, budget) returning the width (a power of 2 from 2 to 4096) with the lowest predicted loss. Fill choice for budgets of 1e9, 1e10 and 1e11 FLOPs.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.