Syllabus Lesson 206 of 246 · Module 19: Bigger, and how much better (scaling laws)
Module 19: Bigger, and how much better (scaling laws)

Real jumps, or a trick of the metric?

Why it matters Headlines say large models suddenly gain abilities at some size: "emergence". Sometimes the jump is real. Often the ability improved smoothly all along and the metric made it look sudden (Schaeffer, Miranda and Koyejo, 2023). Knowing which is which decides whether you can predict it.

The grid recorded two scores on the same held-out text:

  • Per-character accuracy p: the share of positions where the model's top guess for the next character is right. It rises smoothly with size, 0.27 to 0.53 at 1.5 million tokens.
  • All 5 right: the share of positions where it writes the next 5 characters with no mistake. All or nothing.

If each character were right with probability p independently, five in a row would be right p^5 of the time. Check that against the grid first. Then see what the exponent does to a skill that improves evenly. These small models never get far enough to show the full effect, so the second part is a plainly labelled illustration, not a measurement: a per-character accuracy rising in ten equal steps from 0.5 to 0.95.

Words and symbols in this lesson
exponent (alpha, beta)
how fast loss falls with more weights (alpha) or more data (beta)
token
here, one prediction: one character
emergence
an ability that seems to appear suddenly at some size; sometimes made by an all-or-nothing metric
Your turn

Fill pairs: for each width at 1.5 million tokens, (p, p^5, measured all-5-right). Then for the illustration set all10 to p^10 for p_even, and late_share to the share of all10's total rise that happens in the last 3 of the 10 steps. Finally set case.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.