Real jumps, or a trick of the metric?
Why it matters Headlines say large models suddenly gain abilities at some size: "emergence". Sometimes the jump is real. Often the ability improved smoothly all along and the metric made it look sudden (Schaeffer, Miranda and Koyejo, 2023). Knowing which is which decides whether you can predict it.
The grid recorded two scores on the same held-out text:
- Per-character accuracy p: the share of positions where the model's top guess for the next character is right. It rises smoothly with size, 0.27 to 0.53 at 1.5 million tokens.
- All 5 right: the share of positions where it writes the next 5 characters with no mistake. All or nothing.
If each character were right with probability p independently, five in a row would be right p^5 of the time. Check that against the grid first. Then see what the exponent does to a skill that improves evenly. These small models never get far enough to show the full effect, so the second part is a plainly labelled illustration, not a measurement: a per-character accuracy rising in ten equal steps from 0.5 to 0.95.
Words and symbols in this lesson
- exponent (alpha, beta)
- how fast loss falls with more weights (alpha) or more data (beta)
- token
- here, one prediction: one character
- emergence
- an ability that seems to appear suddenly at some size; sometimes made by an all-or-nothing metric
Fill pairs: for each width at 1.5 million tokens, (p, p^5, measured all-5-right). Then for the illustration set all10 to p^10 for p_even, and late_share to the share of all10's total rise that happens in the last 3 of the 10 steps. Finally set case.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.