Fitting a power law with a floor
Why it matters "Bigger is better" is useless for planning. "Loss falls as weights to the power -0.5, towards a floor of 1.6" lets you put a number on a run you have not paid for. Getting that number, and how sure you are of it, is the skill.
The shape that fits most scaling data:
loss = E + A N^(-alpha)
N is the number of weights, alpha the exponent (how fast it improves), and E the floor: the loss no model of this family can beat on this data. Part of it is the text being unpredictable; part is the family's own limit (this model only ever sees 8 characters back). A different family, say one that reads further back, has a lower floor. Without the floor, a fit on small models overestimates how much big ones gain.
How to fit it: for each candidate floor E between 0 and just under the smallest loss, take logs, log(loss - E) = log A - alpha log N, which is a straight line, and fit it by least squares. Keep the E whose curve has the smallest squared error on the losses themselves.
Error bars: there are 3 seeds per size. Refit many times, each time picking one seed at random for every size (a bootstrap), and look at the spread of alpha. Watch the scene: the same points, the fitted curve with and without a floor, and the band of bootstrap fits.
Words and symbols in this lesson
- power law
- y = A x^-alpha: a straight line on log-log axes
- exponent (alpha, beta)
- how fast loss falls with more weights (alpha) or more data (beta)
- floor (E)
- the loss no model can beat on this data
- token
- here, one prediction: one character
- bootstrap
- refitting on resampled data to see how much a fitted number would move
Write fit(N, loss) returning (E, A, alpha) as described (400 candidate floors from 0 to 0.999 x the smallest loss). Fit the 1.5-million-token runs (one mean per width) into main, then run 100 bootstrap refits and set interval to the 5th and 95th percentile of alpha.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.