Weights, data, or compute: what is holding it back?
Why it matters Spending more on the wrong thing buys nothing. A model starved of data does not improve when you make it bigger; a tiny model does not improve when you feed it more. The grid shows both, if you know where to look.
Look along a row of the grid (one width, more and more data) and down a column (one amount of data, wider and wider models).
A run is data-limited when making the model wider barely helps (or hurts) at that amount of data. It is model-limited when more data barely helps at that width.
Words and symbols in this lesson
- token
- here, one prediction: one character
- data-limited, model-limited
- when more width, or more data, barely helps
Fill gain_wider: for 8,000 tokens and for 1.5 million tokens, the loss drop from width 256 to width 1024. Fill gain_more_data: for width 2 and for width 1024, the loss drop from 512,000 to 1.5 million tokens. Then set verdict for two corners: "data-limited" or "model-limited".
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.