Syllabus Lesson 203 of 246 · Module 19: Bigger, and how much better (scaling laws)
Module 19: Bigger, and how much better (scaling laws)

Weights, data, or compute: what is holding it back?

Why it matters Spending more on the wrong thing buys nothing. A model starved of data does not improve when you make it bigger; a tiny model does not improve when you feed it more. The grid shows both, if you know where to look.

Look along a row of the grid (one width, more and more data) and down a column (one amount of data, wider and wider models).

A run is data-limited when making the model wider barely helps (or hurts) at that amount of data. It is model-limited when more data barely helps at that width.

Words and symbols in this lesson
token
here, one prediction: one character
data-limited, model-limited
when more width, or more data, barely helps
Your turn

Fill gain_wider: for 8,000 tokens and for 1.5 million tokens, the loss drop from width 256 to width 1024. Fill gain_more_data: for width 2 and for width 1024, the loss drop from 512,000 to 1.5 million tokens. Then set verdict for two corners: "data-limited" or "model-limited".

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.