Syllabus Lesson 201 of 246 · Module 19: Bigger, and how much better (scaling laws)
Module 19: Bigger, and how much better (scaling laws)

Your own small runs, costed

Why it matters Scaling is a statement about cost: how much better per unit of compute. So every run needs a price tag, and the price tag of training is surprisingly simple.

Training costs about 6 operations per weight per token: 2 for the forward pass (one multiply, one add for every weight) and about 4 for the backward pass (gradients for the inputs and for the weights). That is the FLOP Ledger rule you have used since the start line: scaling.flops(weights, tokens).

Here you train three small models live, each on 32,000 tokens of the course text, about 7 times apart in cost.

Words and symbols in this lesson
token
here, one prediction: one character
FLOPs (6 N D)
training cost: about 6 operations per weight per token
Your turn

Train widths 4, 16 and 64 for 32,000 tokens each (scaling.train(model, 32_000) returns the held-out loss). Fill runs with, for each width, the weights, the FLOPs and the loss. Then set cost_ratio: how many times more the width-64 run cost than the width-4 run.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.