Your own small runs, costed
Why it matters Scaling is a statement about cost: how much better per unit of compute. So every run needs a price tag, and the price tag of training is surprisingly simple.
Training costs about 6 operations per weight per token: 2 for the forward pass (one multiply, one add for every weight) and about 4 for the backward pass (gradients for the inputs and for the weights). That is the FLOP Ledger rule you have used since the start line: scaling.flops(weights, tokens).
Here you train three small models live, each on 32,000 tokens of the course text, about 7 times apart in cost.
Words and symbols in this lesson
- token
- here, one prediction: one character
- FLOPs (6 N D)
- training cost: about 6 operations per weight per token
Train widths 4, 16 and 64 for 32,000 tokens each (scaling.train(model, 32_000) returns the held-out loss). Fill runs with, for each width, the weights, the FLOPs and the loss. Then set cost_ratio: how many times more the width-64 run cost than the width-4 run.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.