Smaller weights (quantisation)
Why it matters Models are stored and run as numbers, and most numbers do not need 32 bits. Storing each weight in 8 or 4 bits makes a model 4 or 8 times smaller and often faster, which is how large models fit on laptops and phones. The question is what it costs.
The simplest scheme, per row of each weight matrix: find the largest absolute value, divide the row into 2^(bits - 1) - 1 equal steps on each side of zero, and round every weight to the nearest step. Store the rounded steps (small integers) and one scale per row.
Tiny numbers
Row [0.9, -0.3, 0.05] at 4 bits: 7 steps each side; scale 0.9 / 7 = 0.129. 0.9 -> 7 steps -> 0.9; -0.3 -> -2.33 -> -2 steps -> -0.257; 0.05 -> 0.39 -> 0 steps -> 0. The small weight disappears.
Words and symbols in this lesson
- quantisation
- storing weights in fewer bits by rounding them to a few steps
Write quantize_row(w, bits) as described. Then set my_guess: the English loss change you expect at 8 and at 4 bits, and fill measured for 8, 4, 3 and 2 bits using adapt.quantize (it applies the same rule to every matrix).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.