Syllabus Lesson 214 of 246 · Module 20: Adapting a trained model cheaply
Module 20: Adapting a trained model cheaply

Smaller weights (quantisation)

Why it matters Models are stored and run as numbers, and most numbers do not need 32 bits. Storing each weight in 8 or 4 bits makes a model 4 or 8 times smaller and often faster, which is how large models fit on laptops and phones. The question is what it costs.

The simplest scheme, per row of each weight matrix: find the largest absolute value, divide the row into 2^(bits - 1) - 1 equal steps on each side of zero, and round every weight to the nearest step. Store the rounded steps (small integers) and one scale per row.

Tiny numbers

Row [0.9, -0.3, 0.05] at 4 bits: 7 steps each side; scale 0.9 / 7 = 0.129. 0.9 -> 7 steps -> 0.9; -0.3 -> -2.33 -> -2 steps -> -0.257; 0.05 -> 0.39 -> 0 steps -> 0. The small weight disappears.

Words and symbols in this lesson
quantisation
storing weights in fewer bits by rounding them to a few steps
Your turn

Write quantize_row(w, bits) as described. Then set my_guess: the English loss change you expect at 8 and at 4 bits, and fill measured for 8, 4, 3 and 2 bits using adapt.quantize (it applies the same rule to every matrix).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.