A big frozen matrix plus two thin ones (See it)
Why it matters LoRA (Hu et al. 2021) is how most models are customised today, because it trains a tiny fraction of the weights and leaves the original untouched. The idea fits in one picture.
Take a weight matrix W, d_in by d_out. Freeze it. Beside it, put two thin trainable matrices: A (d_in by r) and B (r by d_out), with a small rank r. The layer computes
x W + (alpha / r) x A B
A B is a full d_in by d_out matrix, a change to W, but it is built from only (d_in + d_out) x r numbers. B starts at zero, so at first the model is exactly the original.
The GPT has four weight matrices in each of its 2 blocks: qkv (64 by 192), o (64 by 64), w1 (64 by 256) and w2 (256 by 64). Watch the scene, then count.
Words and symbols in this lesson
- LoRA
- low-rank adaptation: a frozen weight matrix plus two thin trainable ones, x W + (alpha / r) x A B
- rank (r)
- how thin LoRA's two matrices are: (d_in + d_out) x r trainable weights per matrix
Write lora_weights(shapes, r): the number of trainable weights LoRA adds for a list of (d_in, d_out) shapes at rank r. Fill shapes with the GPT's 8 target matrices, then set counts for ranks 1, 4, 16 and 64, and overtakes: the smallest of those ranks at which LoRA trains more weights than the whole model (107,072).
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.