Syllabus Lesson 209 of 246 · Module 20: Adapting a trained model cheaply
Module 20: Adapting a trained model cheaply

A big frozen matrix plus two thin ones (See it)

Why it matters LoRA (Hu et al. 2021) is how most models are customised today, because it trains a tiny fraction of the weights and leaves the original untouched. The idea fits in one picture.

Take a weight matrix W, d_in by d_out. Freeze it. Beside it, put two thin trainable matrices: A (d_in by r) and B (r by d_out), with a small rank r. The layer computes

x W + (alpha / r) x A B

A B is a full d_in by d_out matrix, a change to W, but it is built from only (d_in + d_out) x r numbers. B starts at zero, so at first the model is exactly the original.

The GPT has four weight matrices in each of its 2 blocks: qkv (64 by 192), o (64 by 64), w1 (64 by 256) and w2 (256 by 64). Watch the scene, then count.

Words and symbols in this lesson
LoRA
low-rank adaptation: a frozen weight matrix plus two thin trainable ones, x W + (alpha / r) x A B
rank (r)
how thin LoRA's two matrices are: (d_in + d_out) x r trainable weights per matrix
Your turn

Write lora_weights(shapes, r): the number of trainable weights LoRA adds for a list of (d_in, d_out) shapes at rank r. Fill shapes with the GPT's 8 target matrices, then set counts for ranks 1, 4, 16 and 64, and overtakes: the smallest of those ranks at which LoRA trains more weights than the whole model (107,072).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.