How thin is enough? (rank)
Why it matters Rank is LoRA's one real setting. Too thin and it cannot learn the new task; past some point, more rank buys almost nothing. Where that point sits is worth predicting before paying for it.
The course trained LoRA on code at ranks 1 to 64, and full fine-tuning, 3 seeds each, 200 steps (adapt.recorded()). Each row has the rank, the trainable weights, the code loss (new) and the English loss (old).
Call the Curve
Before reading them, guess the code loss at ranks 1, 4, 16 and 64. Anchors: before training 3.25; full fine-tuning about 2.19.
Words and symbols in this lesson
- fine-tuning
- continuing to train a trained model on new data
- LoRA
- low-rank adaptation: a frozen weight matrix plus two thin trainable ones, x W + (alpha / r) x A B
- rank (r)
- how thin LoRA's two matrices are: (d_in + d_out) x r trainable weights per matrix
Fill my_guess (four code losses), then by_rank: for ranks 1, 4, 16, 64 and "full", the mean code loss and the mean English loss over the 3 seeds. Set diminishing to True if going from rank 16 to 64 gains less than a quarter of what going from 1 to 4 gains.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.