Syllabus Lesson 211 of 246 · Module 20: Adapting a trained model cheaply
Module 20: Adapting a trained model cheaply

How thin is enough? (rank)

Why it matters Rank is LoRA's one real setting. Too thin and it cannot learn the new task; past some point, more rank buys almost nothing. Where that point sits is worth predicting before paying for it.

The course trained LoRA on code at ranks 1 to 64, and full fine-tuning, 3 seeds each, 200 steps (adapt.recorded()). Each row has the rank, the trainable weights, the code loss (new) and the English loss (old).

Call the Curve

Before reading them, guess the code loss at ranks 1, 4, 16 and 64. Anchors: before training 3.25; full fine-tuning about 2.19.

Words and symbols in this lesson
fine-tuning
continuing to train a trained model on new data
LoRA
low-rank adaptation: a frozen weight matrix plus two thin trainable ones, x W + (alpha / r) x A B
rank (r)
how thin LoRA's two matrices are: (d_in + d_out) x r trainable weights per matrix
Your turn

Fill my_guess (four code losses), then by_rank: for ranks 1, 4, 16, 64 and "full", the mean code loss and the mean English loss over the 3 seeds. Set diminishing to True if going from rank 16 to 64 gains less than a quarter of what going from 1 to 4 gains.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.