Syllabus Lesson 216 of 246 · Module 20: Adapting a trained model cheaply
Module 20: Adapting a trained model cheaply

Choosing a method under a budget

Why it matters Each method in this module is the right answer to a different question. Picking by fashion wastes money; picking from measurements is the job.

What you measured, in one place:

  • Full fine-tuning (200 steps): code 2.19, English 2.26. Trains all 107,072 weights; a whole new copy of the model to store.
  • LoRA rank 4: code 2.27, English 2.04. Trains 8,192; stores only those.
  • Distillation to a 16-wide student (3000 steps): English 1.98 at about 1/8 of the weights, against 2.18 trained alone.
  • Quantisation: 8 bits, a quarter of the size, no change; 4 bits, an eighth, +0.13.

For each situation, choose the method the measurements support.

Words and symbols in this lesson
fine-tuning
continuing to train a trained model on new data
LoRA
low-rank adaptation: a frozen weight matrix plus two thin trainable ones, x W + (alpha / r) x A B
rank (r)
how thin LoRA's two matrices are: (d_in + d_out) x r trainable weights per matrix
distillation
training a small student to copy a big teacher's full probabilities
temperature (distillation)
dividing scores before the softmax so second choices count more
quantisation
storing weights in fewer bits by rounding them to a few steps
Your turn

Fill choice with one of "full fine-tune", "LoRA", "distillation" or "quantisation" for each situation.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.