Syllabus Lesson 213 of 246 · Module 20: Adapting a trained model cheaply
Module 20: Adapting a trained model cheaply

A big teacher, a small student (distillation)

Why it matters Small models are cheap to run. Distillation (Hinton, Vinyals and Dean, 2015) trains a small model to copy a big one's full answers, not just the right answer: "probably e, maybe a, certainly not z". Those soft answers carry more information per example than the text alone.

The student: a 16-wide GPT (about 8,000 weights). The teacher: the 64-wide GPT. The loss mixes two parts: half ordinary cross-entropy with the text, half cross-entropy with the teacher's probabilities, both softened by a temperature of 2 so the teacher's second choices count.

The course trained the student both ways, 3 seeds, and recorded the held-out English loss at 300, 1000 and 3000 steps.

Predict first: at 300 steps, does the taught student beat the one trained alone? And at 3000?

Words and symbols in this lesson
LoRA
low-rank adaptation: a frozen weight matrix plus two thin trainable ones, x W + (alpha / r) x A B
distillation
training a small student to copy a big teacher's full probabilities
temperature (distillation)
dividing scores before the softmax so second choices count more
Your turn

Set guess for 300 and 3000 steps to "taught wins", "alone wins" or "no clear difference". Then fill gap: for each of 300, 1000 and 3000 steps, mean alone minus mean taught (positive means distillation helped), and set verdict from it with the rule: a gap under 0.03 either way is no clear difference.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.