Looking for an induction head
Why it matters The induction head (Olsson et al. 2022) is the best-known story in interpretability: one head finds the earlier copy of the current token and attends to the token after it, so the model can continue a repeat. It is real in large models. The mistake is to assume a small model copies the same way because it copies.
The course trained a 2-block GPT only to copy: each row is a random run of 4 to 16 letters, repeated to fill the 32 positions. The first run cannot be predicted (loss about ln 26 = 3.26); after it, every letter can be copied. The model learned to copy: its loss on the repeats is about 0.13 to 0.15 (it depends on the sample).
The textbook prediction: some head puts most of its attention on the letter after an earlier copy of the current letter. Measure it. For each head, the share of attention, averaged over the repeat positions, that lands on positions j where the letter before j equals the current letter. Compare with uniform attention, which spreads evenly over all earlier positions. Then remove each head and measure the repeat loss.
Predict first: will one head put more than half of its attention there?
Where the field is now Induction heads are real: Olsson et al. (2022) found them forming, in a sudden phase change, in models trained on ordinary text, including two-layer models with attention only. Why not here? This model has MLPs and layer norm, trained briefly on one synthetic copying task with several possible solutions, and it found another one. That is the point of the lesson, and also its limit: a toy that does not show a published effect is not evidence against it. Results from this course's small models are measurements of those models; at scale, some will hold and some will not.
Words and symbols in this lesson
- interpretability
- finding out what a trained network computes inside, and checking it
- pattern, effect, mechanism
- three kinds of evidence: looked and saw; changed it and the output changed; explained it and the explanation predicted new cases
- induction head
- a head that finds an earlier copy of the current token and attends to the token after it
Set my_guess to "yes" or "no". Write induction_share(A, X, Ls): for attention weights A (B, t, t) of one head, the mean over rows b and repeat positions i (from Ls[b] to t - 1) of the attention on positions j ≤ i with X[b, j - 1] == X[b, i] (j ≥ 1). Fill share[(layer, head)] for the 8 heads and removed[(layer, head)] with the repeat loss when that head is set to zero. Set needed to the head whose removal hurts most.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.