Which parts matter
Why it matters Before explaining a part, find out if it does anything. Removing parts (ablation) is the cheapest effect you can measure, and it changes where you look.
Each block of the GPT has an attention part (writes 'attn_out0', 'attn_out1') and an MLP (writes 'mlp_out0', 'mlp_out1'). Remove one at a time (set its output to zero) and measure the rise in held-out loss. Then remove each of the four heads of block 1 one at a time with interp.head_effects(model, X, Y, 1).
One risk: if two heads do the same job, removing either alone shows almost nothing. So also compare the four single-head rises, added up, with removing all of block 1's attention at once. If removing them together hurts far more than the sum, they back each other up.
Words and symbols in this lesson
- pattern, effect, mechanism
- three kinds of evidence: looked and saw; changed it and the output changed; explained it and the explanation predicted new cases
- ablation
- removing a part (setting its output to zero) and measuring what changes
Fill rise for the four parts, heads1 with the four single-head rises, then set backup to True if removing all of block 1's attention costs more than twice the sum of its heads alone, and biggest to the part whose removal costs most.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.