Syllabus Lesson 221 of 246 · Module 21: Looking inside a model
Module 21: Looking inside a model

Which parts matter

Why it matters Before explaining a part, find out if it does anything. Removing parts (ablation) is the cheapest effect you can measure, and it changes where you look.

Each block of the GPT has an attention part (writes 'attn_out0', 'attn_out1') and an MLP (writes 'mlp_out0', 'mlp_out1'). Remove one at a time (set its output to zero) and measure the rise in held-out loss. Then remove each of the four heads of block 1 one at a time with interp.head_effects(model, X, Y, 1).

One risk: if two heads do the same job, removing either alone shows almost nothing. So also compare the four single-head rises, added up, with removing all of block 1's attention at once. If removing them together hurts far more than the sum, they back each other up.

Words and symbols in this lesson
pattern, effect, mechanism
three kinds of evidence: looked and saw; changed it and the output changed; explained it and the explanation predicted new cases
ablation
removing a part (setting its output to zero) and measuring what changes
Your turn

Fill rise for the four parts, heads1 with the four single-head rises, then set backup to True if removing all of block 1's attention costs more than twice the sum of its heads alone, and biggest to the part whose removal costs most.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.