Syllabus Lesson 231 of 246 · Module 22: Judging claims
Module 22: Judging claims

How you remove a part changes the answer

Why it matters "Removing this part raises the loss by X" sounds like a fact about the part. Part of it is a fact about how it was removed.

In Module 21 you removed parts of the GPT by setting their output to zero. But the rest of the network never saw a zero there in training: zero may be an input it has never met, so the damage measures the shock as well as the loss of information. A gentler removal replaces the output with its average over many inputs (mean ablation): the part stops carrying anything about this input, but the next layer still gets a value of the usual size.

Measure both for the four parts: 'attn_out0', 'mlp_out0', 'attn_out1', 'mlp_out1'. The average comes from cache[site].mean((0, 1)).

Predict first: for block 0's MLP, will mean ablation cost less than zeroing, or about the same?

Words and symbols in this lesson
mean ablation
replacing a part's output with its average instead of zero
Your turn

Set my_guess to "less" or "same". Fill zero[site] and mean[site] with the rise in loss. Set most_overstated to the part where zeroing overstates the effect most.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.