Put one value back: activation patching
Why it matters Activation patching (Meng et al. 2022, Wang et al. 2022) turns "this part lines up with the answer" into "this part carries the answer", by moving one value from one run into another.
Two inputs that differ in one character:
- clean: "the gradient of the los": the GPT gives the next character "s" a probability of 0.91.
- corrupted: "the gradient of the bos": 0.39.
Run the corrupted input, but at one place and one position, swap in the vector from the clean run. If the probability of "s" jumps back to 0.91, that vector was carrying what the "l" told the model. interp.patch(model, clean_ids, bad_ids, site, pos) does the swap and returns the logits; interp.next_prob(logits, answer, model) reads the probability.
Try it at 'resid0' (before block 0, so just the embeddings), 'resid1' (between the blocks) and 'final', for the last five positions.
Predict first: at 'resid1', is the "l" still carried at its own position, or somewhere else?
Words and symbols in this lesson
- activation patching
- running one input with a value taken from another input's run, to see what that value carries
Set my_guess to "own position" or "somewhere else". Fill restored[site] with a list of five probabilities, one per position from len(clean) - 5 to the end. Then set where[site] to the character of clean at the position that restores it.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.