Syllabus Lesson 220 of 246 · Module 21: Looking inside a model
Module 21: Looking inside a model

Put one value back: activation patching

Why it matters Activation patching (Meng et al. 2022, Wang et al. 2022) turns "this part lines up with the answer" into "this part carries the answer", by moving one value from one run into another.

Two inputs that differ in one character:

  • clean: "the gradient of the los": the GPT gives the next character "s" a probability of 0.91.
  • corrupted: "the gradient of the bos": 0.39.

Run the corrupted input, but at one place and one position, swap in the vector from the clean run. If the probability of "s" jumps back to 0.91, that vector was carrying what the "l" told the model. interp.patch(model, clean_ids, bad_ids, site, pos) does the swap and returns the logits; interp.next_prob(logits, answer, model) reads the probability.

Try it at 'resid0' (before block 0, so just the embeddings), 'resid1' (between the blocks) and 'final', for the last five positions.

Predict first: at 'resid1', is the "l" still carried at its own position, or somewhere else?

Words and symbols in this lesson
activation patching
running one input with a value taken from another input's run, to see what that value carries
Your turn

Set my_guess to "own position" or "somewhere else". Fill restored[site] with a list of five probabilities, one per position from len(clean) - 5 to the end. Then set where[site] to the character of clean at the position that restores it.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.