Syllabus Lesson 219 of 246 · Module 21: Looking inside a model
Module 21: Looking inside a model

A probe needs a control

Why it matters A probe (a small linear classifier trained on a model's inside vectors) is the most common tool in interpretability, and the most over-read. High accuracy can mean the model computed the thing, or that the probe did.

You met linear probes in Module 18. Here, train one on the GPT's vectors at two places: 'embed' (the character's embedding plus its position, before any layer) and 'resid1' (after the first block). Three questions, each a 0/1 label for every character:

  • 'vowel': is this character a vowel?
  • 'control': is it one of b, c, g, n, r, s, t? A grouping that means nothing, picked to be as common as the vowels (28.7% of characters, against 28.8%).
  • 'third letter': is it the third letter of its word, or later?

Predict first: if the vowel probe scores close to 100%, does that show the model has learned what a vowel is?

Words and symbols in this lesson
interpretability
finding out what a trained network computes inside, and checking it
probe
a small linear classifier trained to read something from a model's inside vectors
Your turn

Set my_guess to "yes" or "no". Fill scores[kind][site] with probe(site, kind) for the three kinds and the two sites. Then set computed to the one kind whose probe shows something the layer worked out (it is far easier after the block than from the embedding alone).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.