Syllabus Lesson 218 of 246 · Module 21: Looking inside a model
Module 21: Looking inside a model

Three kinds of evidence

Why it matters "The model has a neuron for X" is one of the most repeated sentences in machine learning, and most of the time the evidence behind it is a picture. This module is about the difference between a picture and a proof.

Every claim about what happens inside a network rests on one of three kinds of evidence, from weakest to strongest:

  • Pattern: something inside lines up with something outside. A unit is large when the text is about cats; a head attends to the previous word. You looked, you did not touch.
  • Effect: you changed something inside and the output changed. Remove the head and the loss rises; put a value from another input back in and the answer flips. Now you know it matters, but not exactly what it does.
  • Mechanism: an explanation of what the part computes that predicts what will happen on inputs you have not tried yet, and the prediction holds.

Patterns are cheap and they are where every investigation starts. They become claims only after an effect, and an explanation only after a prediction that could have failed.

Words and symbols in this lesson
pattern, effect, mechanism
three kinds of evidence: looked and saw; changed it and the output changed; explained it and the explanation predicted new cases
Your turn

Fill kind: for each claim, "pattern", "effect" or "mechanism" (the strongest kind its evidence supports).

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.