Syllabus Lesson 240 of 246 · Module 23: Capstone studio: a model card you can run
Module 23: Capstone studio: a model card you can run

One finding inside your model, with a control

Why it matters Module 21's lesson, applied to your own model: a probe's score means nothing until you compare it with what the same probe reads from a network that never trained.

The probe set for your track asks one question about each input (meaning): on the language track, "is the next character a vowel?"; on the images track, "which object is in the photo?".

Train your baseline (seed 0). Read its hidden layer (studio.hidden_units(model, X)) and train a linear probe on the first half of the probe set, testing on the second (ssl.probe, 300 steps). The control: the same probe on an untrained network of the same shape and seed (studio.MLP(track.n_in, width, track.n_out, seed=0)). Random ReLU units already turn inputs into many features, and a probe can read a lot from those; only the part above the control is something training put there.

Words and symbols in this lesson
baseline
the plain run every change is measured against, over the same seeds
Your turn

Fill trained, control and base_rate (the share of the most common label in the test half). Set finding to "above control" if trained beats control by more than 0.02, else "not above control".

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.