Syllabus Lesson 223 of 246 · Module 21: Looking inside a model
Module 21: Looking inside a model

More features than room: superposition

Why it matters If each unit meant one thing, reading a network would be easy. They usually do not, and the toy model of Elhage et al. (2022) shows why: a network can store more features than it has numbers, as long as the features are rarely on together.

The toy: 5 features, each 0 most of the time and otherwise a number between 0 and 1. A matrix W (2 by 5) squeezes them into just 2 numbers, h = W x, and the model reads them back as relu(WT h + b). Feature i matters 0.9i as much in the loss (feature 0 most). Column i of W is the direction where feature i is stored; a column near zero means the feature was dropped.

interp.superposition(sparsity=s) trains it (s = the chance each feature is 0). interp.stored(W) lists the features it kept and interp.angles(W) the angles between them.

Predict first: when features are rarely on (sparsity 0.97), how many of the 5 can 2 numbers keep?

Words and symbols in this lesson
superposition
storing more features than there are numbers, as directions that are rarely on together
Your turn

Set my_guess to a number from 2 to 5. Train at sparsity 0, 0.7 and 0.97 (seed 0) and fill kept[s] with the number of features stored, and smallest_angle[s] with the smallest angle between two stored features.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.