More features than room: superposition
Why it matters If each unit meant one thing, reading a network would be easy. They usually do not, and the toy model of Elhage et al. (2022) shows why: a network can store more features than it has numbers, as long as the features are rarely on together.
The toy: 5 features, each 0 most of the time and otherwise a number between 0 and 1. A matrix W (2 by 5) squeezes them into just 2 numbers, h = W x, and the model reads them back as relu(WT h + b). Feature i matters 0.9i as much in the loss (feature 0 most). Column i of W is the direction where feature i is stored; a column near zero means the feature was dropped.
interp.superposition(sparsity=s) trains it (s = the chance each feature is 0). interp.stored(W) lists the features it kept and interp.angles(W) the angles between them.
Predict first: when features are rarely on (sparsity 0.97), how many of the 5 can 2 numbers keep?
Words and symbols in this lesson
- superposition
- storing more features than there are numbers, as directions that are rarely on together
Set my_guess to a number from 2 to 5. Train at sparsity 0, 0.7 and 0.97 (seed 0) and fill kept[s] with the number of features stored, and smallest_angle[s] with the smallest angle between two stored features.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.