Syllabus Lesson 224 of 246 · Module 21: Looking inside a model
Module 21: Looking inside a model

Unmixing: a sparse autoencoder

Why it matters If features are stored in superposition, looking at units will not find them. A sparse autoencoder (Bricken et al. 2023, Cunningham et al. 2023) tries to undo it: rebuild each vector from many more directions than it has numbers, but only a few at a time.

Take the GPT's 64-number vectors after block 1, for 19,200 characters of lesson text. A sparse autoencoder with 256 features computes f = relu(z We + be) and rebuilds z = f Wd. The loss is the rebuild error plus l1 times the total size of f, which pushes most features to exactly zero.

interp.train_sae(A, features=256, l1=...) trains one; sae.unexplained(A) is the share of the vectors' spread the rebuild misses (0 is perfect) and sae.features(A) gives f.

Predict first: with a strong push (l1 = 0.3), will the rebuild still explain most of the spread (unexplained below 0.2)?

Words and symbols in this lesson
superposition
storing more features than there are numbers, as directions that are rarely on together
sparse autoencoder
a wide layer trained to rebuild vectors from a few active features at a time
Your turn

Set my_guess to "yes" or "no". Train at l1 0.03 and 0.3 (600 steps of 128 vectors each). Fill unexplained[l1] on the held-out vectors A_new, and active[l1]: the mean number of features above zero per held-out character.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.