Syllabus Lesson 227 of 246 · Module 21: Looking inside a model
Module 21: Looking inside a model

What changed inside: waves

Why it matters Nanda et al. (2023) showed that networks that grok modular addition end up computing it with waves: each number becomes a point on a few circles, and adding numbers turns into adding angles. It is one of the few cases where a full mechanism was found and then checked.

Read each embedding column as a list of 23 values, one per number 0 to 22, and split it into waves of frequency 1 to 11 (a wave of frequency k goes round k times from 0 to 22; this split is the Fourier transform). interp.frequency_shares(emb) gives each frequency's share of the embedding (they sum to 1). If the model uses a few waves, a few frequencies hold most of it; spread evenly, the top 3 would hold 3/11 = 0.27.

A pattern alone is not enough: rebuild the embedding from only some frequencies with interp.keep_frequencies(emb, keep) and measure test accuracy. If the top 3 are the mechanism, keeping only them should keep most of the accuracy, and removing only them should destroy it.

Words and symbols in this lesson
pattern, effect, mechanism
three kinds of evidence: looked and saw; changed it and the output changed; explained it and the explanation predicted new cases
Fourier transform
splitting a list of values into waves of different frequencies
Your turn

Load g = interp.grok_model() (the recorded run with weight decay 3, after 40,000 steps). Set top3 to its three largest frequencies, top3_share to their total share, and test_acc for three embeddings: 'all', 'only top 3' and 'all but top 3'.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.