Grokking: memorise first, understand later
Why it matters Power et al. (2022) saw a network reach perfect training accuracy, sit at chance on test data for a long time, then suddenly generalise. It is the clearest case where looking inside explained something the loss curve could not.
The task: a + b modulo 23 (the remainder after dividing by 23; so 20 + 5 is 2). There are 23 x 23 = 529 sums; the model trains on a random half and is tested on the other half. Each number gets an embedding of 32; the two embeddings go through one ReLU layer of 128 and out to 23 scores. It trains on the whole half at every step, with AdamW (Module 5).
Train it live for 1,500 steps, then read the course's recorded 40,000-step runs (about 11 seconds each on a laptop, too long for a lesson), with weight decay 3 and with none.
Predict first: after 1,500 steps, train accuracy will be 100%. What will test accuracy be?
Words and symbols in this lesson
- grokking
- test accuracy rising long after training accuracy is perfect
Set my_guess to "near 0", "near 50%" or "near 100%". Train for 1,500 steps and fill live with (train accuracy, test accuracy). From runs = interp.recorded()['grok'], each run's 'curve' has rows (step, train accuracy, test accuracy, top-3 share): set half_way to the first step where test accuracy reaches 0.5 with weight decay 3, and final_test to each run's last test accuracy.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.