The recipe, for sequence models
Why it matters Module 8's two checks (loss at the start, memorise one batch) still apply. Sequence models add new ways to fail silently, and two cheap checks catch the worst of them before you spend any compute.
The clinic's model is a tiny GPT: width 16, 2 blocks, 2 heads, 16 characters of context, 200 Adam steps on the course text. It trains in a couple of seconds, so you can afford to probe.
Check 1: the starting loss
A fresh model with small weights spreads its guesses evenly over the V characters, so its loss is ln(V). For 95 characters that is 4.55. Far above it means something is scaled wrong; far below means the model is getting help it should not have.
Check 2: no peeking
Change the last character of an input. If any prediction for an earlier position changes, the model can see the future (Module 13), and every training number you get is too good to be true.
| Probe | What it tells you | Healthy run |
|---|---|---|
loss_curve(c) | Training loss every 25 steps, and the loss on unseen text at the end. Always look first. | 4.55 down to about 2.9; unseen about 3.0 |
loss_at_init(c) | Loss before training. A fresh model guessing evenly over 95 characters gives ln(95). | 4.55 |
leak_check(c) | Can a later character change an earlier prediction? | False |
show_batch(c) | One training input and its target, as text. | target = input moved one character left |
update_ratio(c) | After the first Adam step, how far weights moved relative to their size. Not Module 8's rule: that 0.001 to 0.1 range is for plain SGD steps. Adam's first step moves every weight by about the learning rate whatever its gradient, and these weights start near 0.02, so 0.01 / 0.02 gives about 0.5. Compare a case with this healthy run, not with Module 8's range. | about 0.5 |
block_change(c) | How much the first block changes each vector at the start. | about 0.3 |
Words and symbols in this lesson
- ln(V)
- the starting loss of a model that guesses evenly over V symbols: 4.55 for 95 characters
- too good to be true
- training loss far better than a healthy run, unseen-text loss far worse
- block change
- how much one block changes a vector at the start; near 1 means it replaced it
Write expected_start(V) and my_leak_check(c), using the clinic's own _init, _logits and DATA. They must agree with the clinic's probes.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.