Clinic: step size and starting weights
Why it matters Module 8's learning-rate and scale faults return, with Adam's fingerprints on them.
With Adam, every weight moves by about the learning rate on the first step, whatever its gradient. So the update ratio is a direct read-out of the step size: about 0.5 in the healthy run (0.01 against weights near 0.02), about 15 when the rate is far too high (the loss then lurches around), and about 0.0005 when it is far too low (the loss barely leaves 4.55). Starting weights that are far too big make the very first predictions wildly confident and wrong: the starting loss is far above 4.55.
Each case below is a training run with one fault planted in it. For each: spend probes to find the fault, then write a fix as config changes that put the run back to healthy. The faults in this lesson: lr_too_high, lr_too_low, init_too_big. A good diagnostician rarely needs more than two probes per case.
| Probe | What it tells you | Healthy run |
|---|---|---|
loss_curve(c) | Training loss every 25 steps, and the loss on unseen text at the end. Always look first. | 4.55 down to about 2.9; unseen about 3.0 |
loss_at_init(c) | Loss before training. A fresh model guessing evenly over 95 characters gives ln(95). | 4.55 |
leak_check(c) | Can a later character change an earlier prediction? | False |
show_batch(c) | One training input and its target, as text. | target = input moved one character left |
update_ratio(c) | After the first Adam step, how far weights moved relative to their size. Not Module 8's rule: that 0.001 to 0.1 range is for plain SGD steps. Adam's first step moves every weight by about the learning rate whatever its gradient, and these weights start near 0.02, so 0.01 / 0.02 gives about 0.5. Compare a case with this healthy run, not with Module 8's range. | about 0.5 |
block_change(c) | How much the first block changes each vector at the start. | about 0.3 |
Words and symbols in this lesson
- ln(V)
- the starting loss of a model that guesses evenly over V symbols: 4.55 for 95 characters
- block change
- how much one block changes a vector at the start; near 1 means it replaced it
Fill in diagnosis (case to fault name) and fixes (case to a dictionary of config changes) for cases A, B and C. After your fix, every setting must match the healthy run.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.