Syllabus Lesson 96 of 246 · Module 8: Run Clinic I
Module 8: Run Clinic I

Clinic: learning-rate faults

Why it matters The learning rate is the most common cause of a bad run. Each kind of wrong leaves a different mark.

Too high: steps overshoot, the loss jumps around or climbs, and the update-to-weight ratio is near or above 1. Too low: almost nothing moves, the loss sits near 2.30 and the ratio is tiny. Dropped too early: the run learns for a while, then stops improving: from then on the loss only wobbles from batch to batch around a mediocre level (here between about 1.1 and 1.5); the schedule shows the rate collapsing.

Each case below is a training run with one fault planted in it. For each: spend probes to find the fault, then write a fix as config changes. The faults in this lesson: lr_too_high, lr_too_low, lr_dropped_early. Fewer probes is better; a good diagnostician rarely needs more than two per case.

ProbeWhat it tells you
loss_curve(c)Training loss every 50 steps and final validation accuracy. Always look first.
loss_at_init(c)Loss before training. A fresh 10-class model should be near ln(10) = 2.30.
overfit_one_batch(c)Can it memorise 32 examples? A healthy setup gets near 0.
grad_norms(c)Gradient size per weight matrix at the start. Zero means nothing can learn.
update_ratio(c)Plain SGD step size relative to weight size. Healthy is about 0.001 to 0.1 (a rule of thumb for plain SGD; Module 14 shows why Adam differs).
act_stats(c)Hidden activations: mean, largest value, share of dead units.
lr_schedule(c)The learning rate at steps 0, 50, 100 and so on.
batch_labels(c)The labels in the first few batches.
overlap(c)How many validation examples also appear in training.
Words and symbols in this lesson
probe
a quick measurement of a run, such as gradient sizes or dead units
update-to-weight ratio
how much one step changes each weight, relative to its size
Your turn

Fill in diagnosis (case to fault name) and fixes (case to a dictionary of config changes) for cases A, B and C. A fix must remove the fault and train to at least 90% validation accuracy.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.