The recipe: two checks before anything else
Why it matters Most broken runs can be caught in a minute by two cheap checks, before you spend an hour training.
Experienced practitioners start every new setup the same way:
- Check the loss at the start. Before training, a 10-class model that knows nothing gives every class about 0.1, so its loss should be about -ln(0.1) = 2.30. Far above that means something is scaled wrong. Far below means the answer is reaching the model through its inputs. Not every leak shows here: when validation images also sit in the training set, the starting loss is normal (this clinic's healthy run and its leak case both start at 2.46), and only a validation score that is too good gives it away.
- Overfit one batch. Train on a single small batch for a few hundred steps. Any working model and training loop can memorise 32 examples and drive the loss near zero. If it cannot, the bug is in the model, the loss or the gradients, and you have found it without touching the full data.
- Only then, scale up to the full run.
This clinic is a small digit classifier you can break on purpose. config(**changes) makes a setup; train(c) runs it.
| Probe | What it tells you |
|---|---|
loss_curve(c) | Training loss every 50 steps and final validation accuracy. Always look first. |
loss_at_init(c) | Loss before training. A fresh 10-class model should be near ln(10) = 2.30. |
overfit_one_batch(c) | Can it memorise 32 examples? A healthy setup gets near 0. |
grad_norms(c) | Gradient size per weight matrix at the start. Zero means nothing can learn. |
update_ratio(c) | Plain SGD step size relative to weight size. Healthy is about 0.001 to 0.1 (a rule of thumb for plain SGD; Module 14 shows why Adam differs). |
act_stats(c) | Hidden activations: mean, largest value, share of dead units. |
lr_schedule(c) | The learning rate at steps 0, 50, 100 and so on. |
batch_labels(c) | The labels in the first few batches. |
overlap(c) | How many validation examples also appear in training. |
Words and symbols in this lesson
- probe
- a quick measurement of a run, such as gradient sizes or dead units
- overfit one batch
- checking a setup can memorise a few examples
Write sanity(c) returning a dictionary with init_ok (loss at the start within 0.3 of 2.30) and can_overfit (overfit-one-batch loss below 0.1), using the probes. Apply it to the healthy setup and to suspect.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.