Trust the helper?
Why it matters AI assistants will happily diagnose your training runs. They are often right and sometimes confidently wrong. The skill is knowing which, using evidence instead of tone.
An AI helper has looked at ten broken runs and named a fault for each. About a third of its answers are wrong, and they sound exactly as confident as the right ones. For each case, decide: accept or reject. Base every decision on a probe, not on how plausible the claim sounds.
Your score is verdict accuracy: how often your accept or reject matched the truth. (This is not calibration, which Module 9 defines: whether confidence matches how often you are right.) Look out for claims that fit some symptoms but not the decisive one: a stalled loss could be a learning rate that is too low, or one that was fine and then dropped.
| Probe | What it tells you |
|---|---|
loss_curve(c) | Training loss every 50 steps and final validation accuracy. Always look first. |
loss_at_init(c) | Loss before training. A fresh 10-class model should be near ln(10) = 2.30. |
overfit_one_batch(c) | Can it memorise 32 examples? A healthy setup gets near 0. |
grad_norms(c) | Gradient size per weight matrix at the start. Zero means nothing can learn. |
update_ratio(c) | Plain SGD step size relative to weight size. Healthy is about 0.001 to 0.1 (a rule of thumb for plain SGD; Module 14 shows why Adam differs). |
act_stats(c) | Hidden activations: mean, largest value, share of dead units. |
lr_schedule(c) | The learning rate at steps 0, 50, 100 and so on. |
batch_labels(c) | The labels in the first few batches. |
overlap(c) | How many validation examples also appear in training. |
Words and symbols in this lesson
- probe
- a quick measurement of a run, such as gradient sizes or dead units
- verdict accuracy
- how often your accept or reject of a claim matched the truth
For each case A to J, set verdicts[case] to True (the helper is right) or False. You need at least 9 of 10 right.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.