Your own probes
Why it matters Probes are how you see inside a run. Writing them yourself means you will know what the numbers mean when a tool shows them to you.
Three probes do most of the work:
- Gradient size per layer: the length (norm) of each weight matrix's gradient. All zero: nothing learns. Wildly different between layers: one layer is starved.
- Dead share: the fraction of hidden units that output zero for every example in a batch.
- Update-to-weight ratio: learning rate times gradient size, divided by weight size. It says what fraction of each weight one step changes. About 0.001 to 0.1 is healthy; near 1 means each step rewrites the weights; near 0 means nothing moves.
The harness exposes its internals: _data(c) gives the training data, _init(c) the starting weights, and _fb(p, X, y, c) returns (loss, gradients, hidden activations).
Words and symbols in this lesson
- probe
- a quick measurement of a run, such as gradient sizes or dead units
- update-to-weight ratio
- how much one step changes each weight, relative to its size
Write my_grad_norms(c), my_dead_share(c) and my_update_ratio(c) using the first 256 training examples and the starting weights. They must match the clinic's probes.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.