Syllabus Lesson 95 of 246 · Module 8: Run Clinic I
Module 8: Run Clinic I

Your own probes

Why it matters Probes are how you see inside a run. Writing them yourself means you will know what the numbers mean when a tool shows them to you.

Three probes do most of the work:

  • Gradient size per layer: the length (norm) of each weight matrix's gradient. All zero: nothing learns. Wildly different between layers: one layer is starved.
  • Dead share: the fraction of hidden units that output zero for every example in a batch.
  • Update-to-weight ratio: learning rate times gradient size, divided by weight size. It says what fraction of each weight one step changes. About 0.001 to 0.1 is healthy; near 1 means each step rewrites the weights; near 0 means nothing moves.

The harness exposes its internals: _data(c) gives the training data, _init(c) the starting weights, and _fb(p, X, y, c) returns (loss, gradients, hidden activations).

Words and symbols in this lesson
probe
a quick measurement of a run, such as gradient sizes or dead units
update-to-weight ratio
how much one step changes each weight, relative to its size
Your turn

Write my_grad_norms(c), my_dead_share(c) and my_update_ratio(c) using the first 256 training examples and the starting weights. They must match the clinic's probes.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.