A gradient checker you will use all course
Why it matters A wrong gradient does not crash. Training just goes quietly badly. A gradient checker catches the mistake in seconds, and you will use this one in every module.
The idea is Module 2's nudge, done for every weight: nudge weight i up and down by h, measure the loss both times, and divide the difference by 2h. Compare with your formula's gradient.
How close is close?
Use the relative error: |a - b| divided by the larger of |a| and |b| (or 1e-8 if both are tiny). Below 1e-6 is a pass; above 1e-3 almost always means a bug.
Gradient check: comparing an analytic gradient (from your formula) with a numerical one (from nudging).
Recap Slow but trustworthy referee. Use it before trusting any new gradient.
Words and symbols in this lesson
- learning rate (lr)
- how big each training step is
- gradient check
- comparing your gradient formula with a nudge-and-measure estimate
Write numeric_grad(f, w, h=1e-5) (w is a NumPy array of any shape; return an array of the same shape) and grad_check(f, grad, w) returning the largest relative error. Here f(w) takes the weight array and returns the loss as one number, and grad is an array (your formula's gradient at w, already computed), not a function. Do not change w permanently. To visit every position of an array of any shape, for idx in np.ndindex(w.shape): gives each index tuple in turn. The hidden test feeds your checker three subtly wrong gradients; each must score above 1e-3.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.