Find and fix the planted bug
Why it matters The worst bugs in machine learning do not crash. They train, a little worse than they should, and nobody notices. A gradient check finds them in seconds.
A colleague rewrote the studio's gradient code to make it faster: suspect_grads. You do not get to read it, as often in real work (a library, a teammate's branch, generated code). Trained with it, your baseline still learns: on the course's default plan the language score is only about 0.05 worse, and on images, with seed 0, it even scores higher. Both are inside what a careless reader calls noise. Something in it is wrong.
Check it the way Module 3 taught: nudge one weight by a small amount, measure the change in the loss, and compare with the slope the code claims. Do this for 5 entries of each of W1, b1, W2 and b2 on a small model (width 8, a batch of 16), picked among the entries whose claimed slope is not zero (with one-hot inputs, most of W1's slopes are exactly 0, and a check there proves nothing). Measure the loss in float64 with studio.loss64 and a nudge of 1e-4; call a relative error above 0.01 wrong.
Words and symbols in this lesson
- baseline
- the plain run every change is measured against, over the same seeds
- gradient check
- comparing the slope your code computes with the slope measured by nudging a weight
Write check(grad_fn) returning, for each of 'W1', 'b1', 'W2', 'b2', the largest relative error over 5 entries. Set wrong to the sorted names whose error is above 0.01, and cause to one sentence: what in the backward pass sits where the wrong gradients are, and what was probably dropped there. Then write fixed_grads yourself (the forward pass is m.forward; the loss and its slope are studio.loss_and_dout). Then train your baseline (seed 0) with each and print both scores.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.