Syllabus Lesson 239 of 246 · Module 23: Capstone studio: a model card you can run
Module 23: Capstone studio: a model card you can run

Find and fix the planted bug

Why it matters The worst bugs in machine learning do not crash. They train, a little worse than they should, and nobody notices. A gradient check finds them in seconds.

A colleague rewrote the studio's gradient code to make it faster: suspect_grads. You do not get to read it, as often in real work (a library, a teammate's branch, generated code). Trained with it, your baseline still learns: on the course's default plan the language score is only about 0.05 worse, and on images, with seed 0, it even scores higher. Both are inside what a careless reader calls noise. Something in it is wrong.

Check it the way Module 3 taught: nudge one weight by a small amount, measure the change in the loss, and compare with the slope the code claims. Do this for 5 entries of each of W1, b1, W2 and b2 on a small model (width 8, a batch of 16), picked among the entries whose claimed slope is not zero (with one-hot inputs, most of W1's slopes are exactly 0, and a check there proves nothing). Measure the loss in float64 with studio.loss64 and a nudge of 1e-4; call a relative error above 0.01 wrong.

Words and symbols in this lesson
baseline
the plain run every change is measured against, over the same seeds
gradient check
comparing the slope your code computes with the slope measured by nudging a weight
Your turn

Write check(grad_fn) returning, for each of 'W1', 'b1', 'W2', 'b2', the largest relative error over 5 entries. Set wrong to the sorted names whose error is above 0.01, and cause to one sentence: what in the backward pass sits where the wrong gradients are, and what was probably dropped there. Then write fixed_grads yourself (the forward pass is m.forward; the loss and its slope are studio.loss_and_dout). Then train your baseline (seed 0) with each and print both scores.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.