What the network looks at, tested
Why it matters Heat maps of "what the network looked at" are everywhere, and often misleading. Here you make one and then test whether it means anything.
A saliency map is the gradient of the winning class's score with respect to each input pixel: how much that pixel's brightness would change the answer. Your engine computes it for free: make the input a Tensor and call backward.
Test it, do not trust it
If the map really marks the pixels that matter, then blanking the 20 most salient pixels should hurt the network's confidence more than blanking 20 random pixels. That is a deletion test. A map that fails it is decoration.
But beating random pixels is a low bar. On these shapes, the bright pixels are the shape, so a map that simply marks the brightest pixels, with no network involved at all, also beats random. The fair question is whether the network's map beats that no-model baseline. If it does not, it has told you nothing a glance at the picture would not.
Baseline: the simplest thing that could explain a result. A method earns trust only by beating the best cheap baseline, not the worst one.
Words and symbols in this lesson
- saliency map
- the gradient of the winning score with respect to each input pixel
- RECORDED
- a real run the course trained once and ships for you to study
A CNN the course trained once is loaded for you (a recorded network, 96% on new shapes). Write saliency(x) for one image x of shape (16, 16, 1), returning |gradient of the winning score| per pixel as shape (16, 16). To take the gradient of one score, pick its column with E.cols(scores, k, k + 1), add it up with E.total, and call backward on that. Then deletion_drop(x, pixels), where pixels is a list of (row, column) pairs: the drop in the winning class's probability when those pixels are set to 0. Finally measure, over the first 100 test images, the average drop for the top-20 salient pixels and for the 20 brightest pixels, and set beats_brightness to True or False from your own numbers. The test also checks your map against 20 random pixels.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.