Syllabus Lesson 121 of 246 · Module 10: Networks that see (CNNs)
Module 10: Networks that see (CNNs)

What the network looks at, tested

Why it matters Heat maps of "what the network looked at" are everywhere, and often misleading. Here you make one and then test whether it means anything.

A saliency map is the gradient of the winning class's score with respect to each input pixel: how much that pixel's brightness would change the answer. Your engine computes it for free: make the input a Tensor and call backward.

Test it, do not trust it

If the map really marks the pixels that matter, then blanking the 20 most salient pixels should hurt the network's confidence more than blanking 20 random pixels. That is a deletion test. A map that fails it is decoration.

But beating random pixels is a low bar. On these shapes, the bright pixels are the shape, so a map that simply marks the brightest pixels, with no network involved at all, also beats random. The fair question is whether the network's map beats that no-model baseline. If it does not, it has told you nothing a glance at the picture would not.

Baseline: the simplest thing that could explain a result. A method earns trust only by beating the best cheap baseline, not the worst one.

Words and symbols in this lesson
saliency map
the gradient of the winning score with respect to each input pixel
RECORDED
a real run the course trained once and ships for you to study
Your turn

A CNN the course trained once is loaded for you (a recorded network, 96% on new shapes). Write saliency(x) for one image x of shape (16, 16, 1), returning |gradient of the winning score| per pixel as shape (16, 16). To take the gradient of one score, pick its column with E.cols(scores, k, k + 1), add it up with E.total, and call backward on that. Then deletion_drop(x, pixels), where pixels is a list of (row, column) pairs: the drop in the winning class's probability when those pixels are set to 0. Finally measure, over the first 100 test images, the average drop for the top-20 salient pixels and for the 20 brightest pixels, and set beats_brightness to True or False from your own numbers. The test also checks your map against 20 random pixels.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.