Convolution backward
Why it matters To learn its filters, a CNN needs the gradient of the loss with respect to every filter weight. It turns out to be another convolution.
Tiny numbers, in 1D first
Input x = [1, 2, 3, 4], filter w = [10, 20], valid output y = [1*10 + 2*20, 2*10 + 3*20, 3*10 + 4*20] = [50, 80, 110]. Weight w[0] touched x[0], x[1], x[2] (once per output), so its gradient is g[0]*x[0] + g[1]*x[1] + g[2]*x[2]: the input slid against the arriving gradient g. Input x[1] was used by y[0] (with w[1]) and y[1] (with w[0]), so its gradient collects g[0]*w[1] + g[1]*w[0].
The rules
- Filter gradient: slide the input against the arriving gradient (a valid correlation of x with g).
- Input gradient: pad g and slide the filter, flipped, over it (a full convolution).
Same idea in 2D. Conv2d unlocks in the house library after this lesson.
Words and symbols in this lesson
- convolution
- sliding a small grid of weights (a filter) across an image, one dot product per position
- filter, feature map
- the small learned grid of weights; and the image of its responses
Write conv1d(x, w) (valid) and conv1d_backward(x, w, g) returning (dx, dw). Then conv2d_backward(img, k, g) returning (dimg, dk) for the 2D valid convolution. The hidden test checks every gradient against a nudge.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.