Syllabus Lesson 115 of 246 · Module 10: Networks that see (CNNs)
Module 10: Networks that see (CNNs)

Convolution backward

Why it matters To learn its filters, a CNN needs the gradient of the loss with respect to every filter weight. It turns out to be another convolution.

Tiny numbers, in 1D first

Input x = [1, 2, 3, 4], filter w = [10, 20], valid output y = [1*10 + 2*20, 2*10 + 3*20, 3*10 + 4*20] = [50, 80, 110]. Weight w[0] touched x[0], x[1], x[2] (once per output), so its gradient is g[0]*x[0] + g[1]*x[1] + g[2]*x[2]: the input slid against the arriving gradient g. Input x[1] was used by y[0] (with w[1]) and y[1] (with w[0]), so its gradient collects g[0]*w[1] + g[1]*w[0].

The rules

  • Filter gradient: slide the input against the arriving gradient (a valid correlation of x with g).
  • Input gradient: pad g and slide the filter, flipped, over it (a full convolution).

Same idea in 2D. Conv2d unlocks in the house library after this lesson.

Words and symbols in this lesson
convolution
sliding a small grid of weights (a filter) across an image, one dot product per position
filter, feature map
the small learned grid of weights; and the image of its responses
Your turn

Write conv1d(x, w) (valid) and conv1d_backward(x, w, g) returning (dx, dw). Then conv2d_backward(img, k, g) returning (dimg, dk) for the 2D valid convolution. The hidden test checks every gradient against a nudge.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.