Syllabus Lesson 119 of 246 · Module 10: Networks that see (CNNs)
Module 10: Networks that see (CNNs)

A small CNN on digits

Why it matters This is LeNet's idea from 1998: convolution, pooling, convolution, pooling, then a classifier. Build it from the house layers you earned and see it beat the MLP on shifted digits.

The house library nnsim.engine has the layers you built: conv (3 by 3, same padding), relu, pool2 (2 by 2 average), gap (global average pooling), matmul, cross_entropy and Adam. A conv weight for 3 by 3 filters from C_in to C_out channels has shape (9 * C_in, C_out).

The house library, in one card (keep it handy for the rest of the course; print(E.GUIDE) shows it again in any lesson)

  • E.T(array) wraps numbers as a tensor. .data is the array (float32); .grad is None until backward() reaches it.
  • loss.backward() fills .grad on every tensor the loss depends on.
  • opt = E.Adam(params, lr) takes a list of tensors. opt.step() updates every one that has a gradient and then sets its .grad back to None, so you never zero gradients yourself.
  • Images are channels last: (batch, rows, columns, channels). E.conv(x, w) keeps the size (same padding) and turns C channels into F with w of shape (9 * C, F). E.pool2 halves rows and columns; E.gap averages them away, giving (batch, channels).
  • E.cross_entropy(logits, y): logits (batch, classes) as a tensor, y a plain integer array. It returns a tensor holding one number; float(loss.data) reads it.
  • A whole training step is three lines: loss = E.cross_entropy(forward(Xb), yb), loss.backward(), opt.step().
  • Recorded runs: print(R.experiment(name)) lists its settings and the names of its metrics; run = exp.run(setting=value, ...), then run.loss (pairs of step and loss), run.metrics (a dictionary) and run.final_loss.

Call the Curve with the chart: guess the training loss after epoch 1 and after epoch 8 (the last).

Words and symbols in this lesson
convolution
sliding a small grid of weights (a filter) across an image, one dot product per position
filter, feature map
the small learned grid of weights; and the image of its responses
padding (p), stride (s)
zeros added around the edge; how many pixels the filter moves each step
pooling
shrinking a map by keeping the max or average of each block
RECORDED
a real run the course trained once and ships for you to study
Your turn

Fill my_guess, then write train(epochs, seed) returning a function predict(X). Use conv(1 to 16) + relu, conv(16 to 32) + relu, global average pooling, then a linear layer to 10 classes. Reach at least 90% on the test digits, and at least 60% on test digits shifted by 2 pixels (where the MLP scored about 21%). The shifted score moves a lot from seed to seed (the course's own answer gets 0.76 to 0.89 over five seeds), so the bar sits well below that. Adam at a learning rate of about 1e-2 works here; with 3e-3 this small network learns too slowly for the lesson.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.