A small CNN on digits
Why it matters This is LeNet's idea from 1998: convolution, pooling, convolution, pooling, then a classifier. Build it from the house layers you earned and see it beat the MLP on shifted digits.
The house library nnsim.engine has the layers you built: conv (3 by 3, same padding), relu, pool2 (2 by 2 average), gap (global average pooling), matmul, cross_entropy and Adam. A conv weight for 3 by 3 filters from C_in to C_out channels has shape (9 * C_in, C_out).
The house library, in one card (keep it handy for the rest of the course; print(E.GUIDE) shows it again in any lesson)
E.T(array)wraps numbers as a tensor..datais the array (float32);.gradisNoneuntilbackward()reaches it.loss.backward()fills.gradon every tensor the loss depends on.opt = E.Adam(params, lr)takes a list of tensors.opt.step()updates every one that has a gradient and then sets its.gradback toNone, so you never zero gradients yourself.- Images are channels last: (batch, rows, columns, channels).
E.conv(x, w)keeps the size (same padding) and turns C channels into F withwof shape (9 * C, F).E.pool2halves rows and columns;E.gapaverages them away, giving (batch, channels). E.cross_entropy(logits, y): logits (batch, classes) as a tensor, y a plain integer array. It returns a tensor holding one number;float(loss.data)reads it.- A whole training step is three lines:
loss = E.cross_entropy(forward(Xb), yb),loss.backward(),opt.step(). - Recorded runs:
print(R.experiment(name))lists its settings and the names of its metrics;run = exp.run(setting=value, ...), thenrun.loss(pairs of step and loss),run.metrics(a dictionary) andrun.final_loss.
Call the Curve with the chart: guess the training loss after epoch 1 and after epoch 8 (the last).
Words and symbols in this lesson
- convolution
- sliding a small grid of weights (a filter) across an image, one dot product per position
- filter, feature map
- the small learned grid of weights; and the image of its responses
- padding (p), stride (s)
- zeros added around the edge; how many pixels the filter moves each step
- pooling
- shrinking a map by keeping the max or average of each block
- RECORDED
- a real run the course trained once and ships for you to study
Fill my_guess, then write train(epochs, seed) returning a function predict(X). Use conv(1 to 16) + relu, conv(16 to 32) + relu, global average pooling, then a linear layer to 10 classes. Reach at least 90% on the test digits, and at least 60% on test digits shifted by 2 pixels (where the MLP scored about 21%). The shifted score moves a lot from seed to seed (the course's own answer gets 0.76 to 0.89 over five seeds), so the bar sits well below that. Adam at a learning rate of about 1e-2 works here; with 3e-3 this small network learns too slowly for the lesson.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.