Syllabus Lesson 122 of 246 · Module 10: Networks that see (CNNs)
Module 10: Networks that see (CNNs)

Reusing a trained network (transfer learning)

Why it matters You rarely have lots of labelled data for your own task. A network trained on a big related dataset already knows useful features; reusing it lets a tiny dataset go a long way.

The plan: train the conv layers on a big, clean shapes dataset. Then face a new, harder task with only 40 labelled images: the same shapes, but much noisier. Keep the trained conv layers frozen and train only a new final layer on features averaged over the whole image (position-free features generalise from few examples). Compare with training the whole network from scratch on the 40 images.

Frozen: layers whose weights are not updated during training. Fine-tuning: continuing to train a pretrained network on new data.

Words and symbols in this lesson
transfer learning
reusing layers trained on one task for another
RECORDED
a real run the course trained once and ships for you to study
Your turn

The pretraining was done once by the course and its weights are loaded for you (a recorded network). Write train_head(w1, w2, X, y, steps) that trains only a new 16-to-4 output layer on top of frozen conv layers (Adam at 1e-2 works) and returns a predict function, and train_scratch(X, y, steps) that trains all three layers from new random weights. The test runs both on 1,000 noisy test images. Your transfer model must beat the course's own scratch network (fixed settings: 8 and 16 channels, Adam 3e-3, 300 steps, about 58%) by at least 8 points; the course's transfer run gets about 72%. Your own scratch network just has to learn something (above 35%, where chance is 25%). If yours beats the course's scratch network, well done: tuning scratch training well is a real skill, and it is also why a fair comparison fixes the settings in advance.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.