Syllabus Lesson 194 of 246 · Module 18: Learning without labels
Module 18: Learning without labels

Fill in the blank (masked prediction)

Why it matters The simplest way to make a task from unlabelled data: hide part of it and ask the network to fill it in. Language models are trained exactly this way (hide the next word); image models hide patches. To fill a hole well, a network has to understand what is around it.

Hide a 4 x 4 patch in the middle of each photo. The network sees the 8 x 8 window around it (the other 48 pixels, x 3 colours) and predicts the 16 hidden ones (x 3). The loss is the mean squared error. No labels anywhere: the photo is its own answer key.

Two simple fills to beat: the photo's average colour (error about 0.035) and the average of the pixels right around the hole (about 0.019). A network that beats the second has learned something about how shapes and light continue.

Words and symbols in this lesson
masked prediction
hiding part of the input and training the network to fill it in
Your turn

Write the body of the training loop (400 steps of 64 photos). Pass with a validation error under 0.017.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.