Syllabus Lesson 188 of 246 · Module 17: Noise in, pictures out (diffusion)
Module 17: Noise in, pictures out (diffusion)

A small U-Net

Why it matters For pictures, the noise predictor in real diffusion models is a U-Net: it squeezes the picture down to see the big shapes, then grows it back up to full size, with shortcuts that carry the fine detail across. You know every piece already.

A U-Net for 8 x 8 pictures with one channel:

  1. Down: 3 x 3 conv to 16 channels, ReLU. Keep this as skip (8 x 8 x 16). Then 2 x 2 average pool to 4 x 4.
  2. Middle: 3 x 3 conv to 32 channels, ReLU (4 x 4 x 32): each unit now sees a wide area.
  3. Up: upsample to 8 x 8, then join it with skip along the channels (E.concat): 8 x 8 x 48.
  4. 3 x 3 conv to 16 channels, ReLU, then a 3 x 3 conv to 1 channel: the predicted noise, same size as the input.

The skip connection is the U-Net's key idea: the down path throws away detail to see shape, and the skip hands that detail straight to the up path. Without it, the output is blurry.

No training here: you build it and prove it is wired right, with shapes and a gradient check, the way you checked layers in Module 4.

Words and symbols in this lesson
diffusion model
a network trained to remove noise a little at a time; run backwards from pure noise, it generates
U-Net
a network that shrinks a picture to see shapes, grows it back, and passes detail across through skip connections
Your turn

Write unet(x, P) for x an E.T of shape (n, 8, 8, 1), with weights in P as made by the starter. Then check one gradient by nudging: set max_gap to the largest difference between backprop and nudging over the first 5 numbers of P['up2'][0]. The engine works in float32, so nudge by h = 1e-3 or more: with 1e-4 the change in the loss is lost in rounding and a correct gradient looks wrong.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.