Learning to predict the noise
Why it matters The whole training recipe for a diffusion model fits in five lines, and it is plain regression: show the network a noisy point and the step, ask for the noise. No game, no second network.
One training step:
- Take a batch of real points x0 and, for each, a random step t and fresh noise eps.
- Jump to x_t with the shortcut.
- Ask the network for the noise:
model.predict(x_t, t). - Loss = mean of (predicted noise - eps) squared.
- Backward, step.
Call the Curve
Before running, guess the loss at three moments: over the first 10 steps, around step 300, and around step 3000. Two anchors: a network that always predicts 0 scores exactly 1 (the noise has spread 1), and a perfect network cannot reach 0, because at very noisy steps the noise is easy to guess but at nearly clean steps it is almost impossible.
Where the field is now Predicting the noise is, up to a scale, the same as estimating the score: the direction in which the noisy data becomes more likely (Song and Ermon 2019; Ho et al. 2020). That is why the arrows point home. Real image generators do not add noise to pixels: they first squeeze each picture with an autoencoder, as in Module 15, and run diffusion on the small codes (latent diffusion, Rombach et al. 2022). Everything in this module carries over; only the space changes.
Words and symbols in this lesson
- diffusion model
- a network trained to remove noise a little at a time; run backwards from pure noise, it generates
- x0, x_t
- the clean data, and the same data after t steps of noise
- eps
- the bell-curve noise that was added; the network learns to predict it
- on shape, coverage
- the 2D judge: share of samples near the real shape, and share of the shape with a sample near it
Fill my_guess, then write the body of the loop (3000 steps of 256 spiral points; difflab.shape('spiral', 256, rng), difflab.add_noise). Append each step's loss to losses. Pass with a final loss under 0.40 and samples at least 0.75 on shape.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.