Syllabus Lesson 186 of 246 · Module 17: Noise in, pictures out (diffusion)
Module 17: Noise in, pictures out (diffusion)

Noise schedules: a lower loss, worse samples

Why it matters The loss is the number you watch, so it is tempting to pick whatever makes it lowest. Here that instinct is exactly wrong, and seeing why teaches you what the loss measures.

Two schedules for the same 100 steps:

  • Linear (the original DDPM paper): beta rises evenly from 0.0001 to 0.02 x (1000 / T). With only 100 steps, abar falls fast: most steps are nearly pure noise.
  • Cosine (Nichol and Dhariwal, 2021): abar follows a cosine curve, so the signal fades evenly across the steps.

The course trained both on the spiral, 3 seeds each (difflab.recorded()).

Predict first: which schedule ends with the lower training loss, and which makes samples that sit better on the spiral?

Words and symbols in this lesson
noise schedule (beta)
how much fresh noise is mixed in at each step
abar
the share of the original signal left after a number of steps (its square root multiplies x0)
eps
the bell-curve noise that was added; the network learns to predict it
DDPM
sampling by stepping back through every step, with fresh noise each time
on shape, coverage
the 2D judge: share of samples near the real shape, and share of the shape with a sample near it
Your turn

Set guess for both questions to "linear" or "cosine". Then fill table with each schedule's mean final loss and mean on-shape score over the 3 recorded seeds, and noisy_share: for each schedule, the share of the 100 steps where less than 10% of the signal's variance is left (abar under 0.1; the signal's size is sqrt(abar), about 0.32 there). Finally set shows, the same two questions answered from your table. Your guess stays as you made it.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.