Syllabus Lesson 182 of 246 · Module 17: Noise in, pictures out (diffusion)
Module 17: Noise in, pictures out (diffusion)

Adding noise, step by step

Why it matters The image generators behind today's AI art work backwards from noise. To learn to remove noise, a network first needs to see exactly how noise was added. That forward half is pure arithmetic, and it has a shortcut that makes training possible.

Take a clean point x0 (a pixel value, or a point in 2D). At each of T = 100 steps, shrink it a little and add a little fresh noise:

x_t = sqrt(1 - beta_t) x_(t-1) + sqrt(beta_t) eps, with eps drawn from N(0, 1)

The list beta_0 ... beta_99 is the noise schedule. Watch the scene: a spiral of points dissolves into a round cloud of noise.

The shortcut

Shrinking and adding bell-curve noise again and again is the same as doing it once with the totals. Write abar_t for the product of all the (1 - beta) factors up to step t. Then

x_t = sqrt(abar_t) x0 + sqrt(1 - abar_t) eps

in one go: the signal left is sqrt(abar_t), the noise spread is sqrt(1 - abar_t). Training can jump to any step instantly, instead of walking there.

Tiny numbers

Two steps with beta = 0.1 each: abar = 0.9 x 0.9 = 0.81. From x0 = 1 the signal left is sqrt(0.81) = 0.9 and the noise spread is sqrt(0.19) = 0.44. Walking: after step 1, 0.949 x 1 plus noise of spread 0.316; after step 2, everything shrinks by 0.949 again and fresh noise of spread 0.316 joins. The spreads add as squares: 0.949^2 x 0.1 + 0.1 = 0.19. Same answer.

Words and symbols in this lesson
noise schedule (beta)
how much fresh noise is mixed in at each step
abar
the share of the original signal left after a number of steps (its square root multiplies x0)
x0, x_t
the clean data, and the same data after t steps of noise
eps
the bell-curve noise that was added; the network learns to predict it
Your turn

Write walk(x0, t, beta, rng) (one step at a time, t + 1 steps from step 0 to step t) and jump(x0, t, abar, eps) (the shortcut). Then noise 20000 copies of the point (1.0, -0.5) to step 40 both ways and fill stats with each version's mean and spread (per coordinate) next to what the schedule says they should be.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.