Adding noise, step by step
Why it matters The image generators behind today's AI art work backwards from noise. To learn to remove noise, a network first needs to see exactly how noise was added. That forward half is pure arithmetic, and it has a shortcut that makes training possible.
Take a clean point x0 (a pixel value, or a point in 2D). At each of T = 100 steps, shrink it a little and add a little fresh noise:
x_t = sqrt(1 - beta_t) x_(t-1) + sqrt(beta_t) eps, with eps drawn from N(0, 1)
The list beta_0 ... beta_99 is the noise schedule. Watch the scene: a spiral of points dissolves into a round cloud of noise.
The shortcut
Shrinking and adding bell-curve noise again and again is the same as doing it once with the totals. Write abar_t for the product of all the (1 - beta) factors up to step t. Then
x_t = sqrt(abar_t) x0 + sqrt(1 - abar_t) eps
in one go: the signal left is sqrt(abar_t), the noise spread is sqrt(1 - abar_t). Training can jump to any step instantly, instead of walking there.
Tiny numbers
Two steps with beta = 0.1 each: abar = 0.9 x 0.9 = 0.81. From x0 = 1 the signal left is sqrt(0.81) = 0.9 and the noise spread is sqrt(0.19) = 0.44. Walking: after step 1, 0.949 x 1 plus noise of spread 0.316; after step 2, everything shrinks by 0.949 again and fresh noise of spread 0.316 joins. The spreads add as squares: 0.949^2 x 0.1 + 0.1 = 0.19. Same answer.
Words and symbols in this lesson
- noise schedule (beta)
- how much fresh noise is mixed in at each step
- abar
- the share of the original signal left after a number of steps (its square root multiplies x0)
- x0, x_t
- the clean data, and the same data after t steps of noise
- eps
- the bell-curve noise that was added; the network learns to predict it
Write walk(x0, t, beta, rng) (one step at a time, t + 1 steps from step 0 to step t) and jump(x0, t, abar, eps) (the shortcut). Then noise 20000 copies of the point (1.0, -0.5) to step 40 both ways and fill stats with each version's mean and spread (per coordinate) next to what the schedule says they should be.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.