Syllabus Lesson 169 of 246 · Module 15: Squeeze, rebuild, generate (autoencoders and VAEs)
Module 15: Squeeze, rebuild, generate (autoencoders and VAEs)

Two pulls on every code (See it)

Why it matters The fix is to make the codes fill the space in a known shape, so that random points from that shape land where the decoder has learned to draw. A variational autoencoder (VAE) does it with two pulls on every code.

Watch the scene first. Each character is one test digit, drawn where its 2 code numbers put it, as the digit itself; the 7s are in colour so you can follow one group.

  • Pull 1, rebuild well: codes of different digits must stay apart, or the decoder cannot tell them apart.
  • Pull 2, stay near the centre: every code is pulled towards the centre and made a little blurry (a small cloud, not a point), so the codes fill the bell curve N(0, 1) around the centre without gaps.

Sampling then means: draw a point from N(0, 1) and decode it. Because the codes fill that region, the point lands somewhere the decoder knows.

In the VAE the encoder gives two things per code number: a centre mu and a spread, written as logvar (the log of the squared spread, which can be any number). The code is a draw from N(mu, spread). gen.train_vae trains one; rebuild_weight and weight set the strength of each pull.

Predict first: if you remove pull 2, what happens to the codes? If you remove pull 1?

Words and symbols in this lesson
autoencoder
an encoder that squeezes a picture into a few numbers and a decoder that rebuilds it
code (latent)
the few numbers the encoder makes from one picture
VAE
variational autoencoder: an autoencoder whose codes are pulled to fill N(0, 1), so random codes decode to new pictures
mu, logvar
the centre and the log of the squared spread of a code number in a VAE
Your turn

Set guess for both removals to one of "spread out", "all at the centre" or "no change". Then train three 2-number VAEs (a second or two each in the browser) and fill spread with the mean distance of the test codes from the centre, and noise with the mean spread exp(logvar / 2), for each.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.