Syllabus Lesson 173 of 246 · Module 15: Squeeze, rebuild, generate (autoencoders and VAEs)
Module 15: Squeeze, rebuild, generate (autoencoders and VAEs)

Your VAE makes new digits

Why it matters Every piece is ready: the encoder's mu and logvar, the trick to sample through, the rebuild loss and the KL. Put them in one training step and you have a model that makes digits nobody drew.

One step of training, in words:

  1. Encode a batch: mu, logvar = model.encode(E.T(x)).
  2. Draw eps from N(0, 1) with the shape of mu, and make the code with your reparameterisation.
  3. Decode the code to pixel scores: model.decode_logits(code).
  4. Loss = E.bce_with_logits(scores, x) + E.kl_to_standard(mu, logvar) (the house engine's version of your KL).
  5. Backward, then opt.step().

Then judge it with the same judge as before: closeness and variety of 500 new digits drawn from N(0, 1). The rebuild error will be worse than the autoencoder's (about 0.03 against 0.008). That is the price of pull 2, and it buys a code space you can sample from.

Words and symbols in this lesson
autoencoder
an encoder that squeezes a picture into a few numbers and a decoder that rebuilds it
code (latent)
the few numbers the encoder makes from one picture
VAE
variational autoencoder: an autoencoder whose codes are pulled to fill N(0, 1), so random codes decode to new pictures
mu, logvar
the centre and the log of the squared spread of a code number in a VAE
reparameterisation
code = mu + exp(logvar / 2) x eps, with eps drawn first, so gradients reach mu and logvar
closeness, variety
the course judge: distance to the nearest real digit, and bits of spread over the ten digits
Your turn

Write the body of the training loop (600 steps of 256 pictures). Pass with: test rebuild error under 0.035, new digits with closeness at most 1.30 and variety at least 3.0, and at least 3 active code numbers. Use your own loop: gen.train_vae is off limits here.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.