Syllabus Lesson 171 of 246 · Module 15: Squeeze, rebuild, generate (autoencoders and VAEs)
Module 15: Squeeze, rebuild, generate (autoencoders and VAEs)

The VAE loss, step by step (the ELBO)

Why it matters "Rebuild well plus stay near the centre" sounds like two goals glued together. It is one goal: make the real digits as probable as possible under the model. Seeing why tells you what the loss number means.

A VAE is a story of how a digit is made: draw a code z from N(0, 1), then draw each pixel with the probability the decoder gives. The best model gives real digits the highest probability, log p(x). Computing log p(x) exactly needs an average over every possible z, which is out of reach.

The trick is a lower bound you can compute, the ELBO (evidence lower bound):

log p(x) ≥ E[log p(x | z)] - KL(q(z | x) || N(0, 1))

Here q(z | x) = N(mu, spread) is the encoder's guess of which codes could have made x, and E[...] averages over codes drawn from it. Term one is the rebuild score. Term two is pull 2. The gap between log p(x) and the ELBO is exactly how wrong the encoder's guess is (a KL to the true posterior), so pushing the ELBO up raises log p(x), or closes the gap (makes the guess better), or both: the ELBO alone cannot tell you which.

Training minimises -ELBO, which is rebuild loss + KL: the very loss in gen.train_vae.

Fill the gaps on a tiny example

A 4-pixel picture x = [1, 0, 1, 1]. The encoder gives mu = [0.5, -1.0] and logvar = [0.0, -1.0]. One draw of the code gives decoder probabilities p = [0.9, 0.2, 0.7, 0.6] (each pixel's chance of ink).

Words and symbols in this lesson
code (latent)
the few numbers the encoder makes from one picture
VAE
variational autoencoder: an autoencoder whose codes are pulled to fill N(0, 1), so random codes decode to new pictures
mu, logvar
the centre and the log of the squared spread of a code number in a VAE
ELBO
evidence lower bound: rebuild score minus KL; training a VAE pushes it up
Your turn

Fill each step: log_p_x_given_z (sum over pixels of x log p + (1 - x) log(1 - p)), kl (to N(0, 1), summed over both code numbers), elbo, and loss (what training minimises). Then set gap_means to the letter that says what the gap between log p(x) and the ELBO measures.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.