The VAE loss, step by step (the ELBO)
Why it matters "Rebuild well plus stay near the centre" sounds like two goals glued together. It is one goal: make the real digits as probable as possible under the model. Seeing why tells you what the loss number means.
A VAE is a story of how a digit is made: draw a code z from N(0, 1), then draw each pixel with the probability the decoder gives. The best model gives real digits the highest probability, log p(x). Computing log p(x) exactly needs an average over every possible z, which is out of reach.
The trick is a lower bound you can compute, the ELBO (evidence lower bound):
log p(x) ≥ E[log p(x | z)] - KL(q(z | x) || N(0, 1))
Here q(z | x) = N(mu, spread) is the encoder's guess of which codes could have made x, and E[...] averages over codes drawn from it. Term one is the rebuild score. Term two is pull 2. The gap between log p(x) and the ELBO is exactly how wrong the encoder's guess is (a KL to the true posterior), so pushing the ELBO up raises log p(x), or closes the gap (makes the guess better), or both: the ELBO alone cannot tell you which.
Training minimises -ELBO, which is rebuild loss + KL: the very loss in gen.train_vae.
Fill the gaps on a tiny example
A 4-pixel picture x = [1, 0, 1, 1]. The encoder gives mu = [0.5, -1.0] and logvar = [0.0, -1.0]. One draw of the code gives decoder probabilities p = [0.9, 0.2, 0.7, 0.6] (each pixel's chance of ink).
Words and symbols in this lesson
- code (latent)
- the few numbers the encoder makes from one picture
- VAE
- variational autoencoder: an autoencoder whose codes are pulled to fill N(0, 1), so random codes decode to new pictures
- mu, logvar
- the centre and the log of the squared spread of a code number in a VAE
- ELBO
- evidence lower bound: rebuild score minus KL; training a VAE pushes it up
Fill each step: log_p_x_given_z (sum over pixels of x log p + (1 - x) log(1 - p)), kl (to N(0, 1), summed over both code numbers), elbo, and loss (what training minimises). Then set gap_means to the letter that says what the gap between log p(x) and the ELBO measures.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.