Syllabus Lesson 172 of 246 · Module 15: Squeeze, rebuild, generate (autoencoders and VAEs)
Module 15: Squeeze, rebuild, generate (autoencoders and VAEs)

Sampling you can train through

Why it matters The code is a random draw from N(mu, spread). Backprop needs a slope from the loss back to mu and logvar, and "a random number" has no slope. One rewrite fixes it, and it is used far beyond VAEs.

Draw the randomness first, from a fixed bell curve: eps from N(0, 1). Then build the code from it:

code = mu + exp(logvar / 2) * eps

The code has exactly the same spread of values as a draw from N(mu, spread), but now it is ordinary arithmetic on mu and logvar, with eps as a plain input. The slopes are simple: d code / d mu = 1, and d code / d logvar = 0.5 exp(logvar / 2) eps. This is the reparameterisation trick.

Tiny numbers

mu = 1, logvar = 0, eps = 0.5: code = 1 + 1 x 0.5 = 1.5. Slope to mu: 1. Slope to logvar: 0.5 x 1 x 0.5 = 0.25.

You need one new operation for it, exp. The course engine already has one (E.exp); here you build your own, the Module 4 way: forward value, and a backward rule that multiplies the incoming gradient by the local slope (for exp, the slope is exp itself).

In the house engine an operation is a tensor out = E.T(value, (parent,)) with a rule out.back = lambda g: parent._acc(g * local_slope): back receives the gradient arriving at out, and _acc adds a gradient into the parent. E.GUIDE prints the engine's pieces.

Words and symbols in this lesson
code (latent)
the few numbers the encoder makes from one picture
VAE
variational autoencoder: an autoencoder whose codes are pulled to fill N(0, 1), so random codes decode to new pictures
mu, logvar
the centre and the log of the squared spread of a code number in a VAE
reparameterisation
code = mu + exp(logvar / 2) x eps, with eps drawn first, so gradients reach mu and logvar
Your turn

Write exp_op(a) as an engine operation (return an E.T with parents and a back rule). Write reparam(mu, logvar, eps) with exp_op, E.scale, E.mul and E.add. Then check both gradients against nudging and set max_gap to the largest difference you find.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.