Sampling you can train through
Why it matters The code is a random draw from N(mu, spread). Backprop needs a slope from the loss back to mu and logvar, and "a random number" has no slope. One rewrite fixes it, and it is used far beyond VAEs.
Draw the randomness first, from a fixed bell curve: eps from N(0, 1). Then build the code from it:
code = mu + exp(logvar / 2) * eps
The code has exactly the same spread of values as a draw from N(mu, spread), but now it is ordinary arithmetic on mu and logvar, with eps as a plain input. The slopes are simple: d code / d mu = 1, and d code / d logvar = 0.5 exp(logvar / 2) eps. This is the reparameterisation trick.
Tiny numbers
mu = 1, logvar = 0, eps = 0.5: code = 1 + 1 x 0.5 = 1.5. Slope to mu: 1. Slope to logvar: 0.5 x 1 x 0.5 = 0.25.
You need one new operation for it, exp. The course engine already has one (E.exp); here you build your own, the Module 4 way: forward value, and a backward rule that multiplies the incoming gradient by the local slope (for exp, the slope is exp itself).
In the house engine an operation is a tensor out = E.T(value, (parent,)) with a rule out.back = lambda g: parent._acc(g * local_slope): back receives the gradient arriving at out, and _acc adds a gradient into the parent. E.GUIDE prints the engine's pieces.
Words and symbols in this lesson
- code (latent)
- the few numbers the encoder makes from one picture
- VAE
- variational autoencoder: an autoencoder whose codes are pulled to fill N(0, 1), so random codes decode to new pictures
- mu, logvar
- the centre and the log of the squared spread of a code number in a VAE
- reparameterisation
- code = mu + exp(logvar / 2) x eps, with eps drawn first, so gradients reach mu and logvar
Write exp_op(a) as an engine operation (return an E.T with parents and a back rule). Write reparam(mu, logvar, eps) with exp_op, E.scale, E.mul and E.add. Then check both gradients against nudging and set max_gap to the largest difference you find.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.