How far a code is from the centre: KL between two bell curves
Why it matters Pull 2 needs a number to minimise: how far each code's cloud N(mu, spread) is from the target N(0, 1). That number is the KL divergence you met in Module 2, now for bell curves.
Module 2 gave the exact formula for two bell curves: KL between N(m1, s1) and N(m2, s2) is
log(s2 / s1) + (s1² + (m1 - m2)²) / (2 s2²) - 1/2.
With the target N(0, 1) (m2 = 0, s2 = 1) and the spread written as logvar = log(s1²), it becomes 0.5 (mu² + exp(logvar) - 1 - logvar) per code number. Add it up over the code's numbers.
Tiny numbers
mu = 0, logvar = 0 (spread 1): 0.5 (0 + 1 - 1 - 0) = 0, the code already matches the target. mu = 2, logvar = 0: 0.5 (4 + 1 - 1 - 0) = 2. mu = 0, logvar = -2: 0.5 (0 + 0.135 - 1 + 2) = 0.568, so a code that is too sure of itself also pays.
Check it without the formula
KL is an average: the mean of log p(x) - log q(x) for x drawn from p. Drawing many x and averaging gives the same number, a little noisy. Two methods agreeing is how you know the code is right.
Words and symbols in this lesson
- code (latent)
- the few numbers the encoder makes from one picture
- mu, logvar
- the centre and the log of the squared spread of a code number in a VAE
Write kl_normal(m1, s1, m2, s2) from the formula, kl_to_standard(mu, logvar) (NumPy arrays of shape (N, z), returning the sum over code numbers averaged over the N rows), and kl_by_sampling(m1, s1, m2, s2, n) that averages log p(x) - log q(x) over n draws from p.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.