Syllabus Lesson 170 of 246 · Module 15: Squeeze, rebuild, generate (autoencoders and VAEs)
Module 15: Squeeze, rebuild, generate (autoencoders and VAEs)

How far a code is from the centre: KL between two bell curves

Why it matters Pull 2 needs a number to minimise: how far each code's cloud N(mu, spread) is from the target N(0, 1). That number is the KL divergence you met in Module 2, now for bell curves.

Module 2 gave the exact formula for two bell curves: KL between N(m1, s1) and N(m2, s2) is

log(s2 / s1) + (s1² + (m1 - m2)²) / (2 s2²) - 1/2.

With the target N(0, 1) (m2 = 0, s2 = 1) and the spread written as logvar = log(s1²), it becomes 0.5 (mu² + exp(logvar) - 1 - logvar) per code number. Add it up over the code's numbers.

Tiny numbers

mu = 0, logvar = 0 (spread 1): 0.5 (0 + 1 - 1 - 0) = 0, the code already matches the target. mu = 2, logvar = 0: 0.5 (4 + 1 - 1 - 0) = 2. mu = 0, logvar = -2: 0.5 (0 + 0.135 - 1 + 2) = 0.568, so a code that is too sure of itself also pays.

Check it without the formula

KL is an average: the mean of log p(x) - log q(x) for x drawn from p. Drawing many x and averaging gives the same number, a little noisy. Two methods agreeing is how you know the code is right.

Words and symbols in this lesson
code (latent)
the few numbers the encoder makes from one picture
mu, logvar
the centre and the log of the squared spread of a code number in a VAE
Your turn

Write kl_normal(m1, s1, m2, s2) from the formula, kl_to_standard(mu, logvar) (NumPy arrays of shape (N, z), returning the sum over code numbers averaged over the N rows), and kl_by_sampling(m1, s1, m2, s2, n) that averages log p(x) - log q(x) over n draws from p.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.