How different are two guesses (KL divergence)
Why it matters Later you will push one distribution towards another: a generator towards real data, a small model towards a big one. KL divergence measures the gap.
Tiny numbers
Truth P = [0.5, 0.5], a guess Q = [0.9, 0.1]. KL(P || Q) = 0.5 log(0.5/0.9) + 0.5 log(0.5/0.1) = -0.294 + 0.805 = 0.511. Swap them: KL(Q || P) = 0.9 log(0.9/0.5) + 0.1 log(0.1/0.5) = 0.529 - 0.161 = 0.368. Different numbers: KL is not symmetric, so the order matters.
KL divergence: the extra surprise you suffer, on average, by using Q when the truth is P. It is 0 only when they are identical, and never negative.
For two bell curves there is an exact formula, which Module 15 uses for its generative models: KL between N(m1, s1) and N(m2, s2) is log(s2/s1) + (s1 squared + (m1 - m2) squared) / (2 s2 squared) - 1/2.
Recap A gap measure, zero when equal, and order matters.
Words and symbols in this lesson
- KL(P || Q)
- how different two distributions are; zero only when they match; order matters
Write kl(p, q) for two probabilities given as lists or arrays (the tests pass plain lists, so turn them into arrays first), and kl_gauss(m1, s1, m2, s2) using the formula above.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.