Syllabus Lesson 37 of 246 · Module 2: Maths and probability, only as needed
Module 2: Maths and probability, only as needed

How different are two guesses (KL divergence)

Why it matters Later you will push one distribution towards another: a generator towards real data, a small model towards a big one. KL divergence measures the gap.

Tiny numbers

Truth P = [0.5, 0.5], a guess Q = [0.9, 0.1]. KL(P || Q) = 0.5 log(0.5/0.9) + 0.5 log(0.5/0.1) = -0.294 + 0.805 = 0.511. Swap them: KL(Q || P) = 0.9 log(0.9/0.5) + 0.1 log(0.1/0.5) = 0.529 - 0.161 = 0.368. Different numbers: KL is not symmetric, so the order matters.

KL divergence: the extra surprise you suffer, on average, by using Q when the truth is P. It is 0 only when they are identical, and never negative.

For two bell curves there is an exact formula, which Module 15 uses for its generative models: KL between N(m1, s1) and N(m2, s2) is log(s2/s1) + (s1 squared + (m1 - m2) squared) / (2 s2 squared) - 1/2.

Recap A gap measure, zero when equal, and order matters.

Words and symbols in this lesson
KL(P || Q)
how different two distributions are; zero only when they match; order matters
Your turn

Write kl(p, q) for two probabilities given as lists or arrays (the tests pass plain lists, so turn them into arrays first), and kl_gauss(m1, s1, m2, s2) using the formula above.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.