Syllabus Lesson 38 of 246 · Module 2: Maths and probability, only as needed
Module 2: Maths and probability, only as needed

Gate: the maths, in code (AI off)

Why it matters Three pieces of this module come back in every later one: a softmax that survives large numbers, the gradients of a matrix multiply, and the KL divergence. Already know this maths? You can take this gate first, before the other lessons: passing it marks this module done and opens the next one.

AI tutor off. For the softmax, use the shift trick: subtracting the largest score from every score changes nothing on paper, but keeps every exp at most 1. Real logits can be large, and a softmax that returns nan on big inputs silently ruins a training run.

Words and symbols in this lesson
derivative (df/dx)
how fast the output changes per unit nudge of the input
softmax
exp of each score divided by the total: scores become probabilities
cross-entropy
minus the log of the probability given to the right answer
KL(P || Q)
how different two distributions are; zero only when they match; order matters
Your turn

Write, for 2D arrays of logits (one row per example):

  • logsumexp(z): log of the sum of exp along each row, safe for huge values.
  • stable_softmax(z): softmax along each row, safe for huge values.
  • nll(z, labels): the mean cross-entropy computed straight from logits as logsumexp - z[true], never taking the log of a softmax.
  • matmul_grads(X, W, G): for Y = X @ W and a loss whose gradient with respect to Y is G, return (dX, dW), the gradients with respect to X and W, without building any Jacobian.
  • kl_normal(m1, s1, m2, s2): the KL divergence from the bell curve N(m1, s1) to N(m2, s2), where s1 and s2 are spreads (standard deviations).

The checks use new random shapes and numbers every time, and compare with nudging and with a numerical integral.

AI off: no tutor, hints or worked solution on this one. Your own thinking only.

Spotted a problem in this lesson? Report it · Ask about this lesson

Code · runs in your browser
Output

        

Your progress is counted anonymously. Change this