Gate: the maths, in code (AI off)
Why it matters Three pieces of this module come back in every later one: a softmax that survives large numbers, the gradients of a matrix multiply, and the KL divergence. Already know this maths? You can take this gate first, before the other lessons: passing it marks this module done and opens the next one.
AI tutor off. For the softmax, use the shift trick: subtracting the largest score from every score changes nothing on paper, but keeps every exp at most 1. Real logits can be large, and a softmax that returns nan on big inputs silently ruins a training run.
Words and symbols in this lesson
- derivative (df/dx)
- how fast the output changes per unit nudge of the input
- softmax
- exp of each score divided by the total: scores become probabilities
- cross-entropy
- minus the log of the probability given to the right answer
- KL(P || Q)
- how different two distributions are; zero only when they match; order matters
Write, for 2D arrays of logits (one row per example):
logsumexp(z): log of the sum of exp along each row, safe for huge values.stable_softmax(z): softmax along each row, safe for huge values.nll(z, labels): the mean cross-entropy computed straight from logits aslogsumexp - z[true], never taking the log of a softmax.matmul_grads(X, W, G): for Y = X @ W and a loss whose gradient with respect to Y is G, return(dX, dW), the gradients with respect to X and W, without building any Jacobian.kl_normal(m1, s1, m2, s2): the KL divergence from the bell curve N(m1, s1) to N(m2, s2), where s1 and s2 are spreads (standard deviations).
The checks use new random shapes and numbers every time, and compare with nudging and with a numerical integral.
AI off: no tutor, hints or worked solution on this one. Your own thinking only.
Spotted a problem in this lesson? Report it · Ask about this lesson
Your progress is counted anonymously. Change this
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.