Backprop for a two-layer net, by hand
Why it matters This lesson is the whole of backpropagation for a real network. After it, nothing in a deep learning library is magic.
See it first
Play the animation. The error starts at the output as (softmax minus one-hot). It flows back through the second layer, gets blocked wherever a ReLU was off, flows through the first layer, and leaves a gradient on every weight.
The forward pass
H_pre = X @ W1 + b1 # (n, hidden)
H = relu(H_pre)
Z = H @ W2 + b2 # (n, classes)
loss = cross_entropy(softmax(Z), y)
The backward pass, by matching shapes
The trick that makes this easy: every gradient has the same shape as the thing it belongs to. If you know the shapes, there is usually only one way to multiply the pieces that fits. The first two steps, worked:
dZ = (softmax(Z) - onehot) / n # (n, classes) from last lesson
dW2 = H.T @ dZ # (hidden, classes): same shape as W2.
# H is (n, hidden) and dZ is (n, classes):
# only H.T @ dZ fits.
The rest is yours. Work out each one from the shapes it must have:
db2: shape (classes,). Each bias was added to every row.dH: shape (n, hidden), the same as H. It comes from dZ and W2.dH_pre: shape (n, hidden). ReLU passed its input through where it was on and blocked it elsewhere.dW1anddb1: the same two steps as for W2 and b2, one layer down.
Recap Backward is forward in reverse, multiplying by each step's local slope, with shapes as your guide.
Words and symbols in this lesson
- weights (w, W), bias (b)
- the numbers a neuron learns; the bias shifts its output
- ReLU
- max(0, z): keeps positives, zeroes negatives
- one-hot
- zeros with a single 1 at the true class
- backpropagation
- the chain rule run backward to get every weight's gradient
Write forward_backward(X, y, W1, b1, W2, b2) that returns (loss, grads) where grads is a dictionary with keys W1, b1, W2, b2. The hidden test checks every gradient against a nudge.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.