Syllabus Lesson 48 of 246 · Module 3: Neuron to MLP
Module 3: Neuron to MLP

Backprop for a two-layer net, by hand

Why it matters This lesson is the whole of backpropagation for a real network. After it, nothing in a deep learning library is magic.

See it first

Play the animation. The error starts at the output as (softmax minus one-hot). It flows back through the second layer, gets blocked wherever a ReLU was off, flows through the first layer, and leaves a gradient on every weight.

The forward pass

H_pre = X @ W1 + b1      # (n, hidden)
H     = relu(H_pre)
Z     = H @ W2 + b2      # (n, classes)
loss  = cross_entropy(softmax(Z), y)

The backward pass, by matching shapes

The trick that makes this easy: every gradient has the same shape as the thing it belongs to. If you know the shapes, there is usually only one way to multiply the pieces that fits. The first two steps, worked:

dZ  = (softmax(Z) - onehot) / n     # (n, classes)  from last lesson
dW2 = H.T @ dZ                      # (hidden, classes): same shape as W2.
                                    # H is (n, hidden) and dZ is (n, classes):
                                    # only H.T @ dZ fits.

The rest is yours. Work out each one from the shapes it must have:

  • db2: shape (classes,). Each bias was added to every row.
  • dH: shape (n, hidden), the same as H. It comes from dZ and W2.
  • dH_pre: shape (n, hidden). ReLU passed its input through where it was on and blocked it elsewhere.
  • dW1 and db1: the same two steps as for W2 and b2, one layer down.

Recap Backward is forward in reverse, multiplying by each step's local slope, with shapes as your guide.

Words and symbols in this lesson
weights (w, W), bias (b)
the numbers a neuron learns; the bias shifts its output
ReLU
max(0, z): keeps positives, zeroes negatives
one-hot
zeros with a single 1 at the true class
backpropagation
the chain rule run backward to get every weight's gradient
Your turn

Write forward_backward(X, y, W1, b1, W2, b2) that returns (loss, grads) where grads is a dictionary with keys W1, b1, W2, b2. The hidden test checks every gradient against a nudge.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.