Layers that hold their own weights
Why it matters Real networks have many layers. Each layer should own its weights and be able to list them, so training code never has to know what is inside.
A Linear layer holds W and b as Tensors and computes x @ W + b. An MLP holds a list of Linear layers with ReLU between them. Both offer parameters(), a list of their weight Tensors.
Why zero the gradients
Your backward adds to each grad. That is right within one pass, but between steps the old gradient is still sitting there. Forget to clear it and every step uses the sum of all past gradients: training drifts and then blows up. So each step is: zero the grads, forward, backward, step.
Parameter: a weight the training changes. zero_grad: resetting every parameter's grad to zero before the next backward.
Start weights with spread sqrt(2 / inputs), which keeps signals a sensible size through ReLU layers. Module 6 explains why.
Words and symbols in this lesson
- Tensor
- here: an array plus its gradient plus the Tensors it came from
- unbroadcast
- summing a gradient back down to the shape it was stretched from
- zero_grad
- resetting gradients before the next step, because backward adds to them
Write Linear, MLP, zero_grad(params) and sgd_step(params, lr) as described. MLP([64, 32, 10], rng) makes two Linear layers with a ReLU between them and none after the last.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.