Syllabus Lesson 61 of 246 · Module 4: Your own autograd engine
Module 4: Your own autograd engine

Layers that hold their own weights

Why it matters Real networks have many layers. Each layer should own its weights and be able to list them, so training code never has to know what is inside.

A Linear layer holds W and b as Tensors and computes x @ W + b. An MLP holds a list of Linear layers with ReLU between them. Both offer parameters(), a list of their weight Tensors.

Why zero the gradients

Your backward adds to each grad. That is right within one pass, but between steps the old gradient is still sitting there. Forget to clear it and every step uses the sum of all past gradients: training drifts and then blows up. So each step is: zero the grads, forward, backward, step.

Parameter: a weight the training changes. zero_grad: resetting every parameter's grad to zero before the next backward.

Start weights with spread sqrt(2 / inputs), which keeps signals a sensible size through ReLU layers. Module 6 explains why.

Words and symbols in this lesson
Tensor
here: an array plus its gradient plus the Tensors it came from
unbroadcast
summing a gradient back down to the shape it was stretched from
zero_grad
resetting gradients before the next step, because backward adds to them
Your turn

Write Linear, MLP, zero_grad(params) and sgd_step(params, lr) as described. MLP([64, 32, 10], rng) makes two Linear layers with a ReLU between them and none after the last.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.