Syllabus Lesson 40 of 246 · Module 3: Neuron to MLP
Module 3: Neuron to MLP

Walking downhill (gradient descent by hand)

Why it matters Picking weights by hand does not scale to a million weights. The gradient tells every weight which way is downhill at once.

Name it

For the loss L = mean((X @ w + b - y) squared), nudge w1 and L changes by 2 * mean(error * x1), where error is prediction minus truth. That is the derivative of L with respect to w1. All at once: grad_w = 2 * X.T @ error / n, and grad_b = 2 * mean(error).

Read it in plain words

If the predictions are too high (error positive) on houses with a large size, then lowering w1 helps, and the gradient for w1 is positive, so we subtract it. Each step: w = w - lr * grad_w, where lr, the learning rate, sets the step size.

Gradient descent: repeat "compute the gradient, take a small step against it". Learning rate: the size of each step.

Call the Curve

Before you run your training, guess the loss at steps 0, 10, 50 and 200 in my_guess. The code prints your guess beside the truth.

Recap Gradient, step against it, repeat.

Words and symbols in this lesson
weights (w, W), bias (b)
the numbers a neuron learns; the bias shifts its output
learning rate (lr)
how big each training step is
Your turn

Write gradients(X, y, w, b) returning (grad_w, grad_b), and train(X, y, lr, steps) starting from zeros and returning (w, b, losses), where losses holds the loss before each step. Fill in my_guess first. The trained loss must end below 0.01 with lr 0.5 and the 201 steps the starter asks for.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.