Syllabus Lesson 26 of 246 · Module 2: Maths and probability, only as needed
Module 2: Maths and probability, only as needed

Slopes by nudging (derivatives)

Why it matters Training asks one question over and over: if I nudge this weight, what happens to the loss? That question has a name: the derivative.

See it

Take f(x) = x * x at x = 3, where f is 9. Nudge x to 3.001: f becomes 9.006001. The output moved 0.006001 for an input move of 0.001, about 6 times as much. So the slope at 3 is about 6.

The exact slope of x squared is 2 times x, which is 6 at x = 3. The nudge got it almost exactly. Nudging both ways and averaging, (f(x + h) - f(x - h)) / (2h), is even more accurate.

Derivative: how fast the output changes per unit change of the input, at one point. The slope of the curve there.

Slopes you will need

You can always nudge, but these exact slopes come up again and again in this course. Each is easy to check with your slope function.

FunctionSlope
x to the power nn times x to the power n - 1 (so x squared has slope 2x)
exp(x)exp(x): its own slope
log(x)1 / x
sigmoid(x) = 1 / (1 + exp(-x))sigmoid(x) times (1 - sigmoid(x))
tanh(x)1 - tanh(x) squared
a times b, both depending on x(slope of a) times b + a times (slope of b): the product rule
f(g(x)), one inside anotherslope of f at g(x), times slope of g: the chain rule, next lesson

Recap Nudge, measure, divide. That is all a derivative is.

Words and symbols in this lesson
derivative (df/dx)
how fast the output changes per unit nudge of the input
chain rule
the slope of steps in a row is the product of each step's own slope
Your turn

Write slope(f, x, h=1e-5) using the two-sided nudge. It must work for any function of one number.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.