Slopes by nudging (derivatives)
Why it matters Training asks one question over and over: if I nudge this weight, what happens to the loss? That question has a name: the derivative.
See it
Take f(x) = x * x at x = 3, where f is 9. Nudge x to 3.001: f becomes 9.006001. The output moved 0.006001 for an input move of 0.001, about 6 times as much. So the slope at 3 is about 6.
The exact slope of x squared is 2 times x, which is 6 at x = 3. The nudge got it almost exactly. Nudging both ways and averaging, (f(x + h) - f(x - h)) / (2h), is even more accurate.
Derivative: how fast the output changes per unit change of the input, at one point. The slope of the curve there.
Slopes you will need
You can always nudge, but these exact slopes come up again and again in this course. Each is easy to check with your slope function.
| Function | Slope |
|---|---|
| x to the power n | n times x to the power n - 1 (so x squared has slope 2x) |
| exp(x) | exp(x): its own slope |
| log(x) | 1 / x |
| sigmoid(x) = 1 / (1 + exp(-x)) | sigmoid(x) times (1 - sigmoid(x)) |
| tanh(x) | 1 - tanh(x) squared |
| a times b, both depending on x | (slope of a) times b + a times (slope of b): the product rule |
| f(g(x)), one inside another | slope of f at g(x), times slope of g: the chain rule, next lesson |
Recap Nudge, measure, divide. That is all a derivative is.
Words and symbols in this lesson
- derivative (df/dx)
- how fast the output changes per unit nudge of the input
- chain rule
- the slope of steps in a row is the product of each step's own slope
Write slope(f, x, h=1e-5) using the two-sided nudge. It must work for any function of one number.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.