Activations: the bend between layers
Why it matters Without a bend between layers, ten layers are no smarter than one. The choice of bend also decides whether gradients can flow.
Three activations you will meet everywhere:
- ReLU: max(0, z). Cheap, and its slope is exactly 1 for positive inputs, so gradients pass through undamped. Slope 0 for negatives.
- tanh: squashes into -1 to 1, centred on 0. Slope 1 at 0 but nearly 0 for large inputs: it saturates.
- GELU: a smooth ReLU used in transformers, roughly z times sigmoid(1.702 z).
Why the slope matters
Backpropagation multiplies by each activation's slope. A slope near 0 (a saturated tanh, or ReLU on a negative input) blocks the gradient. You will see that cause real training failures in Module 6.
Saturate: to sit in a flat part of the curve, where the slope, and so the gradient through it, is almost zero.
Recap An activation bends; its slope decides how much gradient flows back.
Words and symbols in this lesson
- sigmoid
- 1 / (1 + exp(-z)): squashes any number into 0 to 1
- ReLU
- max(0, z): keeps positives, zeroes negatives
- backpropagation
- the chain rule run backward to get every weight's gradient
Write each activation and its slope: relu, relu_slope, tanh, tanh_slope, gelu, gelu_slope, all on arrays. Use the GELU approximation z * sigmoid(1.702 z). Its slope needs two rules together: the product rule (the slope of u × v is u' × v + u × v') and the chain rule inside the sigmoid (the slope of sigmoid(1.702 z) is 1.702 × s(1 - s), with s = sigmoid(1.702 z)). For ReLU's slope at exactly 0, use 0. The hidden test checks every slope against a nudge.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.