Syllabus Lesson 44 of 246 · Module 3: Neuron to MLP
Module 3: Neuron to MLP

Activations: the bend between layers

Why it matters Without a bend between layers, ten layers are no smarter than one. The choice of bend also decides whether gradients can flow.

Three activations you will meet everywhere:

  • ReLU: max(0, z). Cheap, and its slope is exactly 1 for positive inputs, so gradients pass through undamped. Slope 0 for negatives.
  • tanh: squashes into -1 to 1, centred on 0. Slope 1 at 0 but nearly 0 for large inputs: it saturates.
  • GELU: a smooth ReLU used in transformers, roughly z times sigmoid(1.702 z).

Why the slope matters

Backpropagation multiplies by each activation's slope. A slope near 0 (a saturated tanh, or ReLU on a negative input) blocks the gradient. You will see that cause real training failures in Module 6.

Saturate: to sit in a flat part of the curve, where the slope, and so the gradient through it, is almost zero.

Recap An activation bends; its slope decides how much gradient flows back.

Words and symbols in this lesson
sigmoid
1 / (1 + exp(-z)): squashes any number into 0 to 1
ReLU
max(0, z): keeps positives, zeroes negatives
backpropagation
the chain rule run backward to get every weight's gradient
Your turn

Write each activation and its slope: relu, relu_slope, tanh, tanh_slope, gelu, gelu_slope, all on arrays. Use the GELU approximation z * sigmoid(1.702 z). Its slope needs two rules together: the product rule (the slope of u × v is u' × v + u × v') and the chain rule inside the sigmoid (the slope of sigmoid(1.702 z) is 1.702 × s(1 - s), with s = sigmoid(1.702 z)). For ReLU's slope at exactly 0, use 0. The hidden test checks every slope against a nudge.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.