Syllabus Lesson 84 of 246 · Module 6: Starting weights and normalisation
Module 6: Starting weights and normalisation

Shortcuts (residual connections)

Why it matters Residual connections are why networks with hundreds of layers train at all. Every transformer block uses them.

A plain layer replaces its input: h = f(h). A residual block adds to it: h = h + f(h). If f learns nothing useful, the block simply passes h through unchanged, so extra depth can never make things worse at the start. And on the way back, the gradient has a direct route along the "+ h" path, skipping the layer entirely, so it cannot vanish.

A fair warning about the experiment below: both networks start from the same deliberately small weights, N(0, 0.1). That start is what sinks the plain network. With He starting weights from earlier in this module, the plain 30-layer network trains too (5 of 6 seeds in the course's runs). What shortcuts add is robustness: they train even from a poor start, and they keep working at depths where careful starting weights alone stop being enough.

Here each block's contribution is also scaled by 1 / depth, a common trick so 30 additions do not blow up the signal.

Residual connection (or skip connection, shortcut): adding a block's input to its output.

Call the Curve: guess the final loss after 150 steps for the plain network and for the one with shortcuts. (A loss of 0.693 means a coin toss on this two-class task.)

Words and symbols in this lesson
Xavier, LeCun, He
starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
residual connection
adding a block's input to its output: h + f(h)
Your turn

Fill in my_guess. Then write block(h, W, b, shortcuts, depth): without shortcuts return relu(h @ W + b); with shortcuts return h + relu(h @ W + b) * (1 / depth). The test trains both 30-layer networks from the same small starting weights.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.