Syllabus Lesson 32 of 246 · Module 2: Maths and probability, only as needed
Module 2: Maths and probability, only as needed

How spread adds up, and what a matrix stretches

Why it matters Two facts decide whether a deep network trains at all: how the spread of a sum grows with the number of terms (Module 6 sets starting weights with it), and how much a matrix stretches a vector each time it is applied (Module 11's RNNs apply one again and again).

Spread of a sum

Add n independent numbers, each with variance v. The variances add: the sum has variance n v, so its spread (standard deviation) is sqrt(n v). Tiny numbers: 100 inputs each with spread 1 give a sum with spread 10, not 100 and not 1. A neuron adds up n inputs times weights, so to keep its output's spread near 1, each weight needs a spread of about 1 / sqrt(n). That is the idea behind Module 6's starting weights.

What a matrix stretches

Some vectors only get longer or shorter when a matrix A multiplies them; their direction stays. Such a vector is an eigenvector and the stretch factor is its eigenvalue. Multiply a vector by A again and again and the direction with the largest eigenvalue (in size) wins: after t steps the length grows or shrinks like that eigenvalue to the power t. So a largest size above 1 blows up, and below 1 fades away.

Tiny numbers: A = [[0.5, 0], [0, 1.2]]. Its eigenvalues are 0.5 and 1.2. After 10 steps the first direction has shrunk to 0.510, about 0.001, and the second has grown to 1.210, about 6.2.

Eigenvalue: how much a matrix stretches one special direction. Spectral radius: the largest eigenvalue in size, which decides whether repeated multiplication grows or fades.

Recap Variances add, so spreads grow like sqrt(n). Repeated multiplication grows like the largest eigenvalue to the power t.

Words and symbols in this lesson
vector
a list of numbers, one per feature
Your turn

Three steps:

  1. Write sum_spread(n, sd, trials, seed): with rng = np.random.default_rng(seed), draw trials sums of n normal numbers with spread sd, and return the spread (np.std) of those sums.
  2. Set predicted to the spread the rule gives for n = 100, sd = 0.5 (sqrt(n) x sd).
  3. Write growth_rate(A, steps=200): start from a vector of ones; steps times, multiply by A and rescale the result to length 1; return the length the vector had just before the last rescale. Worked example: for A = [[0.5, 0], [0, 1.2]] the answer is 1.2.

Why step 3 works: repeated multiplication lets the biggest stretch win, so the last one-step length is the largest eigenvalue in size, found without any eigenvalue function. (The check uses symmetric matrices, where this always holds. For some other matrices the two largest are a complex pair and the length wobbles from step to step; you will not need that case.)

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.