How spread adds up, and what a matrix stretches
Why it matters Two facts decide whether a deep network trains at all: how the spread of a sum grows with the number of terms (Module 6 sets starting weights with it), and how much a matrix stretches a vector each time it is applied (Module 11's RNNs apply one again and again).
Spread of a sum
Add n independent numbers, each with variance v. The variances add: the sum has variance n v, so its spread (standard deviation) is sqrt(n v). Tiny numbers: 100 inputs each with spread 1 give a sum with spread 10, not 100 and not 1. A neuron adds up n inputs times weights, so to keep its output's spread near 1, each weight needs a spread of about 1 / sqrt(n). That is the idea behind Module 6's starting weights.
What a matrix stretches
Some vectors only get longer or shorter when a matrix A multiplies them; their direction stays. Such a vector is an eigenvector and the stretch factor is its eigenvalue. Multiply a vector by A again and again and the direction with the largest eigenvalue (in size) wins: after t steps the length grows or shrinks like that eigenvalue to the power t. So a largest size above 1 blows up, and below 1 fades away.
Tiny numbers: A = [[0.5, 0], [0, 1.2]]. Its eigenvalues are 0.5 and 1.2. After 10 steps the first direction has shrunk to 0.510, about 0.001, and the second has grown to 1.210, about 6.2.
Eigenvalue: how much a matrix stretches one special direction. Spectral radius: the largest eigenvalue in size, which decides whether repeated multiplication grows or fades.
Recap Variances add, so spreads grow like sqrt(n). Repeated multiplication grows like the largest eigenvalue to the power t.
Words and symbols in this lesson
- vector
- a list of numbers, one per feature
Three steps:
- Write
sum_spread(n, sd, trials, seed): withrng = np.random.default_rng(seed), drawtrialssums of n normal numbers with spread sd, and return the spread (np.std) of those sums. - Set
predictedto the spread the rule gives for n = 100, sd = 0.5 (sqrt(n) x sd). - Write
growth_rate(A, steps=200): start from a vector of ones;stepstimes, multiply by A and rescale the result to length 1; return the length the vector had just before the last rescale. Worked example: for A = [[0.5, 0], [0, 1.2]] the answer is 1.2.
Why step 3 works: repeated multiplication lets the biggest stretch win, so the last one-step length is the largest eigenvalue in size, found without any eigenvalue function. (The check uses symmetric matrices, where this always holds. For some other matrices the two largest are a complex pair and the length wobbles from step to step; you will not need that case.)
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.