Syllabus Lesson 85 of 246 · Module 6: Starting weights and normalisation
Module 6: Starting weights and normalisation

Gate: make an untrainable deep net train (AI off)

Why it matters Fixing a deep network that will not learn is one of the most common real jobs in this field. Here you do it alone.

AI tutor off. The 30-layer network below never leaves a coin toss. You may change how it is started and how its blocks are wired, using anything from this module, but keep 30 hidden layers of width 16 and 300 steps of plain SGD.

Words and symbols in this lesson
Xavier, LeCun, He
starting spreads sqrt(2 / (inputs + outputs)) or 1 / sqrt(inputs) for tanh, and sqrt(2 / inputs) for ReLU
Your turn

Edit build(rng) and forward(params, X) so that training reaches at least 90% training accuracy on the moons.

AI off: no tutor, hints or worked solution on this one. Your own thinking only.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.