Syllabus Lesson 45 of 246 · Module 3: Neuron to MLP
Module 3: Neuron to MLP

Any curve from bumps (universal approximation)

Why it matters People say a network with one hidden layer can fit any curve. Here you see why, by building it yourself out of bumps.

See it

A very steep sigmoid is a step: 0 on the left, 1 on the right. Two steps, one up at left and one down at right, make a bump: height * (sigmoid(s (x - left)) - sigmoid(s (x - right))). That is two hidden neurons and one output weight. Put bumps side by side, each as tall as the curve at its spot, and you trace any curve, like a bar chart. More neurons, thinner bars, closer fit.

Universal approximation: one hidden layer with enough neurons can get as close as you like to any reasonable curve. It says nothing about how easy those weights are to find by training.

Recap Two neurons make a bump; enough bumps make anything.

Words and symbols in this lesson
neuron
w . x + b, then an activation
weights (w, W), bias (b)
the numbers a neuron learns; the bias shifts its output
sigmoid
1 / (1 + exp(-z)): squashes any number into 0 to 1
hidden layer
a layer between input and output that builds new features
Your turn

Write bump(x, left, right, height, s=60), then approx(x, f, n_bumps) that splits 0 to pi into n_bumps equal pieces and adds one bump per piece, each with the height of f at the middle of its piece. With 10 bumps (20 neurons), the mean squared error against sin on 0 to pi must be below 0.01.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.