A Network That Learns

One neuron can only draw one straight line through the world, and the world is not divided into two by a straight line. Stack them, then. Let some of them look at the input, let others look at what the first ones concluded, and let a last one decide. That much is obvious enough that people thought of it immediately.

The problem is blame. The answer comes out wrong, and somewhere in there are thousands of little dials that each contributed a share of the wrongness. Which ones? By how much? Nobody can watch them all. Turn the wrong dial and you make things worse while believing you are helping.

For years this is where the idea stopped. The machine could be built and could not be taught, because nothing could work out who was responsible for what.

The way out is one of those results that seems modest and is not. It turns out you can work backwards from the mistake, layer by layer, and hand every single dial a precise share of the fault, cheaply enough to do it millions of times. Nudge everything by its share, then do it again with the next example, and again, for a very long time.

Nothing in that process understands anything. There is no plan and no goal, only an error getting slightly smaller, over and over. What comes out the other end is a machine that recognises faces, and nobody can point to where in it the face is.