Why Deep Networks Were Hard to Train

If a few layers can learn something, more layers should learn something richer. Each one gets to work on what the last one figured out, building from edges to shapes to objects. The reasoning is sound, everyone believed it, and for a long stretch it simply did not work. Deep networks trained worse than shallow ones, which is exactly backwards.

The cause turned out to be in the correction. A mistake at the output has to travel back through every layer to reach the early ones, and something happens to it on the way: it gets weaker at each handoff. By the time it arrives at the front of a deep network there is almost nothing left of it. Those first layers, the ones doing the most fundamental work, receive the faintest instruction about what to change.

So the network does learn. The back half of it learns. The front half sits nearly still, which is why adding depth made things worse rather than better.

What is worth noticing is how this got solved, because it is not a story with a hero in it. No single insight fixed it. Several unglamorous adjustments, made by different people for different reasons over years, each recovered a little of the signal. Together they were enough. Progress here looks less like a breakthrough and more like a slow accumulation of small repairs, which is what most progress actually looks like.