The GPU Moment

An idea that works but takes six weeks to test is barely an idea. You get eight attempts a year. You cannot afford to be curious, and you certainly cannot afford to be wrong very often.

This was the real constraint on neural networks for years, and it is easy to overlook because it sounds like an engineering detail rather than an intellectual one. It is not. Nothing about the theory changed when it lifted. What changed was how many times a week a researcher could be wrong, and that turns out to matter enormously to how fast a field moves.

The rescue came from somewhere nobody in the field was looking. The arithmetic that trains a network, done millions of times over with no step depending on the one beside it, happens to be almost exactly the arithmetic that draws a video game. Hardware built to push pixels for teenagers turned out to be hardware built for this, entirely by accident.

Weeks became about a day. The same ideas, the same mathematics, the same people, suddenly able to try thirty things where they had managed one. What had looked like a dead theory turned out to have been a fast-enough-machine problem the whole time, which raises an uncomfortable question about what else is sitting in the literature waiting.