The Transformer

A name can mislead. Attention is the part of this machine everyone can name, and it is not the machine. It is one move: words reaching across a sentence and pulling in what matters from each other. Gathering. And gathering is not the same as doing anything with what arrives.

So there is a second move, and it is the larger one. After the reaching, every word goes off alone and is worked over on its own terms. Two moves, in that order, repeated level after level. Most of what such a machine has absorbed about the world sits in that second half, the one nobody put in the name.

The word itself names a shape rather than a thing: a decision about what parts a network should have and how they connect. Training fills that shape with numbers, and one shape filled differently becomes thousands of different models.

Something was lost along the way. Older designs read a sentence the way you do, left to right, and got word order for nothing in return. Comparing every word to every other at once is faster, but it turns a sentence into a heap. Order is not a decoration on language. It decides who did what to whom, so it has to be put back deliberately.

Then one further choice splits the field in two. Is a word allowed to see what comes after it, or only what came before? Both answers work, they suit opposite kinds of work, and the choice reaches all the way into how each kind of model is taught.

That teaching is the other half of the story. Written text can supply its own questions and its own answers, so learning no longer waits on people to label anything. One broad and expensive stage of it, followed by a much cheaper shaping stage, is the pattern nearly everything since has been built on.