Embeddings

An ID tells the computer which piece of text arrived. It says nothing about how that piece is used. Two words that mean nearly the same thing can be numbered as far apart as any two strangers.

The fix is to stop naming words and start placing them. A word becomes a point in a space, surrounded by neighbours, and what it means is bound up with who those neighbours are. Nobody assigns the position. It is worked out from how the word behaves across enormous amounts of text, and the result is called an embedding.

That buys more than tidy grouping. Once words are positions rather than labels, the space between them has structure of its own. Some relationships turn up as a repeated step: the same move that carries one word to its partner carries a different pair the same way. It is a real pattern and an easy one to oversell, holding approximately, for some pairs, and differently in every model.

It also changes what searching means. If related things sit near each other, then finding what you asked for is a matter of looking nearby, and a document that shares none of your words can still be the closest thing in the space.

One thing a position cannot do is read. A word's place is fixed before any particular sentence arrives, so bank sets off from the same spot whether the river or the money is meant, and only the surrounding words can settle it.

Underneath sits an older question. If a system places a word correctly among all the words it relates to, without ever having encountered the thing the word refers to, how much of meaning has it actually got hold of? That argument will not be settled here. What changes is that you will know precisely what is inside the machine when it is asked.