Tokens

A computer has no way to hold a word. It holds numbers, and nothing else. So before a model can do anything with a sentence, the sentence has to be broken into pieces, and each piece swapped for a number.

Deciding how big those pieces should be turns out to be a real problem with no clean answer, and the compromise everyone settled on is stranger than it sounds. The pieces are not words. Some are, some are halves of words, and some are a word with the space in front of it still attached. They are called tokens, and a separate program cuts them before the model receives a thing.

That cutting happens out of sight, and it explains a surprising amount of what models get strangely wrong. It also decides who pays more, because the pieces were chosen by reading one particular pile of text, and not everybody's language was in that pile.

None of it gives the model any meaning yet. At the end of the cutting, each piece has a number, and a number is still only a name tag.