What Attention Means

Direct access solves one problem and creates another. If a token can receive information from many other tokens, how does it decide which ones matter right now?

Attention answers with scores. For each token, it scores the other positions it is allowed to use. A high score means that position should contribute more information. A low score means it should contribute less.

Those scores become weights. The model uses them to blend information from the available positions into a new version of the token. The stored embedding is still the starting point, but the blend adjusts it to fit this particular sentence.

The scoring is learned during training. Each token produces one numerical pattern for what it is looking for and another for what it can offer. The model compares those patterns, then adjusts the learned weights that produced them when its predictions are wrong.

One comparison may look for one kind of connection while another looks for something different. Running several in parallel gives the layer several learned ways to decide what matters. Programmers do not assign each one a tidy linguistic job.

This gives a distant token a short route into the blend. Within a layer, many position-to-position scores can also be calculated together instead of waiting for one running state to move along the sentence.

The interesting part is how a machine turns "matters" into a number.