What a Trained Model Actually Is

Training finishes. The corrections stop. Something now exists that did not exist before, and it is worth asking plainly what that something is.

It is not the examples. Those are gone, and the thing that remains is far too small to be holding them. It is not a program in any sense you would recognise, because nobody wrote its behaviour down. And it is not a description of anything. Open it and read it and you will find no sentence saying what makes a cat a cat, no rule, no summary, nothing you could argue with. Just an enormous quantity of numbers, none of which mean anything on their own.

Somehow that is enough. The numbers hold what was learned, and the only way to get at what they hold is to run them.

This has consequences people find genuinely uncomfortable. A thing can know how to do something while containing no account of what it knows. It cannot be inspected the way a rulebook can. It can be copied perfectly and instantly, sent anywhere, and run by someone who has no idea what went into it or what it absorbed on the way.

And once it is finished, it is finished. Whatever it learned, it learned then. Using it changes nothing, which is a stranger fact than it first appears, and explains a great deal about how these things behave later.