What the Weights Contain

Somewhere in one of these systems is the fact that Paris is the capital of France. It answers correctly every time, so it must be in there. Go looking for it and you will not find it.

There is no row holding it, no location storing it, nothing you could point at and delete. The knowledge is smeared across an enormous number of numbers that each take part in thousands of unrelated things, and no individual one of them means anything whatsoever. Take any single number away and nothing breaks in particular. Everything gets very slightly worse at everything.

This is not a flaw in the engineering, and it is not temporary. It is what learning from examples produces, and your own memory works closer to this than to a filing cabinet, which is why you can recognise a friend instantly and be unable to say what you recognised.

The consequences are awkward and they compound. Nothing can be removed without damaging what surrounds it. Nothing can be read back, though people are getting better at prising some of it out. And a machine with no index has no way to discover that it does not know something, which is why an answer from the far edge of what it learned arrives in the same voice as one from the middle.

Underneath all of it is a constraint rather than a choice. A model has to hold more distinct things than it has numbers to hold them in, so the things share, and everything difficult here is the bill for that arrangement.