From Compression to Prediction
Compressing information can intriguingly lead to enhanced predictive capabilities. This general scheme is recognizable in many contexts ― organic and artificial.
Life itself
Life can be understood as a local defense against universal entropy (chaos or heightening of chaos). Within any bubble of life, there is a concentration of resources to this aim. At the same time, any form of life needs to be predictive, ‘knowing’ how to respond to specific circumstances in order not to lose the battle against entropy.
From bacterium to Bach, in order to stay alive and thrive, the organism must therefore ‘know’ how to predict and respond in ever-changing circumstances.
Intelligence
The brain/mind is one giant pattern recognizer. Internally, mappings go on from patterns to patterns and from brain parts to brain parts. At each level, information is compressed. This functions as a brainy prediction-then-correction of what is supposed to be the output from one level to another.
The scheme is repeated at the level of the whole brain and of the organism. A massive amount of input (much more than we consciously perceive) is compressed into a tiny information space from which more or less appropriate actions result.
Thus, organic intelligence follows the general scheme throughout evolution.
Large language models (LLMs)
Perhaps unsurprisingly, the general scheme continues into the artificial world.
LLMs are basically a vectorization (parameterization, numberification) of massive amounts of human-produced text. The text is compressed into the numbers while much (but not all) of the information is retained, albeit for the human eye and mind in an implicit manner.
The LLM advantage is that a computer can – through the huge power of number crunching – more easily present the information in new ways that are understandable to humans. Notoriously, this has recently been the case in suprisingly many fields and to a surprising degree even to the developers themselves.
As such, there is no humanly explicit knowledge within the numbers. Not any single number stands for a concept. Nevertheless, the computer can predict the next word in a humanly sentence fairly accurately — mostly. All output of an LLM is a prediction, as is all output of the human brain.
In both cases, the predictions seem pretty intelligent, although necessarily only to a certain degree. In neither of both cases do we exactly know how it’s done.
Relevantly lossy information compression by itself seems to lead to the spontaneous emergence of predictive capability.
As in any setting, there is no emergence of information that is not already inherently – implicitly – present. The mechanism of compression can then be relatively simple, of course, because the resulting intelligence doesn’t come from this mechanism but from the information.
It’s as if the compression puts some pressure on the system through which, time and again, some Gestalt comes forward. These Gestalts must have entered the system through its contact with reality.
Hm, is there eventually no need for a deeper level of comprehension?
Does this mean that combinations of LLMs together with other technologies will surpass our human intelligence pretty soon?
These are pretty crucial questions.
Related Posts
At the Brink of Robotics?
Soon enough, we will see a revolution in robotics development on the scale of the present and pending A.I.-in-knowledge revolution. Together, these revolutions will bring eu-topia or dystopia. We don’t know, but we should not remain idle. Pretty much the same basic technologies This is: apart from some translation that makes the analogies less obvious Read the full article…
How Lisa Prevents LLM Hallucinations
Hallucinations (better-called confabulations) in the context of large language models (LLMs) occur when these models generate information that isn’t factually accurate. Lisa can mitigate these from the insight of why they happen, namely: LLM confabulations happen because these systems don’t have a proper understanding of the world but generate text based on patterns learned from Read the full article…
Can Motivation be Purely Conscious?
Motivation as we know it is present in a system (you, me) that is partly conscious, partly non-conscious. Thus, the question is much more difficult than it appears at first sight. Nevertheless, towards future A.I., it will need to be solved. Purely conscious? This is also purely (even though possibly partly fuzzy) conceptual. Motivation would Read the full article…