{"id":18300,"date":"2024-11-23T17:58:00","date_gmt":"2024-11-23T17:58:00","guid":{"rendered":"https:\/\/aurelis.org\/blog\/?p=18300"},"modified":"2024-11-24T08:09:21","modified_gmt":"2024-11-24T08:09:21","slug":"the-compute-efficient-frontier","status":"publish","type":"post","link":"https:\/\/aurelis.org\/blog\/artifical-intelligence\/the-compute-efficient-frontier","title":{"rendered":"The Compute-Efficient Frontier"},"content":{"rendered":"\n

Research on scaling laws for LLMs<\/a> suggests that while scaling model size (functional parameters), dataset size (tokens), and compute (amount of computing power) improves performance, diminishing returns are becoming evident. This is called the \u2018compute-efficient frontier.\u2019<\/h3>\n\n\n\n

Apparently, it does not depend much on architectural details (such as network width or depth) as long as reasonably good choices are made. Thus, is this a fundamental law in nature or a result of our specific neural network approach to A.I. \u2015 just pointing to the need for innovation beyond scaling?<\/p><\/blockquote>\n\n\n\n

Speculative aspirations<\/strong><\/p>\n\n\n\n

The \u2018race toward AGI\u2019 is speculative, mainly in the sense that it is a race toward the undefined. The diminishing returns through scaling are thus no surprise \u2015 simply because we don\u2019t know what we\u2019re doing or even heading to. This way, we\u2019re just throwing \u2019something\u2019 harder to \u2018somewhere\u2019 and see \u2013 hurray! \u2013 that it works better.<\/p>\n\n\n\n

Indeed, it works better in practical applications such as natural language processing, integration into IoT (Internet of Things), customer service, and spatial computing \u2015 but, as said, with diminishing returns.<\/p>\n\n\n\n

Meanwhile, the big speculation remains speculative. That may even be not such a bad thing in the domain of non-Compassionate A.I.<\/p>\n\n\n\n

But we want to do better in (and through) Compassionate A.I.<\/strong><\/p>\n\n\n\n

Only here do we need to get closer to what we really want in this, which is the complexity of human mental depth<\/em> (which is very different from neural network depth). Thus, we also need to see that human depth is where our thinking (and intelligence) really comes from.<\/p>\n\n\n\n

The prior \u2018A.I. winters\u2019 (downfall from very high expectations) came, in hindsight, from the hubris about ourselves<\/strong> \u2015 mainly at the conceptual level. Thus, researchers overestimated what A.I. can achieve based on a narrow understanding of human intelligence.<\/p>\n\n\n\n

With new possibilities in the field of ANNs, we relatively recently came into a new A.I. spring. These new possibilities are, indeed, scaling model size, dataset size, and compute. Meanwhile, the hubris has remained the same in that researchers generally still don\u2019t see where human depth arises or what it even entails \u2015 even while the same hubris makes them think they do.<\/p>\n\n\n\n

New toys \u2015 same players.<\/p>\n\n\n\n

Can subconceptual processing, as seen in human brains, inspire breakthroughs?<\/strong><\/p>\n\n\n\n

[subconceptual processing<\/em>: non-linear, pattern-based processes underlying conscious thought]<\/p>\n\n\n\n

Humans don\u2019t scale neurons infinitely to increase intelligence; instead, complexity arises from interconnectivity, efficiency, and adaptability. Thus, the aspiration for Compassionate A.I. \u2013 <\/strong>aligned with the complexity of human mental processes \u2013 requires a shift:<\/p>\n\n\n\n