{"id":11971,"date":"2023-04-19T15:06:25","date_gmt":"2023-04-19T15:06:25","guid":{"rendered":"https:\/\/aurelis.org\/blog\/?p=11971"},"modified":"2023-04-19T20:02:35","modified_gmt":"2023-04-19T20:02:35","slug":"forward-forward-neuronal-networks","status":"publish","type":"post","link":"https:\/\/aurelis.org\/blog\/artifical-intelligence\/forward-forward-neuronal-networks","title":{"rendered":"Forward-Forward Neur(on)al Networks"},"content":{"rendered":"\n

Rest assured, I don\u2019t stuff technical details into this blog. Nevertheless, this new framework lies closer to how the brain works, which is interesting enough to go somewhat into it.<\/h3>\n\n\n\n

Backprop<\/strong><\/p>\n\n\n\n

In Artificial Neural Networks (ANN) \u2013 the subfield that sways a big scepter in A.I. nowadays \u2013 backpropagation (backprop) is one of the main buzzwords and is used ubiquitously. Present-day A.I. would be nowhere without it. However, that may change.<\/p>\n\n\n\n

<\/p>\n\n\n\n

\"\"
ANN schema with several layers from left to right<\/figcaption><\/figure><\/div>\n\n\n\n

Shortly put, an explanation of backprop:<\/p>\n\n\n\n

A multilayered ANN-in-training tries to accurately categorize, for instance, images of cats and dogs. But it makes many errors. After each pass forward through the network from entry layers to exit layers, an error estimation is fed into the system at the end side and, from there, backprops to earlier layers. So, signals go forward; error estimations go backward through the same network. ANNs work well this way for some problems, not for others.<\/p><\/blockquote>\n\n\n\n

In any case, the human brain hardly works this way.<\/strong><\/p>\n\n\n\n

Firstly, backprop involves a lot of mathematics, while no calculator exists in the brain. There are only neurons \u2013 and other cells \u2013 and groups of them. The brain may simulate a calculator, functionally approaching it to a small degree. But there is no way the brain could perform as many complex calculations as are needed for backprop.<\/p>\n\n\n\n

Secondly, backprop would mean that, at the cellular level, there is a kind of memory that definitely isn\u2019t there.<\/p>\n\n\n\n

Thus, although low-level information in brainy neuronal networks can go in many directions, a backprop direction can hardly be one of them. Also, there is no evidence of it in reality. Backprop is especially (extremely) implausible with information that evolves over time.<\/p>\n\n\n\n

The brain goes forward.<\/strong><\/p>\n\n\n\n

Neurons in the human brain are unidirectional. As a general flow, the information goes from one side (dendrites) to the cell body and from there to the axon connecting with other cells\u2019 dendrites.<\/p>\n\n\n\n

\"\"
Neuron with flow from left to right<\/figcaption><\/figure><\/div>\n\n\n\n

Thus, all in all, and taking into account many loops, forward processing is what the brain does, also when doing it backwards. For instance, much information travels from the brain to the eye, influencing how the eye itself reacts to what it \u2018sees.\u2019 This is no backprop but a forward feed in the backward (or \u2018top-down\u2019) direction.<\/p>\n\n\n\n

A few months ago, Geoffrey Hinton proposed the Forward-Forward algorithm<\/a> (FF) in neural networks.<\/strong><\/p>\n\n\n\n

Shortly put, if an error is made (cat, no dog). FF handles this network experience in a forward fashion \u2015 erroneous and non-erroneous pass alike. The forward trails of both (thus \u2018forward-forward\u2019) are compared, and the system learns.<\/p>\n\n\n\n

This is more aligned with how the brain works \u2015 even to such a degree that both cases (natural and artificial) can be described pretty much in the same way. I just did so and will do some more.<\/p>\n\n\n\n

Given good and bad examples, a neur(on)al network can proceed in two ways:<\/p>\n\n\n\n