Neural networks are based on an oversimplified model of neuron cells.
As a programmer who has studied neuroanatomy and the structure/function of neurons themselves, I remain astonished at how not like real biological nervous systems computer neural networks still are. It’s like the whole field is based on one person’s poor understanding of the state of biological knowledge in the late 1970s. That doesn’t mean it’s not effective in some ways as it is, but you’d think there’d be more experimentation in neural networks based on current biological knowledge.
The one thing that stands out to me the most is that programmatic “neurons” are basically passive units that weigh inputs and decide to fire or not. The whole net is exposed to the input, the firing decisions are worked through the net, and then whatever output is triggered. In biological neural nets, most neurons are always firing at some rate and the inputs from pre-synaptic neurons affect that rate, so in a sense the passed information is coded as a change in rate rather than as an all-or-nothing decision to fire or not fire as is the case with (most) programmatic neurons. Implementing something like this in code would be more complicated, but it could produce something much more like a living organism which is always doing something rather than passively waiting for an input to produce some output.
And TBF there probably are a lot of people doing this kind of thing, but if so they don’t get much press.
Pretty much all artificial neural nets I have seen don’t do all or nothing activation. They all seem to have activation states encoded as some kind of binary number. I think this is to mimic the effects of variable firing rates.
The idea of a neural network doing stuff in the background is interesting though.
As a programmer who has studied neuroanatomy and the structure/function of neurons themselves, I remain astonished at how not like real biological nervous systems computer neural networks still are. It’s like the whole field is based on one person’s poor understanding of the state of biological knowledge in the late 1970s. That doesn’t mean it’s not effective in some ways as it is, but you’d think there’d be more experimentation in neural networks based on current biological knowledge.
What sort of differences are we looking at exactly?
The one thing that stands out to me the most is that programmatic “neurons” are basically passive units that weigh inputs and decide to fire or not. The whole net is exposed to the input, the firing decisions are worked through the net, and then whatever output is triggered. In biological neural nets, most neurons are always firing at some rate and the inputs from pre-synaptic neurons affect that rate, so in a sense the passed information is coded as a change in rate rather than as an all-or-nothing decision to fire or not fire as is the case with (most) programmatic neurons. Implementing something like this in code would be more complicated, but it could produce something much more like a living organism which is always doing something rather than passively waiting for an input to produce some output.
And TBF there probably are a lot of people doing this kind of thing, but if so they don’t get much press.
Pretty much all artificial neural nets I have seen don’t do all or nothing activation. They all seem to have activation states encoded as some kind of binary number. I think this is to mimic the effects of variable firing rates.
The idea of a neural network doing stuff in the background is interesting though.