Series: The Sequentia Lectures: Unlocking the Math of AI
Part 7: The Frontier – New Paradigms & Unsolved Puzzles
Lecture 67: The Organic AI: Learning from the Ultimate Computer, The Brain
Throughout this series, we’ve used the term “neural network” and drawn analogies to the biological brain. We’ve talked about neurons, connections, and learning. But it’s time for a crucial admission: the Artificial Neural Networks (ANNs) we’ve discussed are, at best, a crude caricature of the real thing.
The human brain is a marvel of computation. It performs tasks of incredible complexity—recognizing faces, understanding language, creating art—all while running on about 20 watts of power. Our current deep learning models, in contrast, are brute-force calculators that require megawatts.
This vast efficiency gap has inspired a research frontier that looks back to the original source code: the brain itself. This is the quest for a more “organic” or biologically plausible AI.
The Problem with Our “Fake” Neurons
The neurons we’ve used in our models are fundamentally different from biological ones. Our artificial neurons:
- Are always “on”: In every forward pass, every neuron calculates a value (even if it’s zero after ReLU) and passes it on. They are constantly computing.
- Communicate with real numbers: They pass continuous, high-precision values (activations) to each other.
- Are synchronous: They operate in lock-step, with one layer’s calculations finishing before the next begins.
Biological neurons are far more efficient and complex. They are largely silent, communicating only when necessary by sending short, discrete electrical pulses or “spikes.” They are event-driven and asynchronous.
Spiking Neural Networks (SNNs): The Next Generation
This observation has led to the development of Spiking Neural Networks (SNNs). SNNs are a “third generation” of neural networks that aim to more closely mimic the brain’s behavior.
- Event-Driven, Not Continuous: In an SNN, neurons don’t continuously output a value. They remain dormant until their internal “membrane potential” (a simulated electrical charge) reaches a certain threshold. When it does, the neuron fires a single spike and then resets.
- Time is a Key Variable: The information in an SNN is encoded not just in which neurons fire, but when they fire. The precise timing and frequency of these spikes carry the information.
- Extreme Energy Efficiency: Because most neurons are silent most of the time, SNNs are theoretically far more energy-efficient. They only compute when there’s an “event”—a spike—to process.
The challenge with SNNs is that the discrete, all-or-nothing nature of the spikes makes them difficult to train using the smooth, calculus-based methods of Gradient Descent and Backpropagation that work so well for traditional ANNs. Developing effective learning rules for SNNs is a huge area of active research.
Neuromorphic Computing: Hardware That’s Built Like a Brain
If our software is trying to mimic the brain, why not our hardware? This is the radical idea behind neuromorphic computing.
Instead of the standard computer architecture (CPU/GPU) that shuffles data back and forth between a processing unit and a memory unit (a major bottleneck), neuromorphic chips are designed from the ground up to be more like a brain.
- Co-located Memory and Processing: On a neuromorphic chip, memory and computation are physically integrated. “Synapses” (connections) and “neurons” (processors) are built together, mimicking the brain’s distributed structure.
- Designed for Spikes: This hardware is optimized to process the sparse, event-driven signals of SNNs with incredible efficiency.
- Parallel and Asynchronous: They are massively parallel, just like the brain, and don’t rely on a global clock, operating in a more natural, asynchronous way.
Companies like Intel (with its Loihi chip) and IBM (with TrueNorth) are pioneering this field, creating hardware that is not just simulating a brain, but is structurally analogous to one.
The Ultimate Puzzle: Unlocking the Brain’s Algorithm
The journey towards a more organic AI is a grand challenge that sits at the intersection of neuroscience, computer science, mathematics, and physics. We still don’t fully understand the learning algorithms the brain itself uses.
By building models like SNNs and hardware like neuromorphic chips, we are not just trying to create more efficient AI. We are also creating tools that help us test our hypotheses about how the brain itself might work.
Can we move beyond the brute-force success of today’s deep learning and discover learning principles that are as elegant and efficient as nature’s? This question is driving the search for a truly “neural” network, one that learns not just from our data, but from the ultimate computer that created it: the human brain.
