Comment on AIFoom Debate - conclusion?

Thu, 10 Mar 2016 23:04:37 GMT

The human brain has about 100bn neurons and operates at 100Hz. The NVIDIA Tesla K80 has 8.73TFLOPS single-precision performance with 24GB of memory. That's 1.92bits per neuron and 0.87 floating point operations per neuron-cycle. Sorry, no matter how you slice it, neurons are complex things that interact in complex ways. There is just no possible way to do a full simulation with ~2 bits per neuron and ~1 flop per neuron-cycle

You are assuming enormously suboptimal/naive simulation. Sure if you use a stupid simulation algorithm, the brain seems powerful.

As a sanity check, apply your same simulation algorithm to simulating the GPU itself.

It has 8 billion transistors that cycle at 1 ghz, with a typical fanout of 2 to 4. So that's more than 10^19 gate ops/second! Far more than the brain . ..

The brain has about 100 trillion synapses, and the average spike rate is around 0.25hz (yes, really). So that's only about 25 trillion synaptic events/second. Furthermore, the vast majority of those synapses are tiny and activate on an incoming spike with low probability around 25% to 30% or so (stochastic connection dropout). The average synapse has an SNR equivalent of 4 bits or less. All of these numbers are well-supported from the neuroscience lit.

Thus the brain as a circuit computes with < 10 trillion low bit ops/second. That's nothing, even if it's off by 10x.

Also, synapse memory isn't so much an issue for ANNs, as weights are easily compressed 1000x or more by various schemes, from simple weight sharing to more complex techniques such as tensorization.

As we now approach moore's law, our low level circuit efficiency has already caught up to the brain, or is it close. The remaining gap is almost entirely algorithmic level efficiency.

Discuss