= 27d567bc2d48d9f40e9ccc20ea370b15)
Douglas_Knight comments on Open thread, Mar. 23 - Mar. 31, 2015 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (181)
One thing I've been wondering about deep neural networks: to what extent are neural networks novel and non-obvious? To what extent has evolution invented and thus taught us something very important to know for AI? (I realize this counterfactual is hard to evaluate.)
That is, imagine a world like ours but in which for some reason, no one had ever been sufficiently interested in neurons & the brain as to make the basic findings about neural network architecture and its power like Pitts & McCulloch. Would anyone reinvent them or any isomorphic algorithm or discover superior statistical/machine-learning methods?
For example, Ilya comments elsewhere that he doesn't think much of neural networks inasmuch as they're relatively simple, 'just' a bunch of logistic regressions wired together in layers and adjusted to reduce error. True enough - for all the subleties, even a big ImageNet-winning neural network is not that complex to implement; you don't have to be a genius to create some neural nets.
Yet, offhand, I'm having a hard time thinking of any non-neural network algorithms which operate like a neural network in putting together a lot of little things in layers and achieving high performance. That's not like any of your usual regressions or tests, multi-level models aren't very close, random forests and bagging and factor analysis may be universal or consistent but are 'flat'...
Nor do I see many instances of people proposing new methods which turn out to just be a convolutional network with nodes and hidden layers renamed. (A contrast here would be Turing's halting theorem: it seems like you can't throw a stick among language or system papers without hitting a system complicated enough to be Turing-complete and hence indecidable, and like there were a small cottage industry post-Turing of showing that yet another system could be turned into a Turing machine or a result could be interpreted as proving something well-known about Turing machines.) There don't seem to be 'multiple inventions' here, as if the paradigm were non-obvious and, without the biological inspiration.
So if humanity had had no biological neural networks to steal the general idea and as proof of feasibility, would machine learning & AI be far behind where they are now?
How many components go into "neural nets"?
At the very least, there are networks of artificial neurons. You seem to accept Ilya's dismissal of the artificial neuron as too simple to credit, but take the networks as the biologically inspired part. I view those components exactly opposite.
Networks of simple components come up everywhere. There were circuits of electrical components a century ago. A parsed computer program is a network of simple components. Many people doing genetic programming (inspired by biology, but not neurology) work with such trees or networks. Selfridge's Pandemonium (1958) advocated features built of features, but I think it was inspired by introspective psychology, not neuroscience.
Whereas the common artificial neuron seems crazy to me. It doesn't matter how simple it is, if it is unmotivated. What seems crazy to me is the biologically inspired idea of a discrete output. Why have a threshold or probabilistic firing in the middle of the network? Of course, you want something like that at the very end of a discrimination task, so maybe you'd think of recycling it into the middle, but not me. I have heard it described as a kind of regularization, so maybe people would have come up with it by thinking about regularization. Or maybe it could be replaced with other regularizations. And a lot of methods have been adapted to real outputs, so maybe the discrete outputs didn't matter.
So that's the "neural" part and the "network" part, but there are a lot more algorithms that go into recent work. For example, Boltzmann machines are named as if they come from physics, but supposedly they were invented by a neuroscientist because they can be trained in a local way that is biologically realistic. (Except I think it's only RBMs that have that property, so the neuroscientist failed in the short term, or the story is complete nonsense.) Markov random fields did come out of physics and maybe they could have lead to everything else.