Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Houshalter comments on Artificial Addition - Less Wrong

36 Post author: Eliezer_Yudkowsky 20 November 2007 07:58AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (117)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: Houshalter 26 June 2014 11:24:25PM 0 points [-]

Artificial Neural Networks have been trained with millions of parameters. There are a lot of different methods of regularization like dropconnect or sparsity constraints. But the brain does online learning. Overfitting isn't as big of a concern because it doesn't see the data more than once.

Comment author: Punoxysm 27 June 2014 01:30:57AM *  0 points [-]

On the other hand, architecture matters. The most successful neural network for a given task has connections designed for the structure of that task, so that it will learn much more quickly than a fully-connected or arbitrarily connected network.

The human brain appears to have a great deal of information and structure in its architecture right off the bat.

Comment author: [deleted] 27 June 2014 05:49:40AM 0 points [-]

The human brain appears to engage in hierarchical learning, which is what allows it to leverage huge amounts of "general case" abstract knowledge in attacking novel specific problems put before it.

Comment author: Houshalter 27 June 2014 06:34:12PM 0 points [-]

I'm not saying that you're wrong, but the state of the art in computer vision is weight sharing which biological NNs probably can't do. Hyper parameters like the number of layers and how local the connections should be, are important but they don't give that much prior information about the task.

I may be completely wrong, but I do suspect that biological NNs are far more general purpose and less "pre-programmed" than is usually thought. The learning rules for a neural network are far simpler than the functions they learn. Training neural networks with genetic algorithms is extremely slow.

Comment author: Punoxysm 27 June 2014 07:12:05PM 0 points [-]

Architecture of the V1 and V2 areas of the brain, which Convolutional Neural Networks and other ANNs for vision borrow heavily from, is highly geared towards vision, and includes basic filters that detect stripes, dots, corners, etc. that appear in all sorts of computer vision work. Yes, no backpropagation or weight-sharing is directly responsible for this, but the presence of local filters is still what I would call very specific architecture (I've studied computer vision and inspiration it draws from early vision specifically, so I can say more about this).

The way genetic algorithms tune weights in an ANN (and yes, this is an awful way to train an ANN) is very different from the way they work in actually evolving a brain; working on the genetic code that develops the brain. I'd say they are so wildly different that no conclusions from the first can be applied to the second.

During a single individual's life, Hebbian and other learning mechanisms in the brain are distinct from gradient learning, but can achieve somewhat similar things.