LESSWRONG
LW

CuriousApe11

Message

Natural Selection vs Gradient Descent

Why is it so often that analogies are drawn between natural selection and gradient descent in a machine learning context? They are both optimizing over a fitness function, but isn't there an important difference in what they are optimizing over? Natural selection is broadly optimizing over the architecture, initial parameters...

May 1, 2023•4

CuriousApe11

CuriousApe11 — LessWrong

CuriousApe11

Message

Natural Selection vs Gradient Descent

May 1, 2023•4

CuriousApe11

Natural Selection vs Gradient Descent

CuriousApe11

Natural selection is broadly optimizing over the architecture, initial parameters of the architecture, and the learning dynamics (how one updates the parameters of the architecture given data), which led to the architecture of the brain and methods of learning like STDP, in which the parameters of the architecture are the neurons of the brain.

Isn't gradient descent instead what we pick to be the learning dynamics, where we then pick our architecture (e.g. transformer)... (read more)