AI Alignment Is Turning from Alchemy Into Chemistry, but for real this time

In April 2023, Alexey Guzey posted "AI Alignment Is Turning from Alchemy Into Chemistry" where he reviewed Burns et al.'s paper "Discovering Latent Knowledge in Language Models Without Supervision." Some excerpts to summarize Alexey's post:

For years, I would encounter a paper about alignment — the field where people are working on making AI not take over humanity and/or kill us all — and my first reaction would be “oh my god why would you do this”. The entire field felt like bullshit. I felt like people had been working on this problem for ages: Yudkowsky, all of LessWrong, the effective

That doesn't change the fact that the pioneers really only pursued neural networks because of their similarity to the actual structure of the brain, not by first-principles reasoning about how high dimensionality and gradient descent scale well with data, size, and compute (I understand this is a high bar but this is part of why I don't think there are any real "experts"). And in their early career especially, they were all mired in the neurological paradigm for thinking about neural networks.

Hinton, who got close to breaking free from this way of thinking when he published the backprop paper, ends it by saying "it is worth looking for more biologically plausible ways... (read more)