This is a special post for quick takes by .CLI. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Inspired by davidad's tweets: 1, 2, 3
It seems commonsense that a deeper (theoretical) understanding helps both engineering as well as safety engineering. Which one do you think does theory help more? And which development helped grow theory research more?
My intuition is that:
How well does this mesh with real-life? In the bridges' case, safety engineering was invented separately, well after we understood how to build bridges - and well after we built a lot of bridges. The pioneers in safety engineering oft have formal math background. This seems to match the intuition above.
That said -
(ramblingly) Does the No Free Lunch Theorem imply that there's no one single technique that would always work for AGI alignment? Initial thought is probably not, because the theorem states that the performance of all optimization algorithms are identical across all possible problems. However, AGI alignment is a subset of these problems.
See Steve Byrne's take on the no free lunch theorem.
No is the answer to "does the NFL theorem prove x" for x we care about I'm pretty sure.
I basically agree with @davidad's comment on the NFL, because basically all of the time, you're aiming to find not just a good solution (which satisficing is), but an optimal solution, and thus what it really proves is that in the worst case, intelligence is necessarily and sufficiently look-up tables, and you need to resort to brute-force search, which is wildly intractable:
https://www.lesswrong.com/posts/yTvBSFrXhZfL8vr5a/?commentId=N3avtTM3ESH4KHmfN
I also disagree with @jsteinhardt on the min-max/optimal solution being unacceptably bad, assuming perfect specification of what we value (this is obviously not a safe assumption in practice, but is probably safe from a philosophical/theoretical perspective.)
[just now learning about the no free lunch theorem] oh nooo, is this part of the reason so many AI researchers think it's cool and enlightened to not believe in highly general architectures?
Because they either believe the theorem proves more than it does or because they're knowingly performing an aestheticised version of it by yowling about how LLMs can't scale to superintelligence (which is true, but also not a crux).
is this part of the reason so many AI researchers think it's cool and enlightened to not believe in highly general architectures
I do hear No Free Lunch theorem get thrown around when an architecture fails to solve some problem which its inductive bias doesn't fit. But I think it's just thrown around as a vibe.
I think there should be more effort into researching the limits of controllability for self-improving machines. That aspect of rapid self improvement seems pretty important to me since it's there regardless of which architecture we use to get to the singularity. If the singularity is dangerous no matter how we get there, or how aligned our first try is, then, [clears throat and raises sign] don't build AGI?