= 27d567bc2d48d9f40e9ccc20ea370b15)
V_V comments on How to Study Unsafe AGI's safely (and why we might have no choice) - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (47)
I think your points are valid. I don't expect FOOM from anything, necessarily, I just find it plausible (based on Eliezer's arguments about all the possible methods of scaling that might be available to an AI).
I am pitching my arguments towards people who expect FOOM, but the possibility of non-FOOM for a longish while is very real.
And It is probably unwarranted to say anything about architecture, yo'ure right.
But Suppose we have human-level AIs, then decide to consciously build a substantially superhuman AI. Or we have superhuman AIs that can't FOOM, and actively seek to make one that can. The same points apply.
It seems to me that this argument (and arguments which rely on unspecified methods and the like) boils down to breaking the world model to add things with unclear creation history and unclear decomposition into components, and resulting non-reductionist magic infested mental world model misbehaving. Just as it always did in the human history, yielding gods and the like.
You postulate that unspecific magic can create superhuman intelligence - it arises without mental model of necessary work, problems being solved, returns diminishing, and available optimizations being exhausted - is it a surprise that in this broken mental model (broken because we don't know how the AI would be built), because the work is absent, the superhuman intelligence in question creates a greater still intelligence in days, merely continuing the trend of it's unspecific creation? If it's not at all surprising then it's not informative that mental model goes in this direction.