= 27d567bc2d48d9f40e9ccc20ea370b15)
Squark comments on How to Study Unsafe AGI's safely (and why we might have no choice) - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (47)
I think it's either that or we won't be able to build any human-level AGI without WBE.
Agreed. However hominid evolution was clearly not pure luck since it involved significant improvement over a relatively short time span.
Evolution produced something which is adapted to a very wide range of environments, including environments vastly different from the environment in which evolution happened. E.g., US Astronauts walked the surface of the moon which is very different from anything relevant to evolution. We call this something "general intelligence". Ergo, it is possible to produce general intelligence by a process which has little of it.
My point is that it's unlikely the point of diminishing returns is close to human intelligence. If this point is significantly below human intelligence then IMO we won't be able to build AGI without WBE.
It is an AI which contains an interpreter as a subroutine. My point is, if you somehow succeed to freeze a self-modifying AI at a point in which it is already interesting and but not yet dangerous, then the next experiment has to start from scratch anyway. You cannot keep running it while magically turning self-modification off since the self-modification is an inherent part of the program. This stands in contrast to your ability to e.g. turn on/off certain input/output channels.
Why?
It wasn't pure luck, there was selective pressure. But this signal towards improvement is often weak and noisy, and it doesn't necessarily correlate well with intelligence: a chimp is smarter than a lion, but not generally more evolutionary fit. Even homo sapiens had a population bottleneck 70,000 which almost led to extinction.
It is my intuition that if something as complex and powerful as human-level intelligence can be engineered in the foreseeable future, than it would have to use some kind of bootstrapping. I admit it is possible than I'm wrong and that in fact progress in AGI would be through a very long sequence of small improvements and that the AGI will be given no introspective / self-modification powers. In this scenario, a "proto-singularity" is a real possibility. However, what I think will happen is that we won't make significant progress before we develop a powerful mathematical formalism. Once such a formalism exists, it will be much more efficient to use it in order to build a pseudo-narrow self-modifying AI than keep improving AI "brick by brick".